IPP Progress Report for the week 2012.04.23 - 2012.04.27

(Up to IPP Progress Reports)

Eugene Magnier

I've been frustrated over the past several weeks by relative photometry failing to determine good solutions for some areas. I've continued to look into this, and I've identified what I think are the remaining causes of problems in the current code. First, I had been failing to select all of the images appropriate to a given region if that region included the 0,360 boundary. The details are a bit obscure, but the bottom line is that in those regions, only a portion of the images were being selected. This problem had been persisting for some time. Second, I found a couple of loop index errors in the code when I threaded the photometry analysis, which was causing problems over the last 2 weeks or so. Those are now fixed (including one annoying segfault). I've finally started to get reasonable answers across the entire sky. The plots show that, when looking at only the ubercal measurements, there are very clean residuals across the sky, as Eddie & Doug showed. When non-ubercal images are included, the scatter increases for some images, but not excessively (up to 1-2%). The scatter is mostly coming from 2D structure in the effective zero point across an exposure. Since I am fitting only a single constant per exposures, the presence of clouds leaves behind a small residual pattern (often just a tilt) with a mean of zero, but non-zero edges. This is not surprising, and shows that I am getting the mean zero points right, but also shows that we can consider fitting a 2D variation of the zero points for exposures that are (a) sufficiently bad to warrant it and (b) well-enough connected to photometric (or well-calibrated) exposures to have a sensible tie down.

Other things I've been working on:

  • dvomerge into a parallel DVO database -- this is the path forward on merges for the big database
  • helping Bill with psphot kron mags
  • helping Chris with the ghosts & false positives

Serge Chastel

  • ippb02 exposures have now been fixed. Instances that could not be recovered have been moved form the nebulous instance table to the lost_instances one.
  • Czar on Wed and Thu
  • Fixed a few issues with mysql replication
  • Continued working on the updatedb to provide statistics for nebulous (See the very preliminary http://ippmonitor.ipp.ifa.hawaii.edu/nebMonitor/ or http://ippmonitor.ipp.ifa.hawaii.edu/nebMonitor/blah.php pages). The intent is to be able to provide weekly (or as frequent as possible) reports looking like the one I created a few weeks ago about the disk usage on the cluster storage nodes.

Heather Flewelling

  • megacam
    • determined too many megacam things
      • yes, there is a recipe difference between now and then
      • yes, there is a code change between now and then - it shouldn't cause problems, but nearly half of the exposures have funny crosstalk, and the difference causes a different exposure to be dropped (it shouldn't), which causes the crosstalk ones to dominate.
      • investigated the warps on a skycell (which ones are included, why?)
      • the cross talk images are raw images, not processed. Downloaded the processed ones, ingested them, and cleaned up the database to prepare for reprocessing of megacam.
  • while waiting for different revisions of ipp to compile to find where megacam breaks, started working on how to make ipp compile on my desktop.
  • dvos for LAP/ThreePi are continuing to be built
  • PSPS
    • (with Thomas and Roy) ingest SAS.20120403 again (this is the relphoted one, and also we ingested objects)
  • I went to the astrocoffee, and I only mention it here because the librarian suggested that Pan-STARRS should have a list on ads (like other telescopes do) of pan-starrs related papers...

Roy Henderson

  • ORR for two days
  • started work on developer's documentation for PSVO on the PSPS system wiki
  • worked on a Python DRL client for PSPS, ran into issues with the wsdl format
  • PSVO:
    • added a MyDB plugin menu. Queries include show tables, drop table and current space usage
    • fixed bug when parsing results from MyDB exec sp_spaceused function: problem was whitespace in column name
  • more ippToPsps training for Heather and Thomas
  • fixed bug in ippToPsps with Kron fluxes, just introduced last week
  • a small number of support emails for PSPS

Mark Huber

  • Czar Monday and Tuesday (kicking Roy off the schedule), usual issues: unsticking registration during night, stdscience stalling, tracking down odd/slow processing exposures, distribution flushing from destreak_restore pileup to get out time sensitive products.
  • MD08.GR (grand reprocessing) -- http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/MD.GR0#MD08.GR0
    • continuing from mostly setup last week to finishing this week
    • ~8 days overall with nights+weekend (processing + setup/inspection/verification/documentation time). 1 filter (~500 exposures) can finish chip->warp processing in 4-6 hours (chip rate ~110 exposures/hour limiting factor), and can also process along normal nightly science with the new nodes at about half that rate.
      • tested Serge's reconfig of condor the previous week, but with ganglia down, ended up not using condor this round (no LAP currently to contend with), manually monitoring would end up cutting in to too much other work.
      • adding extra cuts to help constrain target PSF in convolved stacks (relates to PSF evaluation in ppStack work).
      • lost images (Serge investigated and has tool now, but they are gone) and always fun manual burntool to fix.
      • correlating night stack reprocessing faults with PSF problems (also ties to PSF evaluation in ppStack work).
    • additional reftest to included all exposures and just from previous year (2011).
    • added staticsky run with improved convolved stacks.
    • looked at worse case of a "school of ghosts" in g-band.
    • updating processing node access on Harvard/odyssey system (delayed from recent node user shakedown until now needed).
    • MD08-z observed early on weekend, restarted normal MD08 processing with new refstack.
  • ppStack tests -- http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ppStack_testing_201111
    • still working through 7TB of tests on ipp059.0 to clean up while winding down setup to switch again to ppSub.
  • ppSub tests -- http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ppSub_testing_201201
    • finished the tedious process of setting up additional template and test images for local photpipe with an additional filter based on some other QUB examples with problems while waiting for QUB to prepare their IPP vs photpipe comparison (out on travel still).
    • normal hangups tracing problems in two pipelines. Sample of convolved reference/template image (default), auto convolution choice with photpipe and convolved input image with IPP finished, slowly making comparison images.

Bill Sweeney

  • This week was spent investigating the low values for Kron Radius measured by psphot as compared with SExtractor. Added some new recipe values for fine tuning the calculations. Wrote and executed scripts to run psphot on 40 simulated images provided by Durham and gather the results. Wrote mana scripts for comparing to expected and SExtractor measurments and plotting the results. We found a number of minor problems in the code and one big one that improves the situation greatly. The keys were to use the correct integration region and to smooth the image. Reported the results to the DRAVG. With a prototyped smoothing function psphot now gets KR values within a couple of percent of SExtractor. Still need to do an efficient implementation.
  • Investigated and fixed problems with IPP build at MPIA.
  • Debugged some ppImage processes that got stuck. Eventually killed them off.

Chris Waters

  • Noise: Constructed noisemap boosts for all time regimes back to the beginning of the mission.
  • Ghosts: Identified bugs in the ghost image masking process, and updated the ghost model. Still not completely correct, as high radius ghosts are missed by masks.
  • Crosstalk: Identified that bright diffraction spikes can create crosstalk ghosts similar to known crosstalk rules for stellar cores.
  • Began work on videodark/mask files to allow us to mask regions with suppressed corner glow due to video readout.