IPP Progress Report for the week 2012.03.12 - 2012.03.16

(Up to IPP Progress Reports)

Eugene Magnier

I spent the week understanding detailed issues with the ubercal'ed dvo database / reference database. I completed an initial run of the full sky relphot analysis on Monday. However, I discovered surprising NAN values in the database, which I tracked down to the ubercal table. It turns out that ubercal sets the zero point for rejected images to NAN (I thought they were excluded from the table). I updated the code to tread NAN zero-pointed images as not part of ubercal, and re-ran the process. I also ran relastro on the full sky, but discovered that too many images had unreliable solutions. Looking through the analysis, I realized this was due to the selection stars used to perform the calibration: in order to avoid blowing up the memory footprint, I limit the density of stars to some specific value. After basic S/N cuts, I was selecting that subset randomly. This allowed too may objects with only a limited number of overlap detections to be used, which meant some images had very few stars. I made a few changes: 1) I bumped up the density (from 1000 to 2500 stars per square degree), 2) I modified the selection to preference objects with many measurements, 3) I relaxed the S/N cuts a bit to give a bit better sampling for all images. This helped to reduce greatly the number of bad astrometry failures, but pushed the memory up alot. I needed to reduce the fraction of the sky per chunk to fit in the footprint of ipp064 (48GB). The analysis now runs on 13 chunks: 12 RA slices from -60 : 60 dec and the polar cap. Although the I/O time is quite reasonable, with the parallelized analysis, the processing time is now substantial : something like 40 hours is needed to do the analysis of all chips in the database. I am looking at threading that part of the code for the future.

We also jointly spent alot of time dealing with hardware issues. This was largely driven by A/C failures in both Manoa and at the ATRC. For several days, we had to keep our backup storage machines (ippbNN) off because of the A/C at the ATRC. Similarly, 2 of our manoa development / operations machines are off for now. It is unclear how long this will continue.

We also had crashes of ipp014 and ipp015 caused by glitches with the CPUs. There is a concern that at least one (if not both) have CPU fan problems. For a while, it was unclear if ipp015 would fail to reboot, so I updated part of the distributed dvo database code to allow data to be stored on an alternate machine. This is currently used for ipp015.

Serge Chastel

  • Fixed gpc1 replication. Ingested 2012-03-12T12:05 dump on ipp001 which was seriously broken
  • Helped fix nebulous on Tuesday.
  • Czar on Wed, Thu, Fri.
  • Liaison while Rita and Haydn were at mhpcc.

Heather Flewelling

  • dvodb work
    • policy changes for LAP: make separate minidvodbs (cam is one, staticsky is the other)
    • merge staticsky and staticsky multi: now just staticsky.
    • all of this is checked in and tested and recompiled for ippdvo
    • query problem (no solution yet):
      • found query such that if there are 2 cam_ids or sky_ids for a given exp_id (or stack_id), it will take the most recent processing
      • is that what we want to do? Or should we avoid queueing these at all? Should we drop the bad ones so that we don't have the tools 'guess' which is the best (this is bill's idea)
    • sent email to Bill about his ideas for staticsky changes
  • sick one day

Roy Henderson

  • 4 day week (off on Wednesday)
  • development:
    • fixed bug where loader programs kept reconnecting to scratch Dbs for every batch
    • sped up queuing some more with more efficient queries
    • added logging options to constructor of base-class for all ippToPsps programs. Default is now stdout
    • created a quick 'console' GUI to help loading management:
      • manage clients, pause, kill, remove, change config
      • basic config editing
    • various fixes to get stacks running again (hadn't produced stacks since some Db changes a while ago)
    • JOINing to Bill's skycells table to get RA/Dec for stacks when queuing. Handy for density plot as well as queuing certain regions of sky
    • finished on-the-fly config changing from loaders, so no need to stop client to change loading survey
    • new method to write a Config object to database table
  • loading:
    • LAP galactic plane loading - very slow, played with configuration
    • loaded test data for SA3 to test datastore for Conrad and Thomas
    • loaded SA3 to production (only took a few hours)

Mark Huber

  • Czar Monday and Tuesday
  • Caught up with DRAVG minutes
  • ppStack work: checked in ppSub:STACK SYS.ERR=0 configuration change to ipp-20120216 once LAP finished and monitored nightly science stacks. Continued looking into the convolved PSF growth in different skycells, filters.

Bill Sweeney

  • This was another week working on psphotStack. Progress was finally made on the memory usage problem. The primary issue is that in busy skycells (near the galacitc plane or containing a globular cluster) many large footprints that contain many sources are being created. In the merge process, where the "missing" sources found in one filter but not in another are prepared for forced photometry analysis, these footprints were being copied for each "missing" source. Since some of these were so large - as big as the input images - and complex - hundreds of thousands of spans, and contained tens of thousands of sources, these consumed all memory and swap space. Implemented a cache so only one copy is made for each input and this eliminatedd. However there are certain places in the code which handle these large footprints sub-optimally. For one skycell in the galactic plane it took about 18 hours to complete the analysis. But it did complete.
  • Another issue that needs to be resolved is duplicate sources being reported at the position of sources found in pass 1 of the psphot analysis. I found several cases where isolated stars were detected in pass 1, and another weak source was detected in pass 2 which finds detections on an image where the pass 1 sources have been subtracted off. During the psphot matching stage sometimes a perfectly good star is matched with the perfectly good star in other filters, but sometimes one good detection gets matched with the pass 2 source. Thus in the output we will have 2 sources with very different measurements in the different filters.
  • Added a feature to the postage stamp extractor program to return an entire copy of the input image. This is useful for two reasons.
    • The LAP stacks are stored with ASIN scaling of the pixel values. The output from ppstamp is linear scaling so it "undoes" that which is useful for examining stack images.
    • The stamp outputs for chip stage replace the inaccurate WCS solution from the telescope, with the astrometric solution from the camera stage.

Chris Waters

  • Migrated processing to use the new dark (det_id 853).
  • Constructed B-mode dark, and confirmed that it does a better job correcting B-mode data than either the old mixed dark or the new A-mode dark. The slope trends for these three darks are all identical with dateobs, but are shifted relative to each other. No substantial progress made in determining why a given date chooses the particular mode, or why the B-mode disappeared after 2011-05-01.
  • Ran script to manually fix corrupt copies related to the ipp064 RAID crash. This should permanently resolve the XY26 issues that we've had in the past.
  • Confirmed that UNKNOWN data in the disk usage scripts includes all data not in nebulous.
  • Looked at false detections in the dark test footprint using single and 12x detections as the proxies for false and real objects. Applying poor and bad flag cuts removes the majority of "bright" false positives, leaving only a tail that could plausibly be events that randomly move into the 5-sigma detection limit. Comparing these objects to warp and stack images shows that some are in fact real objects, but the majority seem to be streak/burntool/etc variations.