IPP Progress Report for the week 2011.11.21 - 2011.11.25

(Up to IPP Progress Reports)

Eugene Magnier

This was a short week with the turkey-based holiday, and a bit shorter for me. I spent some time exploring the neighbor-detection flags that are raised in ppSub. These are meant to mark detections which are likely to be from the wings of poorly subtracted stars: one bit is raised if the diff detection is close to a source in both the A and B image, a second bit is raised if the diff detection is close to only one or the other. I discovered that a counting error resulted in only a small fraction of the available A or B image detections being used for the matching, making the flag frequently wrong. I fixed this and also added a flag to note cases where the positive image detection is likely to be exactly the diff detection (ie, the closest detection is the positive image of the object itself).

Serge Chastel

  • Condor documentation

Heather Flewelling

  • czar
  • dvodb
    • dvoverify finished - All is well
    • wrote a script to check that a merge finished - ThreePi? and LAP are both fine.
    • investigated multiple camRun problem in addRun - this bug has been fixed a while ago for LAP, but also affects ThreePi?.
  • vacation/thanksgiving

Roy Henderson

  • Put PSPS loading data into czartool for easier monitoring.
    • removed revision/db upgrade stuff from czartool: not necessary, as I will maintain a dump of schema shape instead.
    • added tables to CzarDb of same shape as IPP ones: 'pending', 'processed', 'faults' for each stage
    • changes to ippToPsps metrics program to write new CzarDb tables
    • adapted czarplot to deal with ipptopsps stages
    • put ipptopsps metrics program on a timer so it can routinely write to CzarDb
    • updated CzarDb maintenance code to trim and optmize new tables
    • updated czarpoll to produce ippToPsps plots
    • created new PHP page in czartool for display
  • Duplicates broke PSPS merge. Investigated and found that duplicate frames were in DVO. Helped Sue remove them before re-merging. This prompted some extra duplicate protection measures:
    • created ipptopsp 'test' database to reduce confusion when I'm testing new code
    • increased 'processing now' limit from 2 to 4 hours, as when network is slow it is feasible that a batch could take longer than 2 hours
    • removed 'loaded_to_datastore' constraint when checking if a stage_id has already been processed as it is a little dangerous (could be called just after a batch is processed by one client, but before it uploads that batch to the datastore)
  • Other ippToPsps coding:
    • found bug in loading code: batches that failed at the tar and zip stage prior to publication to datastore were being labelled as successfully processed, then failing to publish
    • populated some new fields for stacks
  • Other
    • my crontab'd backup script for various Dbs (ipptopsps, czadb, mailing lists) now dumps a copy of the schema as well
    • cleaned-up some failed batches
    • work setting-up re-loading of SAS data to PSPS

Mark Huber

  • psphotstack tests with additional runs working through issues for footprint, SAS2.123, MD04
  • MD06 sample nightly stacks and y-band stack run, MD08 reprocessing setup for condor and prep for test stacks
  • revived and extracted data sample for ppstack/ppsub tests but ran out of time for tests
  • vacation+thanksgiving

Bill Sweeney

  • deployed the video photometry stage. Processed 1 month's worth of data (January 2011). Now waiting for feedback from Pavlos.
  • dealt with multiple destreak failures due to host problems.
  • Fixed bug in destreak introduced in the latest tag. This will require reprocessing data processed between 2011-11-11 and 2011-11-24.
  • Started familiarizing myself with outputs of static sky analysis in preparation for creating the new "staticsky calibration" stage.
  • modified the postage stamp server to distribute get_image data products around the cluster using nebulous. This avoids filling up the data store server's disk.
  • debugged and fixed a problem with update processing that was causing postage stamp requests to get stuck waiting for updates.

Chris Waters

  • Diskspace: rsynced data from ippb02 to ippb03 to allow disk balancing code to write to ippb02 (as it was previously 100% full). It appears to be full again, so some more work will need to be done.
  • SSTF: Processed more data using 4-sigma detection limit, as well as quickstack/warp diffs.
  • Sky background/pattern correction: Propagated changes back through camera formats, and ran initial tests that this improved the background levels on those cells. Still need to check that the magic masking does not suffer due to these changes before committing to the trunk and tag.
  • Thanksgiving.