IPP Progress Report for the week 2010.05.31 - 2010.06.06

(Up to IPP Progress Reports)

Eugene Magnier

Heather Flewelling

• processing czar tuesday and wednesday
  • set to goto_cleaned ancient cam/fake/warps etc that were never going to process
  • found a warptool bug
  • cleaned up IPP operations wiki
• finished addtool and merged
  • found a case where dvomerge fails to merge and sent to Gene
• reported the IPP wiki was down on monday
• made ISP/GPC1 sky brightness plot for Ken - problems with observations (only had y band GPC1/ISP obsevations), and ISP still looks 'funny', but not like chevrons (not like stars, either).

Bill Giebink

Roy Henderson

• PSPS
  • Generated more test data for Jim to figure out bug in astrometry code
  • Generated a list of queries to insert frameName for a given frameID for Sue to run
  • Surprisingly large amount of time spent on Wiki configuration (icon, menu, style, permissions etc)
  • Loaded remainder of demo-month 3PI data, thanks to Heather's new DVO database

• IPP
  • Changed code to take exposure name as an argument and pass it through to the FITS files
  • Numerous changes to incorporate image and detection flags from the IPP
  • Settled a reliable query to pull out all exposures from a given period and coded it into my 'run' script

Paul Price

• Background restored pipeline:
  • Defined database tables: chipBackgroundRun, chipBackgroundImfile, warpBackgroundRun, warpBackgroundSkyfile
  • Wrote bgtool to implement database operations
• Changed "all images rejected" error in ppStack to bad data quality
• MOPS interface format:
  • Reworked ppMops to write SMF-like format (including detection efficiencies), purging duplicates
  • Added functions psFitsReadTableAllColumns and psFitsWriteTableAllColumns to read/write all columns without the additional memory overhead of psFitsReadTable
  • Discovered and fixed bugs in psTree for spherical coordinates
  • Sent example of new interface format to Larry for approval

Bill Sweeney

For Bill, this report covers May 24 - June 5.

• processing czar 1 day, 1 holiday, 3 vacation days
• investigated and worked around chip processing problem due to pre-mature schema changes to the gpc1 DB
• implemented priority ordering for warp stage
• investigated how a row got deleted from chipProcessedImfile. Failed to find the cause in the mysql binary logs. It is still a mystery.
• Fixed a couple more cases where the update process stops and the postage stamp server does not handle the error.
• Did some cleanup of the code in the postage stamp web interface to prepare for further development.
• wrote a wiki page on shepherding the postage stamp server
• investigated why 2 postage stamp requests got stuck. It turns out the warpRun's corresponding magicDSRun were in the state failed_cleanup. This prevented the warpSkyfiles from getting queued for update. Cleaned up the magicDSRun faults and the jobs worked. We still need more "revert" tasks to handle cases like these.
• Fixed a bug that Heather reported where warptool -updaterun didn't work unless the run had been at least partially processed. In the process of investigating this, found and fixed a potential problem with -updaterun -set_state update by disallowing that mode to be used for that purpose.
• Did some reading on http access control for application to the postage stamp server and data store.
• Niaal reports that he as twice tried to submit a postage stamp request and nothing seemed to happen. Investigated this and it turns out his request was over 2MB in size and PHP disallows posts larger than that. Updated the upload page to check for this error (and others) and notified the user.
• While debugging this I noticed in the apache error logs that there was a poor user who submitted a large postage stamp request on 5/24 and has been constantly querying the data store looking for his results ever since. It turned out that the request faulted parsing and due to a bug in error handling, the request got into a state were I didn't notice the errors. Fixed the problem that caused the error.

Chris Waters