IPP Progress Report for the week 2012.03.05 - 2012.03.09

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Updated the Raw_Images_Files_in_Nebulous wiki page

Heather Flewelling

• Designed new way to handle staticsky stage in dvodb and policy changes on minidvodb building.
• Verification of SSP injection (there are ~100 in a funny state due to nebulous), and started working on integrating JT's SSP processing scripts into ipp
• Other stuff I did: finished JT Stacks
• Discovered Staticsky faults quite a lot for multi-filter (sent those to Bill for debugging)
• Built SAS.20120306 and MD04.20120307 for Roy (required queueing staticsky stage for each), turned back on ThreePi and LAP addstars

Roy Henderson

• big improvement to speed of ippToPsps queuing: now pulls everything from gpc1, then figures out pending stuff in each RA/Dec box, rather than querying (slow) gpc1 for same info
• problems with foreign key constraints in DVO tables: found this was due to wrong database engine (defaults to MyISAM). Wrote code to fix
• with the above in place, I added foreign key constraint on detections table for easier purging of unecessary data
• fixed bug when queuing stacks: don't know RA/Dec, so all are queued in one box
• fixed bug in density plotting when running for different batch types
• added this density plot to ipptopsps czartool page
• fixed bug where it would not allow repeated production of IN batches
• fixed-up program for deleting batches from datastore to be consistent with other programs (i.e. has same base-class as all ippToPsps programs)
• IN batches now contain a survey ID in their manifest files. Should be less annoying on the PSPS side
• started testing new (development) version of Jython
• committed database schemas for czar and ipptopsps databases for easy re-creation

• Documentation:
  • began updating main ippToPsps documentation
  • new page describing ippToPsps database, complete with sample SQL

• Objects and Ubercal:
  • meetings with Gene, Jim and Ken
  • updated planned schema changes page on wiki
  • fixed-up PSPS schema parsing script to deal with Objects
  • started wiki page for new Object batch, detailing all known mappings

Mark Huber

Bill Sweeney

• Another week of psphotStack debugging. Updated my script that reads log files and categorizes the failures. Focused on debugging the problems that cause assertion failures or core dumps.
  • Found cause of assertion failure in pmModel_SERSIC (nan value in model)
  • Runs were often failing to find any stamps for psf model fitting. The problem turned out to be an obscure little off by one bug in a little used psModules function.
  • Followed a case where a souce unexpectedly had no pixels associated with it. Turned out it was a child of another source that failed model fit.
  • A number of segvs were occuring because input cmf files were corrupt. (ipp064) Wrote a script to find obvious corruption and to replace the broken file with another instance which is good. The test was to look for a negative IPP_IDET. This seems to have worked for the 20 or so cases that we had
• With the bug fixes above we are now able to run psphotStack on the new stacks with a failure rate of only .4% (Note that the area tested so far is near the southern galactic cap. With older stacks in the galactic plane we get many cases of memory use explosion (> 160GB per process). We still need to do some work there.
• Researched changing the psphotStack output file names to include the stack_id and filter in the file name. It turned out that this is non trivial with the current file rule setup. It will be easy to add a new file rule type.
• Some postage stamp requests got stuck. It turned out that bycoord requests found data from 2008 which cannot be updated correctly. Updated the parser to exclude data prior to 2009-04-01 unless the user explicitly asks for it.

Chris Waters