IPP Progress Report for the week 2012.03.05 - 2012.03.09

(Up to IPP Progress Reports)

Eugene Magnier

  • Updated relastro to work with the parallel dvo database. This was more complicated than expected (needed to update the code a la relphot to use the small internal structures for measurements, carrying only the needed information, to avoid large memory footprints).
  • helped Ken out with the NSF proposal
  • reference database is nearly ready -- large scale tests seem to work well. Initial attempts with setphot & relphot worked fine (run time is about 2hrs, as opposed to 4 days!)
  • Met with Roy, Jim & Ken to plan the ingest of calibrated dvo data into PSPS

Serge Chastel

Heather Flewelling

  • Designed new way to handle staticsky stage in dvodb and policy changes on minidvodb building.
  • Verification of SSP injection (there are ~100 in a funny state due to nebulous), and started working on integrating JT's SSP processing scripts into ipp
  • Other stuff I did: finished JT Stacks
  • Discovered Staticsky faults quite a lot for multi-filter (sent those to Bill for debugging)
  • Built SAS.20120306 and MD04.20120307 for Roy (required queueing staticsky stage for each), turned back on ThreePi and LAP addstars

Roy Henderson

  • big improvement to speed of ippToPsps queuing: now pulls everything from gpc1, then figures out pending stuff in each RA/Dec box, rather than querying (slow) gpc1 for same info
  • problems with foreign key constraints in DVO tables: found this was due to wrong database engine (defaults to MyISAM). Wrote code to fix
  • with the above in place, I added foreign key constraint on detections table for easier purging of unecessary data
  • fixed bug when queuing stacks: don't know RA/Dec, so all are queued in one box
  • fixed bug in density plotting when running for different batch types
  • added this density plot to ipptopsps czartool page
  • fixed bug where it would not allow repeated production of IN batches
  • fixed-up program for deleting batches from datastore to be consistent with other programs (i.e. has same base-class as all ippToPsps programs)
  • IN batches now contain a survey ID in their manifest files. Should be less annoying on the PSPS side
  • started testing new (development) version of Jython
  • committed database schemas for czar and ipptopsps databases for easy re-creation
  • Documentation:
    • began updating main ippToPsps documentation
    • new page describing ippToPsps database, complete with sample SQL
  • Objects and Ubercal:
    • meetings with Gene, Jim and Ken
    • updated planned schema changes page on wiki
    • fixed-up PSPS schema parsing script to deal with Objects
    • started wiki page for new Object batch, detailing all known mappings

Mark Huber

  • ppStack
    • Sample simulated set run for selecting different target PSF.
    • MD04 deepstack test runs with new ppStack setting (SYS.ERR=0) on grizy skycell.055 and 058 finished after debugging fault in intermediate output product testing code. Working through issue with target PSF being set larger than allowed input limits and further review of the test run ongoing with the growth of the convolved stack PSF.
    • LAP sample test runs with new ppStack setting (SYS.ERR=0). Noticed few cases rejecting an additional poorer image, something to keep an eye on, but convolved stack target PSF well matched.
  • QUB liaison/transients: generated sample of sub-stacks and diffims in g,r for Morgan Fraiser on early detection images of PS1-12in.
  • Stumbled into odd memory fault with ppStack on ippc11, reboot seemed to have cleared.

Bill Sweeney

  • Another week of psphotStack debugging. Updated my script that reads log files and categorizes the failures. Focused on debugging the problems that cause assertion failures or core dumps.
    • Found cause of assertion failure in pmModel_SERSIC (nan value in model)
    • Runs were often failing to find any stamps for psf model fitting. The problem turned out to be an obscure little off by one bug in a little used psModules function.
    • Followed a case where a souce unexpectedly had no pixels associated with it. Turned out it was a child of another source that failed model fit.
    • A number of segvs were occuring because input cmf files were corrupt. (ipp064) Wrote a script to find obvious corruption and to replace the broken file with another instance which is good. The test was to look for a negative IPP_IDET. This seems to have worked for the 20 or so cases that we had
  • With the bug fixes above we are now able to run psphotStack on the new stacks with a failure rate of only .4% (Note that the area tested so far is near the southern galactic cap so the source density is low). With older stacks in the galactic plane we get many cases of memory use explosion (> 160GB per process). We still need to do some work there.
  • Researched changing the psphotStack output file names to include the stack_id and filter in the file name. It turned out that this is non trivial with the current file rule setup. It will be easy to add a new file rule type.
  • Some postage stamp requests got stuck. It turned out that bycoord requests found data from 2008 which cannot be updated correctly. Updated the parser to exclude data prior to 2009-04-01 unless the user explicitly asks for it.

Chris Waters

  • Dark model investigations. Looked at the parameter space of our current dark, and identified that it does not fully model the true behavior of the detectors. Looking at the residuals pointed to the dark having two modes, one of which seems to be dominant after 2011-05-01. Constructed a new dark for this mode, finding that it fits the dark structure somewhat better than the current dark. Reran SAS footprint using this new dark and checked that the number of single-detection objects decreased significantly compared to the previous reductions. Results listed at: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model
  • Minor work rearranging processing power.