IPP Progress Report for the week 2012.07.16 - 2012.07.20

(Up to IPP Progress Reports)

Eugene Magnier

  • updated the psphot call used by ppSub to perform the trail fitting & modified the recipes to invoke trail fitting in DIFF analysis.
  • created a version of psphot that can be applied directly to ppSub output diff images (+ psf models) to test fitting
  • updated the psphot output functions to complete the trail fitting support
  • merged the DIFF and STACK psphot output formats into the autogenerated set used for the other (CHIP) output formats
  • updated the DVO schema to include fields needed by PSPS mean objects: stackFluxPSF, stackFluxKron, and associated errors
  • added code to calculate the distance to the skycell center to use in primary detection selection for stack photometry

Serge Chastel

  • deneb-locate.py: find files on the cluster (limited to the /export/<node>.<disk>/nebulous directories)
  • float and double in C/MS SQL/MySQL explanation to Heather and Conrad
  • added the '-size' option to deneb-locate.py for Mark
  • added the ippb and the stsci nodes to the 'who's using the cluster' page
  • czar on friday

Heather Flewelling

  • ipptopsps
    • discovered why OB has duplicate objids: the regions in skytable.fits overlap
      • updated ipptopsps to cull objects that lie outside of a ra/dec region
      • copied over the older skytable.fits that does not have overlaps
      • redoing all SA_7 Batches for psps (easier to do this than for them to remove bad batches, and SAS is small)
    • ipptopsps proposed schema changes (see bold items in http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ippToPsps_Object)
      • investigations with Gene/Conrad/Serge about size of ra/dec columns. Discovered that DOUBLE in mysql/C/python = FLOAT in ms sql server.
    • new SA7 load
      • problems with ipptopsps: if doing testing (to psps_test) and doing actual (to psps_jhu) at the same time from the same dvo, will run into problems (do not do this).
      • 2 failed with out of bounds obj id
      • rest loaded and became mergeworthy (after conrad restarted some stuff)
  • dvo
    • ThreePi?.V3.2012.Q2 died a painful death:
      • perl script ran out of memory (?) 18 times, and retried the same dvomerge 18 times
      • verified that in the dvodb (in images.dat and in the dvo itself) that there are 18 instances of that minidvodb
      • added (with help of gene) macros in pantasks.pro to not requeue tasks that have unknown_err (the merges did)
      • moved old ThreePi?.V3.2012.Q2 -> moved old ThreePi?.V3.2012.Q2.broken
      • requeued ThreePi?.V3.2012.Q2 and restarted
      • it died the same painful death: one of the minidvodbs is hosed.
    • SAS_v8 started
      • staticsky queued for staticskystage
      • cam stage being built as SAS.20120720
  • ipp
    • found bugs/fixes related to mac compile but not checked in:
      • one in psThread.c - one of the functions wants something returned
      • one in glueforge - reverting to older (r 34079) fixed it

Mark Huber

  • Czar Wednesday and Thursday
    • fixing broken files, looking into load spikes on stsci machines/NFS, monitoring for RAM overuse by ppImage
    • temporarily worked through reallocating nodes for more stack processing to catch up with ppStack fixes
    • while watching IPP processing, also local gofer for transfering specialized JT MD stacks for the PS1-SNIa project
  • ppStack/LAP: finally cornered another bug, originally found troubling the MD10.refstack evaluation that was then also found in the ~1000 of LAP stacks, related to the normalization of the stacks and rejecting the first input image with the tighter FWHM input cuts. Quick fix added, more detailed investigation is still needed into the whole code section. Monitored the LAP stacks to catch up with rest of processing including the forced reprocessing of the earlier faulty set.
  • MD10.refstack skycells finished in a somewhat heterogenous manner of N inputs, criteria and code version. Continuing full field summary to 2011 refstack and one of the best nightly stacks for comparison with the new input cut settings.
  • SAS skycal further tested versus deeper MD stack setting and a full sample run with SAS_v8.

Bill Sweeney

  • Spent this week testing and optimizing the footprint analysis that is performed when culling new detections. Implemented two different recipe changes to deal with NAN pixels (saturated stars, chip and cell boundaries). Found an inefficiency in how the data is organized. Changing the way that pixels are marked improved the performance by about a factor of 4.
  • Changed the cosmic ray masking code to behave better in the case of very large footprints by limiting the size of the region searched to a configurable value (50 x 50 pixels default)
  • Read two PS1 photometry papers.
  • Two days remembering how to be processing czar.

Chris Waters

  • Launched SAS_v8 processing.
  • LAP: Added a far-too-large area to cover the Kepler field, and set up queue for the RA 10hr stripe.
  • Tracked down ppVizPSF to confusion about how it reads command line options. Attempted to resolve this, but ran into trouble with segfaults.
  • Read up on Rauscher, B.J. et al. noise correction algorithm. I'm doubtful we can easily adapt this to our data, and I don't believe it can be applied to historical data.
  • Wrote more on detrend paper.
  • Did a few tests on PSPS using the SAS_v7 data. Objects now seem to accurately reflect the information stored in the Detections table.