IPP Progress Report for the week 2010.06.14 - 2010.06.18

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• ippMonitor
  • histograms plots
  • scatter plots (x vs y) / regression line
• ohana
  • fixed minor inconsistency in create function

Heather Flewelling

• minidvodbs
  • debugged minidvodb merge step, reverted, and successfully merged ThreePi.V1 a dozen or so times
  • halted @ minidvodb_id 138 (ipp004 was at 99% full)
  • rsynced minidvodbs to ipp022
  • rsynced masterdvodb to ipp037
  • restarted merging (only) to ipp037
• ippMonitor
  • fixing bugs that CZW found
  • working on detrend verification histograms
  • while working on this, found query that worked on gpc1 database but not on ipp001 gpc1_0 database (Bill investigated this).
• czar on wednesday and thursday

Bill Giebink

Roy Henderson

• Build issues with FitsConverter on Maui cluster. Usual suspect: cfitsio. Fixed makefile. (PSPS)
• Changes to the way I publish batches to the datastore to match up with Conrad's new DXLayer (IPP)
• a fair bit of work to correct the exposure times in the PSPS Db:
  • hacked a version of my ippToPsps code to run through all currently loaded exposures and pull out exposure times then write SQL updates to scripts (IPP)
  • checked these SQL scripts into svn for access from SSMS (PSPS)
  • ran each script (one for FrameMeta, one for Detection) on each 'cold' Db on each of the six servers (very slow and frustrating) (PSPS)
• Off sick for three days

Paul Price

• Have psAbort() and psAssert() log a backtrace before bailing: attempting to track down thread bug
• MOPS support:
  • Moved current IPP-MOPS DataStore to IPP-MOPS-20100614
  • Reverted ppMops in branch ipp-20100610 to produce 'new old' format (with columns for anti-dipole stats, but not SMF-like)
  • Turned off automated cleanup to keep as many diffs as possible for re-photometry
  • Added more search options to pubtool -definerun
  • Republishing all SweetSpot and 3pi diffs since 2010-02-01: pubtool -definerun -rerun -client_id 1 -data_group ThreePi.% -data_group SweetSpot.% -dateobs_begin 2010-02-01 -dbname gpc1
  • Added new analysis task: diffphot
  • Added feature to ppArith: binary operation with a constant instead of an image
  • Fixed pubtool definerun and pending queries so they can find cleaned diffs
  • Re-republishing all SweetSpot and 3pi diffs since 2010-02-01 (now including the cleaned ones): pubtool -definerun -rerun -client_id 1 -data_group ThreePi.% -data_group SweetSpot.% -dateobs_begin 2010-02-01 -set_label mops.20100616 -dbname gpc1
  • Queued 5293 exposures (~30k skycells) for diffphot: diffphottool -definerun -set_workdir neb://@HOST@.0/gpc1/mops.20100617/ -set_label mops.20100617 -set_note 'Re-processing diff photometry for MOPS' -data_group ThreePi.% -data_group SweetSpot.% -dateobs_begin 2010-02-01 -dbname gpc1
  • Approx 10% of diffphots fail (fault=4) because diff PSF doesn't include clump parameters
  • Realised diffphot doesn't respect magic
  • Preparing to publish results
• Fixed small coordinate bug in pmPSFEnvelope
• Excessively large values in database dumps
  • MySQL bug: values of DBL_MAX in database are dumped with insufficient precision and rounded outside of legal bounds
  • Added code to psLib to catch these and truncate to FLT_MAX
  • Identified 52 instances in database; Bill fixed
• Fixed dsproductls: MOPS' DataStore doesn't include the comment header, so removed overzealous checking that content is non-empty
• Discovered and fixed (I hope) race condition in threading system
• Fixed psMessageDestination to append log/trace messages (finally!)

Bill Sweeney

Chris Waters

• Finished, committed, merged, and did final debugging on the warp/stack/diff summary tools, and on the stack association code.
• Nebulous work
  • Wrote code to migrate instances off of a specified volume
  • Wrote API to find which instances are on a given volume
  • Looked into the distribution of slow database queries, with the goal of shifting the timeout from 1s to 2s (although 3 or 4 wouldn't be excessive).
• Processing czar, with a focus on queuing old data for cleanup.
• Made mask stats summary page, showing some statistics about the masking fraction at various processing stages. ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Mask_Stats_Week_One