IPP Progress Report for the week 2010.06.14 - 2010.06.18

(Up to IPP Progress Reports)

Eugene Magnier

I used the newly generated MD04 astrometry and photometry reference database to re-process all of the MD04 data taken since Fall 2009. I selected a 'good' sample of images (good photometric conditions and good seeing) and made an initial deep stack in all filters. The write-up on this analysis is available on the wiki as PS1_MD04_RefStack. With the internal reference photometry and astrometry database, we get much more consistent zero points from the on-the-fly analysis and much better astrometry: the astrometry seems to be in the range of 15mas, consistent with other tests. The photometry is less clear because we need to check for the correct airmass extinction terms. Zero points seem to be cluster within about 1% of the nominal zero points, but there are some outliers which we need to understand (the bright outliers are more surprising than the faint outliers). There is also a problem with the exposure time map produced by the stacking since these deep stacks over run the dynamic range of the output files as defined for the standard processing. I made this set available to the consortium, but I'll be doing some more re-processing of this data, including a version with 0.258 arcsec static sky pixels.

I also spent some time last week improving the dvomerge processing speed, and fixing some further bugs discovered by Heather. This system is now working, and the 3pi data since the start of demo month are now being merged. Heather rsynced the mini dvo dbs to our rsync server, and I have pointed the consortium at that data.

Serge Chastel

  • ippMonitor
    • histograms plots
    • scatter plots (x vs y) / regression line
  • ohana
    • fixed minor inconsistency in create function

Heather Flewelling

  • minidvodbs
    • debugged minidvodb merge step, reverted, and successfully merged ThreePi.V1 a dozen or so times
    • halted @ minidvodb_id 138 (ipp004 was at 99% full)
    • rsynced minidvodbs to ipp022
    • rsynced masterdvodb to ipp037
    • restarted merging (only) to ipp037
  • ippMonitor
    • fixing bugs that CZW found
    • working on detrend verification histograms
    • while working on this, found query that worked on gpc1 database but not on ipp001 gpc1_0 database (Bill investigated this).
  • czar on wednesday and thursday

Roy Henderson

  • Build issues with FitsConverter on Maui cluster. Usual suspect: cfitsio. Fixed makefile. (PSPS)
  • Changes to the way I publish batches to the datastore to match up with Conrad's new DXLayer (IPP)
  • a fair bit of work to correct the exposure times in the PSPS Db:
    • hacked a version of my ippToPsps code to run through all currently loaded exposures and pull out exposure times then write SQL updates to scripts (IPP)
    • checked these SQL scripts into svn for access from SSMS (PSPS)
    • ran each script (one for FrameMeta, one for Detection) on each 'cold' Db on each of the six servers (very slow and frustrating) (PSPS)
  • Off sick for three days

Paul Price

  • Have psAbort() and psAssert() log a backtrace before bailing: attempting to track down thread bug
  • MOPS support:
    • Moved current IPP-MOPS DataStore to IPP-MOPS-20100614
    • Reverted ppMops in branch ipp-20100610 to produce 'new old' format (with columns for anti-dipole stats, but not SMF-like)
    • Turned off automated cleanup to keep as many diffs as possible for re-photometry
    • Added more search options to pubtool -definerun
    • Republishing all SweetSpot and 3pi diffs since 2010-02-01: pubtool -definerun -rerun -client_id 1 -data_group ThreePi.% -data_group SweetSpot.% -dateobs_begin 2010-02-01 -dbname gpc1
    • Added new analysis task: diffphot
    • Added feature to ppArith: binary operation with a constant instead of an image
    • Fixed pubtool definerun and pending queries so they can find cleaned diffs
    • Re-republishing all SweetSpot and 3pi diffs since 2010-02-01 (now including the cleaned ones): pubtool -definerun -rerun -client_id 1 -data_group ThreePi.% -data_group SweetSpot.% -dateobs_begin 2010-02-01 -set_label mops.20100616 -dbname gpc1
    • Queued 5293 exposures (~30k skycells) for diffphot: diffphottool -definerun -set_workdir neb://@HOST@.0/gpc1/mops.20100617/ -set_label mops.20100617 -set_note 'Re-processing diff photometry for MOPS' -data_group ThreePi.% -data_group SweetSpot.% -dateobs_begin 2010-02-01 -dbname gpc1
    • Approx 10% of diffphots fail (fault=4) because diff PSF doesn't include clump parameters
    • Realised diffphot doesn't respect magic
    • Preparing to publish results
  • Fixed small coordinate bug in pmPSFEnvelope
  • Excessively large values in database dumps
    • MySQL bug: values of DBL_MAX in database are dumped with insufficient precision and rounded outside of legal bounds
    • Added code to psLib to catch these and truncate to FLT_MAX
    • Identified 52 instances in database; Bill fixed
  • Fixed dsproductls: MOPS' DataStore doesn't include the comment header, so removed overzealous checking that content is non-empty
  • Discovered and fixed (I hope) race condition in threading system
  • Fixed psMessageDestination to append log/trace messages (finally!)

Bill Sweeney

  • On Vacation

Chris Waters

  • Finished, committed, merged, and did final debugging on the warp/stack/diff summary tools, and on the stack association code.
  • Nebulous work
    • Wrote code to migrate instances off of a specified volume
    • Wrote API to find which instances are on a given volume
    • Looked into the distribution of slow database queries, with the goal of shifting the timeout from 1s to 2s (although 3 or 4 wouldn't be excessive).
  • Processing czar, with a focus on queuing old data for cleanup.
  • Made mask stats summary page, showing some statistics about the masking fraction at various processing stages.
  • see: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Mask_Stats_Week_One