IPP Progress Report for the week 2011.07.18 - 2011.07.22

(Up to IPP Progress Reports)

Eugene Magnier

We learned early last week that the perl parsing of the camera config files was extremely slow. I updated ppConfigDump to take a set of requested values and updated the scripts to use a call to ppConfigDump to replace the full perl parsing. This was a big help. We have had several days over the past week with 800-1000 exposures processed in 24h. We have mostly done a better job keeping up with the processing load.

I spent a chunk of last week trying to understand our storage usage. We have been very tight on storage over the past month despite moving lots of raw data to the ATRC machines. Looking back on the past 6 months, it seemed that the rate of storage usage was higher than we expected. Looking at the output products from specific exposures in the full and cleaned states (with and without the pixel products), I realized that we were leaving more bytes behind in the 'cleaned' stages than expected. The current total per processed and cleaned exposure is between 30-50% of the raw data size. There are a couple of products which can be removed on cleanup : the 'pattern' data is generated on update and can be removed. We also leave behind the uncensored camera mask because we do not have a camera 'cleanup' stage. Finally, the output jpegs for the diff stage are very large : they are apparently binned 4x4 instead of 16x16 like the other stages. Addressing these issues can bring the size of a cleaned processed exposure to ~15-20% of a raw exposure. We may also be able to make a more compact storage of the censored and uncensored camera stage smf files, but that will take a bit more work. The other reason storage has been very tight is that the LAP processing has been queuing too many exposures without pushing them all to completion and cleanup. These fixes, and cleaning up the past LAP data got us back into a comfortable place with 10% of the cluster free at MHPCC.

Serge Chastel

  • MOPS czaring at IPP level
  • Set SVN for operational files
  • Helped Heather with nfs and eventually started to write a new tool to replace(?) nebdiskd.

Heather Flewelling

  • czar (fixing faults, etc)
    • problems with publish - too many faults to allow pub to finish - the faults were all 'old' data that had been cleaned up
  • transferred SAS stacks to TD.
  • generated plots of times of different stages (and differnt tasks in those stages) for ThreePi?.%. This showed that there was a problem in chip stage (slowdown), which Gene found was caused by the mdc parser.
  • talked to Jim about John's stacks
  • generated plots for # of streaks (by filter and data_group) for ThreePi? data. streaks are improved since code changes in April.
  • investigated problem in diffs - found bad astrometry not getting caught by psastro. Gene is investigating...
  • answered questions to roy about gpc1 database (creating a mock one from scratch). There is some problems generating 'my' minidvodb tables, I need to fix this.

Roy Henderson

  • Only worked two days this week (vacation for three).
  • IPP to PSPS test loop:
    • set-up ippToPsps (installing Java, Jython, scratch MySQL database etc) on development cluster
    • needed to add some tables and columns to test-gpc1 database (with Heather's help)
    • inserted enough metadata to test-gpc1 about a single camera exposure to run ippToPsps
    • code changes to get IN (initialization) batch working with new Batch class
    • missing primary header values in test smf files. Changed interface code to ignore these and proceed, with errors added to log
    • missing SOURCEID and IMAGEID in .hdr extension: can't interface with DVO without these
  • tweaks to czartool after big changes last week
  • created PSPS wiki 'News and Updates' page, linked to from PSI
  • added some new plug-in queries to PSVO for stacks

Mark Huber

  • MD10.V3 work continued, thanks to everyone's help reference stacks have been made, enabled, and distributed to the datastore.
    • reference stacks for yz (higher background) to be available for the first observations of the field at the start of the OC, then gri after Chris implemented the new Dark model. Summary details to be added to wiki page.
    • reprocessing of MD10 exposures through warp of a "better" sub-set of exposures necessary to make reference stacks is available for reprocessing of nightly stacks. The yz filters are complete, gri have ~100 each remain to process.
    • SSdiff testing with new refstacks and automatic choice for convolution direction. Additional testing seems to be necessary.
    • Discussed options for setup of new MD10 for photpipe stack+diffims for transients.
  • Further exercise enabling the new RINGS.V3 skycells for 3PI observations and monitoring the nightly processing.
  • simtest IPP to PSPS test sample of camera SMFs, stack CMFs, and generated DVO catalogs produced for Roy. Looking into the configuration to recover the missing parts.
  • Full Czar experience on Friday with lots of help.

Bill Sweeney

Chris Waters

  • Dark: Identified that the previous quadratic temperature model was established for a different (warmer) temperature regime. The current temperature range appears to be in the nearly linear tail of an exponential. This helps explain why fitting with a linear model was yielding less noisy data: a quadratic is generally unjustified by the data. Constructed a new full exposure dark model that uses at most a linear temperature dependence, and confirmed that it introduced less noise than the current dark (~2 counts compared with ~8 counts), as well as being successfully verified against random dark data. Promoted this dark to the currently used model.
  • Offnight diffs: These do not appear to be generated at the moment, and it looks like this will require a day or so to fully debug.
  • LAP processing: Queued SAS2.123 and SAS.footprint for gri using the new dark. Dropped LAP/14hr y-data to free diskspace and to prepare for a jump to 16hr.
  • Diskspace: Rewrote diskspace accounting script to account for diskspace better. Queued chip-stage pattern files for cleanup, as these had not been cleaned before. As they are regenerated by the update process, there is no reason to retain them.
  • LAP code: Began rewrite of LAP code to deprecate quickstacks in order to speed up processing.