IPP Progress Report for the week 2011.11.28 - 2011.12.02

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

Heather Flewelling

• cooked lots of dvodbs for ORR (multiple iterations of MD04, SAS, SAS.footprint)
• restarted ThreePi + merging
• answered Roy and Jim questions
• development work on addstar to properly handle staticskyRuns with multiple filters
• sent email to Gene about modifications to staticsky to reduce insanity in addstar/psps ingestion

Roy Henderson

• Code changes prior to SAS/MD4 loading:
  • populating un-convolved fluxes in StackApFlx table
  • incorporating filterID into stackDetectID to avoid key violation for multi-filter stacks

• SAS loading:
  • started testing with a test DVO provided by Heather
  • much confusion over stack_id/sky_id/path_base for new stacks in DVO
  • worked on very convoluted way of getting correct cmf file for a given stack_id for new multi-filter stacks
  • had to stop loading SAS and pull all batches from datastore after issue discovered with z-frames
  • restarted loading and completed with new DVO created by Heather

• MD4 loading:
  • dusted-off old code to convert DVO databases to MySQL
  • converted new MD4 DVO to MySQL in 24 hours
  • processing was much faster, but still too slow to load in time for ORR. So...
  • made a database dump of MD4 and restored it to numerous hosts in order to run multiple clients
  • loading completed in 24 hours

• Other:
  • completed loading of RA 6-18 hrs RA for OLD survey, i.e. half the sky
  • very carefully republished all the failed/OnHold batches from a list provided by Conrad
  • investigated issue of large numbers of NULL fluxes in new stacks
  • updated the PSPS news page with plots after merge completion

Mark Huber

• MD: evaluating extent of MD06,08 lost on ipp064.
• staticsky
  • preparation of MD04, SAS.footprint.123, SAS2.123 data sets for loading dvo+PSPS. New reprocessing of MD04 camera exposures destreaked. New re-run of SAS fixed missing footprint stacks but are now missing SAS2.123 stacks.
  • continued testing faults in psphotstack

Bill Sweeney

• Unfortunately Bill was ill this week and did not get much acomplished.
• Wrote some scripts to set up reprocessing of the 3000 or so exposures that got destreaked incorrectly between 11-November and 24-November. This project was hampered by the loss of data in ipp064.
• Made a couple of minor changes to the postage stamp server to support easier selection of stack skycells for get image requests.

Chris Waters

• ipp064: rsynced data from partially recovered raid to ipp065, to allow for some salvage operations.
• SSTF: Ran test sets with improved flagging code.
• Sky/background issue: finished proposed set of changes to PATTERN.ROW/CELL choices. Modified PATTERN.ROW behavior to fit trends in slope and offset as a function of row. This is needed to ensure that PATTERN.ROW does not subtract off sky variations while correcting row-to-row offsets. This seems to improve the background subtraction around these cells in most cases. Checking the magic fraction shows a minor increase, with the original processing averaging 0.027 masking, and the new processing averaging 0.028.
• Diskspace: Due to wave 4 RAID concerns, set all wave 4 nodes to nebulous repair, blocked the targetting shuffle from moving data to these nodes, and added the stare nodes to nebulous and updated summit copy to properly target these hosts for stare data.