IPP Progress Report for the week 2011.12.05 - 2011.12.09

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Sick on Friday
• Condor-IPP: Documentation, code quality, more tests.

Heather Flewelling

• czar czar czar czar!
• dvodb/addstar testing
  • tests of dvomerging ipp004 minidvodbs to ipp005 bigdvodb, from ipp005. dvomerge is about a factor of 2 faster, and the other parts (relphot/resort/verify) are much slower.
  • tests of rsyncing bigdvodb from ipp005 to ipp004 after each minidvodb merge. It takes roughly the same amount of time as the merge.
  • these seem to scale a bit with number of stars
• dvodb/addstar coding: working on additional tables/tools/tasks/scripts to do the following:
  • split the dvodb tasks (make verify/dvomerge a separate task from resort/relphot) to make it faster to merge/relphot on different computers, to make it easier to merge multiple things together (multiple times), and to consistently make rsync copies of the big dvodb

Roy Henderson

• Sick for 3.5 days

• PSVO
  • changes to schema groups broke login stage: Fixed
  • changes to types in DRL SOAP response broke query result parsing: Fixed
  • released new jar and Mac versions

• monitored loading of SAS and MD4
• restarted processing of OLD survey as new frames are being merged into DVO

• PS1SC mailing-list user pruning: removed lots of bouncing email addresses.

Mark Huber

• back to tracing and testing ppStack/ppSub for understanding the overly large PSF in convolved stacks with large number of inputs

Bill Sweeney

• Made changes to the psastro recipe to enable it's use in the new static sky calibration stage. Ran some tests and started thinking about the implementation.
• wrote scripts to recover distribution bundles lost due to disk outages. Attempted to recover the M31 raw data sets but in the end there were too many errors. Will rebuild from scratch.
• postage stamp server was busy. Still spending a lot of time stopping requests from getting stuck due to inconsisent destreak states and lost files. This problem will mostly go away now that we can release data without magic censoring.
• modified postage stamp server to allow inputs for stack get image jobs to be selected by skycell_id. This makes getting lap stack images easier.

Chris Waters

• ipp064: finished recovery scan of rsynced data. Added new nebulous volume, and reinserted the salvaged data into that new volume. This is still hosted on ipp065 until we get word that ipp064 has been reformatted and is ready to serve data again. It appears that ~80% of all files that could be rsynced were salvaged, although this doesn't clarify whether we can easily resume processing for runs that were partially lost.
• Detectibility: documented how the detectability server code works, and pointed out areas where I suspect the faults keeping it from working correctly are.
• LAP: resumed LAP processing, although the loss of ipp064 has caused a number of delays and problems. I'm resolving them as I find them, mostly by dropping exposures and stacks from the run.
• SSTF: relaunched diff test set using 4sigma photometry and the new set of flags.
• Sky/pattern: Ran M31 data through magic, and had a very large number of streaks. It looks like this was caused by attempting to deconvolve the reference stack to match the better seeing of the input warps. Checking the detection counts shows only a minor change between the original processing and my new processing with the improved pattern correction lists.
• Diskspace: Attempting to resolve the diskspace data pointed out an overly greedy regexp in the script that scans the data. This has caused the camera stage to assume all cmf files were part of it. I think this only is a problem for this stage, but it makes coming up with a final answer somewhat difficult. I have finalized a "best guess" mapping file that appears to give reasonable results for most stages. This suggests that we have a good guess as to what is using the disk space.