IPP Progress Report for the week 2011.12.05 - 2011.12.09

(Up to IPP Progress Reports)

Eugene Magnier

  • fixed the OFF_CHIP bug introduced recently (flag was raised for all detections due to a mismatch of images)
  • psphotStack was failing to measure the photometry for the matched positions. This was due to the models not being generated for the matched sources, but it also led me to an issue with the matching process : the match radius was too small, so many faint objects were not correctly matched. Some objects in the wings of bright stars were also likely to be mis-matched (or result in matches that probably should not be generated). I addressed this by bumping up the match radius somewhat.
  • Extensive communications with Doug trying to understand the faint end non-linearity in his analysis of the MD photometry. He has updated his model to include a constant flux offset for all objects in a given exposure (if, eg, the sky were off, or something equivalent). This offset scales with FWHM2 as one would expect if the sky were off. The odd thing is that the offset is such that all fields and all filters have nearly the same offset in DN per pixel. In other words, if this is an error in the sky subtraction (over-subtracting the sky), the error that is made is always the same amount, regardless of the sky flux and the crowding. This seems surprising. I ran a set of tests with modifications to the sky analysis (using the algorithm which is more robust, but possibly more sensitive to crowding) to see if this affects the results.

Serge Chastel

  • Sick on Friday
  • Condor-IPP: Documentation, code quality, more tests.

Heather Flewelling

  • czar czar czar czar!
  • dvodb/addstar testing
    • tests of dvomerging ipp004 minidvodbs to ipp005 bigdvodb, from ipp005. dvomerge is about a factor of 2 faster, and the other parts (relphot/resort/verify) are much slower.
    • tests of rsyncing bigdvodb from ipp005 to ipp004 after each minidvodb merge. It takes roughly the same amount of time as the merge.
    • these seem to scale a bit with number of stars
  • dvodb/addstar coding: working on additional tables/tools/tasks/scripts to do the following:
    • split the dvodb tasks (make verify/dvomerge a separate task from resort/relphot) to make it faster to merge/relphot on different computers, to make it easier to merge multiple things together (multiple times), and to consistently make rsync copies of the big dvodb

Roy Henderson

  • Sick for 3.5 days
  • PSVO
    • changes to schema groups broke login stage: Fixed
    • changes to types in DRL SOAP response broke query result parsing: Fixed
    • released new jar and Mac versions
  • monitored loading of SAS and MD4
  • restarted processing of OLD survey as new frames are being merged into DVO
  • PS1SC mailing-list user pruning: removed lots of bouncing email addresses.

Mark Huber

  • back to tracing and testing ppStack/ppSub for understanding the overly large PSF in convolved stacks with large number of inputs

Bill Sweeney

  • Made changes to the psastro recipe to enable it's use in the new static sky calibration stage. Ran some tests and started thinking about the implementation.
  • wrote scripts to recover distribution bundles lost due to disk outages. Attempted to recover the M31 raw data sets but in the end there were too many errors. Will rebuild from scratch.
  • postage stamp server was busy. Still spending a lot of time stopping requests from getting stuck due to inconsisent destreak states and lost files. This problem will mostly go away now that we can release data without magic censoring.
  • modified postage stamp server to allow inputs for stack get image jobs to be selected by skycell_id. This makes getting lap stack images easier.

Chris Waters

  • ipp064: finished recovery scan of rsynced data. Added new nebulous volume, and reinserted the salvaged data into that new volume. This is still hosted on ipp065 until we get word that ipp064 has been reformatted and is ready to serve data again. It appears that ~80% of all files that could be rsynced were salvaged, although this doesn't clarify whether we can easily resume processing for runs that were partially lost.
  • Detectibility: documented how the detectability server code works, and pointed out areas where I suspect the faults keeping it from working correctly are.
  • LAP: resumed LAP processing, although the loss of ipp064 has caused a number of delays and problems. I'm resolving them as I find them, mostly by dropping exposures and stacks from the run.
  • SSTF: relaunched diff test set using 4sigma photometry and the new set of flags.
  • Sky/pattern: Ran M31 data through magic, and had a very large number of streaks. It looks like this was caused by attempting to deconvolve the reference stack to match the better seeing of the input warps. Checking the detection counts shows only a minor change between the original processing and my new processing with the improved pattern correction lists.
  • Diskspace: Attempting to resolve the diskspace data pointed out an overly greedy regexp in the script that scans the data. This has caused the camera stage to assume all cmf files were part of it. I think this only is a problem for this stage, but it makes coming up with a final answer somewhat difficult. I have finalized a "best guess" mapping file that appears to give reasonable results for most stages. This suggests that we have a good guess as to what is using the disk space.