IPP Progress Report for the week 2012.05.21 - 2012.05.25

(Up to IPP Progress Reports)

Eugene Magnier

  • ran into some problems with the big parallel dvomerge tests. local I/O hangups (possibly due to other operations on the same machine) caused the table writes to fail in some cases. New code in dvomerge mostly kept this from being a problem: a backup copy was kept that could be used for recovery. But it was a bit tricky to tell which dvomerge succeeded where. I added more tracking in the table headers of prior merges, and also enabled a replace option in dvomerge. A few files on ipp053 failed in a way which was not caught by the file system, breaking 3 or 4 tables. Fortunately, these could be recovered from the snap-shot generated 2012.05.03 (and re-merged with WISE and SuperCOS). Bottom line of the tests: large parallel dvomerge operations can succeed, but we need to carefully check for failed tables and re-merge them if needed. One nice feature is the dvomerge recovery method (in which previously merged tables are automatically detected) worked great.
  • discovered a problem with the WISE public release tables: about 8000 objects had an undocumented value for the photometric quality. I had to re-create the WISE ddvo database, and use the -replace option with dvomerge. I sent a bug report to Roc Cutri, who has since added that field to the documentation.
  • I created a script to query the big DVO database and assemble g-i, i-[J,H,K] color-color diagrams, and then fit for the stellar locus in this diagram. The goal was to check on the synthetic photometry comparison issues found earlier. For better or worse, the large effect that I had previously suspected was from 2MASS was not present. I tracked it back to the influence of USNO-B on the synthetic photometry. Despite the low weight, the large lever arm of the USNO-B photometry meant that the extinction was driven largely by that photometry, pushing around the predicted grizy mags. The large-scale pattern visible in that comparison (PS1_i_ vs Synth_i_) corresponds in fact to the POSS plate footprints. There is an effect from 2MASS calibrations visible in the g-i,i-H and g-i,i-K plots, but it is at a much lower level than I earlier expected. Those plots suggest primarily K-band calibration issues at the 2-4% level, not too inconsistent with the 2MASS claims.

Serge Chastel

  • Off on Mon and Tue
  • Still working on files monitoring.

Heather Flewelling

Almost entirely a week of PSPS:

  • liason - yanked raw files for Craig
  • PSPS
    • investigation why stackDetectionFull had funky numbers (Nigel noticed this)
    • found at least one of the bugs: if stack mag = null then set psps instFlux = ipp instFluxSig *2. There was a missing factor of exposure time, causing the limiting mags to all be around 16th.
    • lots of querying PSI to investigate SA5/SA6.
    • made SA6 with bug fix
    • more sorting out of PSPS accounts I need

Roy Henderson

  • lots of long emails for PSPS-help explaining the use of infoFlag
  • added section to PSPS FAQ regarding above

Mark Huber

Bill Sweeney

  • Continued working on psphot. Attempted to test the new kron code psphotStack on the SAStest.v4 stacks and found all sorts of problems. At about the same time PSPS users reported insane magnitudes in the corresponding data there. Tracked this down to be a difference between the background levels in the raw (unconvolved) stacks and the convolved stacks. Fixed that by modeling the background separately.
  • Found that the kron radii measured with the convolved stacks were unreasonably large. Reevaluated the results of the investigation reported last week and changed the recipe values to remove the window and weight function.
  • Investigated why psphotStack was still getting duplicate detections during pass 2 even with the footprint fix reported last week. The problem was with the management of the data inside psphotStack. Fixed.
  • Wrote a tool to extract the extended source parameters from our cmf files and to read them into mana. Used this to evaluate the extended source measurements on one of the SAS stacks. Reported to DRAVG. http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG_telecon#psphotStack_photometry
  • fixed data store "feature" (now considered a bug) where if the last_fileset argument to a data store product list function is not found the whole list is returned. This is not part of the spec but is what conductor does. Unfortunately it caused chaos with the postage stamp server.

Chris Waters

  • Constructed new mask files that remove static issues due to crosstalk bleeds/etc. If these need to be rolled back for some reason, I have a script located in /data/ippc18.0/home/watersc1/static_masks.20120521/register.abort that resets the det_type to MASKTEST, moving them out of the way of processing.
  • Cleaned up LAP stacks from the distribution server.
  • Re-enabled shuffle to move data to the ATRC.