IPP Progress Report for the week 2010.03.22 - 2010.03.26

(Up to IPP Progress Reports)

Eugene Magnier

I spent much of this week finishing up DVO optimization, both for ingest and for relative photometry and astrometry. These optimization improvements have sped up the relastro processing by a large factor (moving an N2 process to an NlogN process). As a result, I was able to demonstrate a full-scale relastro analysis (4 iterations) for ~7900 exposures in about 10 hours. This is not quite as fast as one might like, but it makes monthly calibrations feasible for the full 3pi.

I also finished up the improvements to the CR masking and source size analysis in psphot, and added the statistics desired for improved assessment of the difference image sources (for distinguishing the dipoles, for example). Before turning on CR masking in the real processing, I need to run a somewhat larger test, but the other features can be used now in the standard processing analysis.

Heather Flewelling

  • returning from Arizona conference 3.22
  • sick on Tuesday 3.23
  • vacation 3.24 - 3.31

Bill Giebink

  • Prepared DIMM data histograms
  • Put together a brief synopsis of what is required to run RoboDIMM from PS2 dome
  • Attended conference Wednesday-Saturday

Roy Henderson

  • PSPS
    • Data loading
      • Results of test loading: looked into errors from test batches: a duplicate object ID and some 'inf' values causing problems
      • Finally got an Object batch loaded, so now we've tested initialisation, stack, detection and object
    • Other
      • Worries about compressed binary data needed for diffs: did some research, met with Conrad and Jim to discuss
      • Started work on modularising DXLayer, specifically the a fits2csv chunk
  • IPP development
    • pspsSchema2xml script now generates C headers with enums for PSPS tables.
    • Started changes to ippToPsps to accommodate new PSPS schema
  • Other stuff
    • Dentist on Thursday

Paul Price

  • Cluster cleanup: destreaked warps, diff convolved images
  • Added 3 kinds of exposure maps to ppStack (number, exposure time, weighted exposure time). Set static BSCALE,BZERO so these are not too large.
  • Updated ppCoords so it can calculate cell coordinates in addition to chip coordinates
  • Set up system to collate microtest magic clusters in cell coordinates; published data to Camera group.
  • Investigating missing skycells in warp-warp diffs:
    • Found that warp-warp diffs don't always include all skycells available.
    • One possible cause is use of '-available' flag with difftool -definewarpwarp. Removed this flag: no longer necessary, and bad for current pipeline operating model. Suspect another cause as well.
    • Almost all warp-warp diffs have been affected, at a typical level of 10% of skycells missing. No, turns out this 10% are edge skycells, ignored due to constraints on good_frac.
    • Warp quality flags are all reasonable, and represent different stages of skycell overlap, from none to some with no sources, to some sources with no good PSF, to good.
    • Bad quality warps extend into the centre --- should have good PSF measurement there, but it's failing.
    • The SMF files are corrupted, with many extensions empty. Dave Monet had told me about this, but I hadn't realised how extensive it is. I have fixed psastro (was not checking return values) previously, so this should be a solved problem, but we're now dealing with its effects. Hopefully, that means that the problem is not so extensive as we thought.
    • Sifting through SMF files to identify bad files.

Bill Sweeney

  • Modified distribution so that destreaked files can be read from an alternate location for raw stage. This allows us to distribute raw files while continuing to keep the originals in place.
  • fixed some problems with raw file destreaking that had crept in as it hasn't been tested lately.
  • distributed M31 raw images from run5c
  • Investigated MPIA's problems with their data store. It turned out that apace wasn't running. We need to clarify what my responsibilities are maintaining their IPP installation. Their experience is that they can't get the IPP built and working reliably without help (That is, without me doing it).
  • extracted certain tables from OTIS and IPP databases that Bertrand Goldman at MPIA is using for some analysis that he is doing.
  • fixed problem with warptool -towarped that I'd introduced. (Fixed it further on March 29). "partial run update" still needs some work.
  • added statistic nondiff_frac to magicDSFile table. This gives the fraction of pixels masked by magic because the file had no overlap with a diff skyfile that was included in the streak detection.

Chris Waters

  • Diskspace: Finish up monitoring raw Imfile compression for engineering data. Will probably need to flag more images for compression soon.
  • Detrends: Wrote script to bundle detrend images for distribution. First set done and posted, and the remainder are being bundled.
  • Processing czar tasks.
  • Detectability: Work to correctly respond to a detectability query from the postage stamp server.