IPP Progress Report for the week 2010.03.08 - 2010.03.12

(Up to IPP Progress Reports)

Eugene Magnier

I spent much of the week preparing for the McLaren? review committee, and recovering. We spent some time this week addressing the storage issues - We have started to compress and delete a chunk of about 13k exposures (some darks, biases, and donuts from before April 2009). Towards the end of the week, I spent some time finishing up the outstanding CR masking and size classification issues in psphot. Over the weekend, we had some trouble with the DVO ingest of 3pi data. This was caused partly by a repeating failure of a single image and exacerbated by DVO lack of support for 64bit file sizes. I started to address the DVO large file size issue, which will probably take until Wednesday. Bill and I are also working out some modifications to the way DVO ingests work to avoid these issues: we will test the ingest of a night's worth of data into a local dvo database, followed by a bulk merge of the dvo databases. There is already a program for doing this bulk merge, though it needs to be tested more carefully. This modification of the ingest process may resolve the long-standing (unaddressed) concern of how to ship DVO databases to the consortium: we may be able to ship the nightly updates and have these merged by the consortium members. Some process control would be needed to guarantee the object IDs are identical (the order matters), but this would be much easier than shipping the full DVO databases.

Heather Flewelling

  • isp:
    • created darks, flats, and shutter corrections
    • various debugging and camera configs required to create detrends
  • ifaps1:
    • we have customers! I spent some time getting them started on accessing our data
    • restarted download client (and added SAS) once the RAID was recovered. 3 disks were dead.
    • copied dvo databases to ifaps1
  • created plots for Ken: more gpc1/isp plots of skybrightness
  • database searches for Ken
  • queued up microtest suite, stacked, diff, magic.
  • queued up M31c
  • calcite cell hunting and gathering for Tonry

Bill Giebink

  • Reduced and plotted DIMM data
  • Ran RoboDIMM Thursday and Friday
  • Recovered DNSM from failure during MECO work
  • Serviced DNSM
  • Put together RoboDIMM data tarball for Eric Bell

Roy Henderson

  • PSPS JHU sandbox testing:
    • Test of big merge - 36 full sized exposures. Successful
    • DXLayer crash when loading 'init' batch: debugged (was requiring min/max object ID)
    • Database table name issue in ODM (photoZRecipe): debugged with Suzanne at JHU
    • Found bug when loading 'init' batch. Issue with VARBINARY types in FITS->csv. Fixed.
    • Attempted to load some diffs. Ready on IPP side, but changes needed at JHU
  • Maui cluster loading:
    • Working with Conrad to load Jim's simulation data on Maui cluster
    • Converted Jim's stack and Object sim data, but problem with detections. Fixed.
    • Problem submitting batches to Maui ODM. Samba mount issue?
  • IPP-side development:
    • Finished off XML configuration work. Now, table descriptions, 'init' data, IPP->PSPS mappings etc all stored as XML for full flexibility

Paul Price

  • Running STS background-preserved processing
  • Cluster dramas
  • Copied AF-approved opposition data for Ocarina.
  • Discovered camera masks were being by psastro without error checking, leading to failures in warp because mask had zero size. Fixed psastro and added check for non-zero size in PS::IPP::Config->file_exists().
  • Attempting to process Microtest suite. Discovered problems with automatic burntooling; reverted nightly_stacks.pro
  • Debugged sudden background changes causing magic masking: bad cells (all pixels zero) weren't being recognised as such. Changed CELL.BAD from -100 to 0 and set MASK.BUILD in ppImage recipe to ensure this 'high' value doesn't damage residuals.
  • Discovered and fixed bug in script that deletes diff convolved images
  • Debugged assumptions on data volume
  • Marked destreaked ThreePi?.nightlyscience and all STS.nightlyscience warps for cleanup
  • Debugged failing magic runs: many had missing convolved diff files, which have been recreated.
  • Reworked PS1_DataRequestManagement page
  • Using own astrometry with MD04 DVO database was fairly straightforward once I discovered the correct command lines and changed the psastro recipe. This work is not continuing because the excess clipping in the stack is caused by a different issue.
  • Discovered and fixed half-pixel bug in pswarp producing bad positions in warp CMF files. Fixing this issue also results in improved stacks because the warp positions are used to produce the fake image that is the target for PSF matching.

Bill Sweeney

  • debugged and fixed problem with magic masking.
  • started working on fixing import of sdss data into dvo
  • monitored dvo database creation. Discovered a problem causing dvo database corruption.
  • fixed some problems with the postage stamp server caused by incorrect user input.
  • hacked the distribution system to rebundle some data that was cleaned before ROE was able to download it
  • various searches for data to clean up.

Chris Waters

  • Observed at UKIRT.
  • Added microtest handler into automation configuration.