IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

I have finished my own testing of the integration of the ubercal zero points and flat-field values into the DVO system. The program 'setphot' loads the zero points and imports the flat-field corrections into an internal table, and sets the images to use the zero points and applies the zero points and flat-field values to the detections. The program 'relphot' determines relative photometry for the non-ubercal images using all overlaps, keeping the ubercal image zero points fixed. The test suite generates a fake set of images, zero points and flat-field corrections, ingests into dvo, then checks that setphot and relphot get the expected answers. This all worked quite well (after fixing bugs). I then applied the ubercal corrections to a subset of the 3pi dvo database, and extracted a sample of detections after the corrections were applied. I sent those off to Eddie for comparison to the ubercal'ed detections. The results look mostly quite good (to floating point precision), but there are a small number of ~0.5mmag discrepancies that we need to track down.

Relatedly, Heather reported that the 3pi dvo db merges have caught up with current data, so I have rsynced a copy of the database. I will use that snapshot for the generation of the reference database with the ubercal zero points.

Serge Chastel

  • Nebulous partitioning
  • Triggers to monitor nebulous shuffling activity

Heather Flewelling

  • jt stacks
  • psphot tests for ken
  • rsyncing LAP/ThreePi
    • ThreePi? is merged (minus a few minidvodbs), Gene is making a copy
  • helped Roy with queries for smf grabs for dvodbs

Roy Henderson

This week I began to tackle the problem of slow DVO access in regions of the sky where we have lots of coverage, such as the medium deep fields. In these regions the underlying DVO FITS files are huge and cause significant slow-down in loading to PSPS because the current method of data retrieval results in repeated access to the same files. In the past I have overcome this issue by ingesting entire DVO databases into a MySQL database. This is fine for a medium deep field, but impractical for a full-sky DVO that may, in the future, also contain MD fields. So, a more generic solution is required whereby I can pre-load a region of sky from DVO into a database, process all the camera and stack catalog files for that region, then move on to another region. For optimal efficiency, it's important that when moving onto a new bordering region I do not discard already-ingested data that may overlap both regions. It is also necessary to take file modified dates into account, a bit like rsync, so that recently changed files are re-ingested and unchanged ones are not discarded unnecessarily.

Since I already had code to load entire DVO databases into MySQL for faster access I adapted it to ingest a portion at a time, defined by a box of RA/Dec. Steps included:

  • code to load SkyTable.fits table into MySQL database
  • code to query the above table and get region files contained in RA/Dec box (could be whole sky, or anything smaller)
  • new Db table to maintain list of DVO files imported, including modified date and size and an index number (so we can delete associated detections later)
  • method to determine the total file-size of a DVO region in order to decide optimal time to load
  • rsync-style system that checks file modified dates so that we don't re-ingest unchanged files or ignore recently modified ones
  • process can be stopped and restarted as it remembers what has and hasn't been ingested
  • implemented a method to delete a DVO region from the Db, but retain Images.dat and SkyTable from DVO (to save time)

Future work will be an algorithm to decide which region of the sky is optimal to load next, i.e. figuring out the size available smf/cmf files versus the size on disk in DVO - there is no point ingesting 30GB of DVO data if I only have 2 smf files to load to PSPS.

  • Other work:
    • made changes to gpc1 queries for getting stuff in DVO to reflect changes made by Heather a few weeks ago
    • some email help for Kent Wood regarding use of fast rather than slow queue
    • some mailing list admin including a new list for STSCI

Mark Huber

  • Primarily worked on issues with ppSub and started writing up details. MD04 test set difference images suggest not entirely a need for the need of a convolution direction choice but rather something odd may be happening with the convolution itself.
  • Sick on Tuesday.

Bill Sweeney

  • Reworked the detectability query scripts to work properly in the case of inputs needing to be regenerated. Tested with stack stage for the first time. Fixed a bug in psphotForced. Deployed to the operational postage stamp server system.
  • continued to work with MPG to insure that they have downloaded all M31 data from 2010-2011. Only 46 exposures left to go.
  • Monitored 5 filter staticsky processing. We are getting failures in about 10% of the skycells. Needs further investigation.
  • Made some minor modifications to the cleanup process to work more smoothly in the case of errors.
  • two busy days as processing czar

Chris Waters

  • Background continuity: Spent most of the week sorting through mathematical problems that were preventing the continuity code from generating plausible results. Eventually solved these problems, and ran g-band footprint stacks again including this correction. The results are mixed, with some OTAs working cleanly, and others showing introduced edge features. This still isn't working enough to be moved into the tag at this point.
  • LAP: monitored lap processing, and kicked it as necessary to keep things moving.
  • Began work on automating MOPS WS diffs and offnight diffs.