IPP Progress Report for the week 2010.12.06 - 2010.12.10

(Up to IPP Progress Reports)

Eugene Magnier

I have been working on psphotStack, the program which performs the photometry and morphology analysis of the stacks. The output from this program is the main item required to finish off the IPP->PSPS interface for the stacks. There have been a number of pieces of metadata missing that are needed by PSPS. To get that metadata in place, I needed to modify ppStack as well, to pass the information from the input warps and the analysis to the stack, so that psphotStack has the needed data. There were also a couple of implementation issues that needed to be resolved. First, the original implementation did not include the possibility of having more than one set of PSF-matched images (for the radial aperture fluxes). I realized that additional PSF-matched images can be generated from the first with a simple 1D Gaussian convolution with a pre-determined size (because the initial match generates an image with a known PSF size which is more-or-less Gaussian). I update the code to enable additional smoothing passes followed by additional radial aperture photometry passes. I have also modified the output files to carry all of these measurements.

The other implementation problem was a confusion between the stack photometry analysis needed generally and the specific issue of matched-PSF radial photometry. I had originally coded the analysis to perform all stack photometry analyses (detection, PSF generation, source classification, psf and extended model fits, petrosian profiles, etc) on the PSF-matched images. Instead we want only the radial photometry to be performed on the PSF-matched images and all other analyses to be done on the input stack images (optionally the convolved or unconvolved versions). I re-worked the analysis to this end.

Serge Chastel

  • OC133 OSS data transfer is finished. ipp005 is now clean and relevant data are on mops01.

Warps: On the MOPS side, the warp files are located in /data/mops01.0/133.OSS/warps.oc133. They were individually md5-verified. On IPP side, the corresponding files were removed from ipp005 (precisely from /data/ipp005.0/heather/oss/warp, +1.4TB). The warps files are grouped in directories whose name is an exposure (e.g.: warps.oc133/o5500g0260o). Each exposure directory contains the skycells files derived from that exposure. For instance, warps.oc133/o5500g0260o is associated to 86 skycells (1058.069, [...], 1059.123, [...], 1148.024). There are 3 files for each skycell: cmf, fits, mask.fits (e.g. warps.oc133/o5500g0260o/o5500g0260o.248170.wrp.130510.skycell.1148.003.cmf, warps.oc133/o5500g0260o/o5500g0260o.248170.wrp.130510.skycell.1148.003.fits, warps.oc133/o5500g0260o/o5500g0260o.248170.wrp.130510.skycell.1148.003.mask.fits).

Diffs: All files were transferred to /data/mops01.0/133.OSS/diffs.occ133. They were individually md5-verified. The diffs are grouped in directories which names are skycells (e.g. diffs.occ133/skycell.1148.003). Each skycell directory contains a varying number of files according to the diffs (but that number is multiple of 10). There are 5 files (cmf, inv.cmf, fits, mask.fits, inv.mask.fits) for both the unmagicked-nondestreaked and the magicked-destreaked diff (hence 5*2 = 10 files). Unmagicked-nondestreaked filenames contain the SR_ string while magicked-destreaked do not. For instance (from diffs.occ133/skycell.1148.003/), you can find the magicked-destreaked 551743718.gpc1:OC133.OSS.20101117:RINGS.V0:skycell.1148.003:RINGS.V0.skycell.1148.003.dif.93864.fits and the unmagicked-nondestreaked 555780169.gpc1:OC133.OSS.20101117:RINGS.V0:skycell.1148.003:SR_RINGS.V0.skycell.1148.003.dif.93864.fits. Note also that unfortunately the files do not have the numeric prefix (551743718 and 555780169).

I copied/pasted this e-mail in the /data/mops01.0/133.OSS/README file.

  • Fixed publish scripts
  • Database replication: nebulous is now replicated on ippdb02. But a mysql server can be the salve of only one master. It means that it is not possible to duplicate gpc1 and ippRequestServer on ippdb02.
  • Helped Heather for static mask

Heather Flewelling

  • czar for 2 days
  • worked on creating a way to autogenerate most of the static masks - Serge was very helpful on the algorithm for the masking.
  • worked on various detrends (rejecting bad images in iterations and such)
  • processed g band 3pi for static masks (in various ways)
  • investigated the 'edge nans' - these come from the dark model.
  • investigating merge faults - due to segfault in relphot so merging is off for now

Roy Henderson

  • PSPS
    • worked on a script which audits the current status of the loading to PSPS, i.e. what has loaded, what's merged, what didn't get processed etc.
    • of the exposures as yet unprocessed by ippToPsps, it generates a list of SMF files unavailable from Nebulous
    • started to process the 3351 exposures that failed first time around. So far so good (1288 successful, only 18 failures).
    • fixed (minor) Db corruption after numerous batches were marked as deleted when not (after datastore Db moved hosts)
    • PSPS VO
      • created a TableModel class in order to display SOAP result sets correctly in a table
      • made new query GUI component so users can query PSPS via SOAP then display results in dynamically sized table
      • ability to select range of values in a table to be plotted (not yet implemented)
      • casjobs errors are reported to a popup
  • IPP
    • czar on Monday
    • czartool
      • added multiple constructors to Plotter class to make things cleaner
      • some work re-arranging schedule for czarpoll: now 'trimming' database after metrics are created
      • removed ability to stop/run servers; added link to metrics page
      • fixed bug where it was querying on 'new' instead of 'update' for update server
  • other
    • some mailing list admin: removing bouncing emails
    • doctor all Wednesday afternoon

Bill Sweeney

  • A new production tag was defined and installed. This uses the new more robust script for chip processing that I worked on the past few weeks. A couple of problems updating old data were revealed that needed debugging and minor code changes to fix. The system survived a severe postage stamp and update storm late in the week and over the weekend with fewer errors than ever.
  • debugged and a problem with distribution triggered by compressed config dump files.
  • Queued STS data for processing against the new template stack. We are using a new ppSub recipe that does forced photometry in the input and template images. There were some problems that Gene fixed and an optimization that I implemented. We are still debugging failures that occur with some skycells.
  • fixed a bug in publish.
  • updated the IPP build installed at MPIA.
  • Investigated why certain nightly stacks were not processed.

Chris Waters

  • Ran script to gzip compress all mdc files on disk. This appears to have freed roughly 25TB of disk space.
  • Finished and tested realtime burntool code. Successfully burntools images at registration and automatically queues science exposures for chip->warp processing. Rewrote (but have not yet tested) the nightly science code to better handle the stack and diff stage processing, since it no longer needs to worry about chip.
  • Started work on log(flux) scaling mode for better compression of stack images.