IPP Progress Report for the week 2011.01.31 - 2011.02.04

(Up to IPP Progress Reports)

Eugene Magnier

I spent the week working on some code issues in the relative astrometry and photometry analysis to be ready to create the full-sky reference database. The work needed involved better handling of the per-measurement errors in the image fits and adding fields to track the systematic errors for each image. I also improved the visualization tools to be sure the code is working correctly. I ran some modest-scale tests (600 square degrees) of relastro and am getting acceptable residuals, with the bulk of images in the range 20-30mas. To be clear, this number is the scatter of all stars on a given image with S/N > 100; I do not attempt to back out the purely systematic term from the Poisson noise, so the Poisson contributions may be elevating this value a bit. A more accurate number will come from the analysis of the per-object residuals as a function of S/N, which I have not examined. Even so, this is not quite as good as we probably can do. I have not yet tried to chase down the worst outlier images or worry too much about the outlier rejection. I have the impression from residual plots that there are still systematic structures visible in the residuals as a function of position, so some improvement can probably be made. The fits may also be somewhat driven by the fact that each 2MASS measurement contributes to the fit, even though they are not statistically independent (and have worse systematics than PS1). Even though improvements can be made, I am going to use the code as is to generate the full-sky reference database to be used by the Grand Reprocessing.

In the relative photometry area, I have assembled a dvo database of all MD fields, and will be working to get the zero point history from that for comparison with the 3pi database (and to merge the results into a common solution).

I also created an updated master 3pi database snapshot up through 2011.02.01 for release to the community.

Serge Chastel

  • MOPS czar
  • improved nebulous log files parsing
  • Wrote script using rpcinfo to replace nebdiskd and detect nfs failures
  • Changed configuration of nebulous mysql server
  • Analyzing nebulous source code

Heather Flewelling

  • mops - ran a subset of PI data with gene's branch (and published it). Made plots of the number of sources found in diff for the old processed and the gene-branch.
  • magictest - ran from start to magic a selection of images with gene-branch and the new dark and static mask. The static mask needs a few tweaks, but otherwise it looks good. See http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/MagicTest20101228 for more information.
  • ippMonitor - now has pages for minidvobs and addstar
  • addstar/dvodb - we are now up to date on minidvodbs (and they are merged). Currently building new minidvodbs but the merging is off while Gene adds 2MASS to the big dvodb.
  • czar on Friday - nothing too exciting to report.

Roy Henderson

  • PSPS
    • meeting and discussion with Daniel regarding the PSVO/MOPS development
    • ippToPsps
      • stacks: now populating R1->R10 flux values in StackApFlx table
      • more wrapper code to hide awfulness of cfitsio
      • created a sort-of unit test for detection batches, to help validate data after code changes
    • PSVO
      • improved query panel: now a large text area with a 'listener' so that syntax highlighting can be used and, perhaps later, errors flagged while typing
      • Db drivers now built into jar file and on CLASSPATH
      • ability to choose the SAMP client where you wish to send data, rather than just 'all'
  • IPP
    • Czar Monday and Tuesday
  • Other
    • some time-consuming mailing list admin, manually removing and adding numerous MPG people
    • hospital appointment all Wednesday afternoon
    • wrote a PS1SC blog article about the IPP

Bill Sweeney

  • Most of this week was spent reworking the distribution system to store distribution bundle files and logs in nebulous. Implemented and tested the changes thoroughly. This should reduce problems that we have had from time to time where nodes get data written to them when they are over the 98% full.
  • Searched for, found, and cleaned up more data. Changed retention times to deal with space crunch.
  • Changed the postage stamp server status page to be generated periodically by the server rather than in response to each user's web request. Added a lot more information about requests and jobs that are in progress or that have finished in the last 24 hours. It shows that the update fault problems are not as bad as I had feared.
  • Investigated why we get so many faults during in update processing. Decided that we need to enhanced the postage stamp dependency check program to tell stop the update process if a component faults repeatedly (it already does give up on the postage stamp request).
  • Continued to queue and monitor the processing and distribution of STS data from September and October 2010. We have a couple more nights left to go. (These data were delayed due to the lack of an i band template stack.)
  • Discovered that destreak cleanup is deleting the censored pixel images! This prevented MOPS from getting stamps from muggle images. Fixed that.
  • Reviewed postage stamp dependency checking. Started optimizing that script. Found a bug that prevented some diff updates from being queued.

Chris Waters

  • Updated DQstats code to use detection efficiency CDF to report quality back to OTIS. Added columns as requested.
  • Work on nebulous to attempt to speed up shuffling files around. Discovered that the faster code pushed a lot of processing into the future, which results in effectively no speed improvement. Began running old shuffling code instead.
  • Updated registration code to correctly handle ISP data. Changed some underlying registration SQL to better deal with large amounts of unregistered data.
  • Work on burntool --continue and burntool --fixit modes. Continue should be finished, fixit postponed until the future.