IPP Progress Report for the week 2011.08.01 - 2011.08.05

(Up to IPP Progress Reports)

Eugene Magnier

I've been concentrating on coding related to the extended source analysis, particularly speeding up the analysis. I've threaded the petrosian and non-linear model fits (finally) so that we use the multiple cores more efficiently. I should be able to wrap up these improvements early this week and fold them into the trunk. This will make testing of the extended source parameters much more efficient.

I also worked on extracting parallaxes for our new PS1+WISE T-dwarf discovery (Liu et al paper), and a second test object with a known parallax. I generated stand-along DVO databases for this analysis and ran relastro on the database to improve the per-chip calibrations. The parallax measurement worked very well, and gives values for the test case that are similar to the reported parallaxes. The test case had a fortuitous amount of PS1 coverage from the past 3 years, so it is actually quite representative of what we can expect from the completed survey.

I put together an initial plan for the hardware upgrades over the next couple weeks. We need to upgrade motherboards and hard disks in Wave 1 machines, and also upgrade the Manoa cluster. The timing is constrained by the data shuffling needed to move the data from the 10TB partitions while they are upgraded.

Serge Chastel

  • MOPS czar
  • mysql replication issues because of ippdb03 crash
  • Modified replication scripts to save master coordinates
  • Updated associated documentation
  • NFS check is now running on ippdb0[0-3]
  • Reviewed PSPS Test Plan for Conrad
  • http+nfs stopped on neverland (were they needed?)

Heather Flewelling

  • vacation of awesomeness

Roy Henderson

  • DVO duplicates investigation:
    • some confusion over DVO IMAGE_ID-EXTERN_ID relationship. Finally got code making sense
    • added nice, safe code to re-create DVO master tables in MySQL
    • ran code, found duplicates, and showed Gene. Appears to be a chip-boundary issue
  • ippToPsps coding:
    • all programs now take a single argument: a path to a config
    • rewrote convertPhotCodesToXml.pl to produce VOTable-compliant output
    • a lot of awk scripting to convert old IN data format to VOTable
    • main ippToPsps program can now produce and publish IN batches. Old C and Perl code has been purged. Very satisfying.
    • extra methods for logging class, including some debug messages and a 'title' method
    • now using one, and only one, name for batch types: P2, ST, IN. Changed directory structure for configs accordingly
    • added batch types to config
    • new section for config enabling a list of IDs (cam_id or stack_id) to process (useful for testing)

  • documentation
    • lots of new comments for ippToPsps
    • a lot of new documentation for ippToPsps on the wiki. Pretty much up-to-date now
  • investigated crazy high fluxes seen in last load to PSPS:
    • changed code to enable processing of a specific exposure (or exposures)
    • found smfs with crazy PSF_INST_MAG values (i.e. < -20)
    • implemented a cut for PSF_INST_MAG < -17.5, as recommended by Gene

Mark Huber

  • Additional testing of ppSub and convolution direction for MD10.V3 stack-stack diffs. Prep for new MD01.V3 tessellation.
  • simtest IPP->PSPS: still working on the configuration. Changed from original simtest field to MD04.
  • Czar Friday.

Bill Sweeney

  • Created a new M31 skycell tessellation. Ran some tests and reported to the M31 team. After feedback ran more tests. One more round and we should be ready to make the new reference stack.
  • Debugged and fixed a problem with the postage stamp server.
  • Spent a day finding the root cause of an error that caused ppStack to fail occasionally.
  • Fixed a bug in the destreak revert process that I introduced a couple of weeks ago that caused certain runs to fail. Spent half a day recovering from the damage that the bug caused.
  • Out sick and at doctor for 3/4 day.

Chris Waters

  • DQStats: Finally fixed the SQL to only publish new results, but to correctly publish all new results.
  • Refcat zeropoints: Reprocessed SAS2 z-filter data as part of testing of the new LAP code, using the synthetic catalog instead of the new reference catalog. This eliminates the position dependent zeropoint offsets that were observed, pointing to issues in the reference catalog in those regions.
  • LAP: Further coding and debugging of LAP code to eliminate quickstacks. Once final tests are complete, a new set of LAP runs will be queued.
  • ROC: Continued correcting these files to match real disk files. It appears (from a not entirely unbiased sample) that the majority of missing imfiles can either be reconstructed from N-th copies still on disk, or that enough other files in a given ROC can be recovered to fix individual missing files. No statistics are complete on this, and more work needs to be done to allow this to run automatically.
  • Out one day due to illness and the inadvertent slaughter of my laptop by a glass of water.