IPP Progress Report for the week 2013-02-04 - 213-02-10

(Up to IPP Progress Reports)

Eugene Magnier

  • finished up the ipptopsps integration with the parallel dvo, tested against SAS2.
  • helping Heather with the parallel dvomerge, etc, operations

Serge Chastel

  • IPP
    • Added timeout of 60 seconds for deneb-locate.py. Shouldn't hang forever now.
    • Fixed nebdiskd bug (there is no "Available Space" column in the mountedvol table hence leading to confusion).
    • Look at the IPP/MOPS report part for more
    • Creation of a mops account on the MHPCC cluster since we want to copy files from mops@mhpcc to ipp@neo.
    • Worked on an IPP framework installation dedicated for MOPS on the NEO cluster.
      • Identified non-robust middleware between pantasks_server and pcontrol at the protocol level. My pantasks_server was linked against a more recent version of readline. This version was sending an extra string to pcontrol which was not complaining or showing any error (but wasn't doing anything). Relinking against the right version of the library fixed the problem but pcontrol should be in my opinion less tolerant to that kind of problem.
      • Nasty bug in pswarp. It sometimes fails when the number of threads is set to 1 but succeeds when it’s set to 4.
      • Nasty bug in ppImage. It sometimes converges after a really long-time when the number of threads is set to 4 but succeeds when it’s set to 1.
      • I had to modify the prepare_output function in chip_imfile.pl deletes files that are produced at the beginning of the chip stage. The steps (I simplify a bit since there are also steps involving data control and database operations) in the chip stage are the following: 1) "Prepare output"; 2) Detrend raw files; 3) Do photometry. It is supposed to be possible to control the execution or non-execution from the recipe. In the "IPP for MOPS" context we are only interested in the third step. Unfortunately, the "Prepare output" behavior is not controlled by a recipe: files deletion occurs systematically... but those produced by step 2) and necessary for step 3) are deleted. To fix the problem, I had to modify the original chip_imfile.pl.
      • Using the same notations as in the previous point, ppImage hard-codes the 2) and 3) steps. No way to insert another step (like for instance the MOPS synthetics generation). Facade and Command Design Patterns would have helped a lot. I can tell more about this.
  • MOPS
    • Implemented the data copy from the MHPCC cluster to the NEO cluster. Requires connections to the IPP nebulous database (ssh neb-ls is way too slow) and gpc1 database. Replication slaves on ippdb04 and ipp001 are used (no the masters). Implemented also the installation of data where the IPP expects them.
    • Renamed the tessellation from GPC1.RINGS.V3 to the expected RINGS.V3 in order not to change the recipes and the pantasks modules too much. (My intention was able to be able to distinguish between GPC1, MegaCam...).
    • Installed paramiko (python ssh client) on the NEO cluster.
    • Meeting with Larry about comets (but no significant progress on this topic from me).

Heather Flewelling

focused on dvo and merging:

  • continued to build minidvodbs for different slices. slice 18 and 8 are complete (well, 8 is complete to 9.5hr - 10 hr due to winter), slice 10 and 12 are building (see wiki)
  • rebuild of ippdvo's ipp to pick up parallel dvodb stuff
  • a bit hacky to make parallel mergedvodb - much care and handholding for the first, followed by the rest are easy (see wiki)
  • build/rebuild of mergedvodb - had problem with permissions, then problem with addstar -resort.
    • merges are now lightening fast
    • addstar -resort is lightening fast

Mark Huber

  • switching off md.gr work again -- updated MD.GR0 with V3 wrap-up, summary table and stack photometry status; added info to Image_Artifacts of two possible cell problems
  • returning to tracing WSdiffim photometry again, catching up on notes and test runs where left off with strange PSF

Bill Sweeney

  • Looked at a few of the stacks that generated excess detections. Wrote a wiki page to discuss the issue http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/staticsky.20120706_excess_detections
  • Continued working on changes to the "background preserved pipeline" used by the M31 team to incorporate an additional mask application. This "auxiliary mask" will be supplied by the M31 team to specify areas to be excluded from the warped images.
    • Ran into some interesting problems with the mask files that MPG supplied. Learned a few new things about the camera configuration system. A keyword in the image headers confused our system. Fixed the problem by editing the mask files.
    • Decided on the plan of attack. The M31 exposures will be processed through warp the the standard recipes. Then the chip stage will be re-run with background subtraction and pattern correction turned off. Those will then be sent though the chip_bg stage which will apply the auxiliary mask and finally the warp_bg stage will use the astrometry and skycell map from the standard processing to produce warps that will be delivered to MPG.
    • Studied the way registered detrends work. It will be straightforward to use that for the final versions of the auxiliary masks
  • Updated the ipp at MPIA. Needed to fix some build problems uncovered with their version of gcc.
  • Investigated some problems with cleanup. I am being too quick to raise the error_cleaned error. Will repair this next week.
  • queued LAP staticsky processing for stacks in the region being processed now. This is not yet automated.
  • Two days as processing czar. One busy, one not so mcuh.

Chris Waters

  • More trail fitting tests for MOPS. Sorted out how the S/N is calculated for object skipping (it uses the Kron S/N).
  • PATTERN.MEDIAN tests. Results: GPC1_Bias_Pattern_Study
  • Code changes for background model support in new tag.
  • Began work on fixing the slow code in WS diff queuing.