IPP Progress Report for the week 2011-09-12 - 2011-09-18

(Up to IPP Progress Reports)

Eugene Magnier

  • Vacation

Serge Chastel

  • Replication issues with isp
  • ippc19 replication trouble
  • Replication status in czartool now displays "seconds behind master"
  • IPP czar on Thursday and Friday
  • Optimization of gpc1, partial optimization of nebulous (see czar log for details)
  • Studying pantasks to see how features could be added

Heather Flewelling

  • many plots/wiki/investigations of false positives in SAS.
  • found a diagonal feature on long exposure g band images.
  • LAP/ThreePi dvodb czar
  • sent M. ISP skyprobe images of SN field.

Roy Henderson

  • Vacation

Mark Huber

  • MD.V3 stack summary continued.
  • czar Tuesday, Wednesday
    • worked through tracking down empty diff detection table trouble in magicDS.
  • psphotStack test runs on SAS2.
  • simtest IPP->PSPS test configuration debugging continued.

Bill Sweeney

  • My changes to ppMops to reduce the memory footprint were incomplete. Spent most of the week rewriting the code to use the function psFitsWriteTableAllColumns() to use a more compact internal data representation.
  • started looking at the cause for the much higher streak counts from M31 this year. (The censored data is unusable at present.) No conclusions as yet but some observations.
    • The problem is primarily with i band exposures.
    • The problem was apparent before the change in skycell tessellation.
    • It does not have a large correlation with seeing.
    • Certain areas of the exposures are triggering a large number of false streak detections.
  • Continued to look for problems with throughput.

Chris Waters

  • LAP: Finished reworking of LAP code to use a one exposure = one processing system. This should cut down on some work by eliminating the need to reprocess an exposure multiple times. This also uses calls to the update system to regenerate the warps for exposures that have been processed as part of that LAP and then cleaned up. This will also help reduce the work load.
  • Processing speed: Rewrote the host allocation code used in the pantasks input files for all servers, allowing us to use a single central host list. This also should ensure that each host is used more efficiently than in the past. This is especially important given the refurbishment of the wave 1 nodes, which now have more available processing power. Preliminary indications show that we are completing more jobs/day than before.
  • Targeting: Shuffle code to retarget the raw OTA data is running. It has a large error rate, but this is due to the paranoid checking it performs before attempting to move the files. Re-examining the files that are causing the errors suggests that the majority are not real, but an artifact of checking data at the ATRC over the slower network. The remainder of error-causing files (usually a file with a bad copy due to the shuffle issues from before June) are being repaired by a script written to do so.
  • Identified possible slow-down in chip stage processing: we are making an unindexed database request for the average ccd temperature across the exposure for each chip processed. This is slow, and as far as I can tell, unnecessary to any part of processing. This query seems to be putting extra load on the gpc1 database, and if we can eliminate it, we may be able to gain in the total throughput.