PS1 IPP Czar Logs for the week 2011-08-01 - 2011-08-07

(Up to PS1 IPP Czar Logs)

Monday : 2011-08-01

Gene is czar today

  • 13:40 Bill queued 13 warps (2011) and 24 chip runs (2010) for processing with a new M31 tessellation. M31.V3.test is the label
  • 21:04 Bill queued 36 diffs for M31.V3.test

Tuesday : 2011-08-02

  • 09:27 Serge: priorities of STS.nightlyscience and M31.nightlyscience slightly decreased so that MOPS quads in ThreePi? are processed before them.
    labeltool -dbname gpc1 -updatelabel -label STS.nightlyscience -set_priority 399
    labeltool -dbname gpc1 -updatelabel -label M31.nightlyscience -set_priority 399

Wednesday : 2011-08-03

  • 14:53 Serge: warp 231489 stuck. Following Bill's suggestions, I perl --cam_id 243744 and warp.reset in stdscience
  • 15:14 CZW: Stopped all processing to allow the for the mysql dump of gpc1 database.
  • 16:27 CZW: Restarted processing as the gpc1 database dump has finished.
  • 17:00 Bill queued some M31 warps with a modified tessellation named after it's designer M31.arno
  • 17:20 Bill queued new chipRuns for STS exposures from April that were processed before the STS.V4 tessellation was created (68 exposures)

Thursday : 2011-08-04

  • 8:30: CZW: Unstuck registration: --uri --filename neb://ipp026.0/gpc1/20110804/o5777g0502o/o5777g0502o.ota42.fits --summit_id 368481 --exp_name o5777g0502o --inst gpc1 --telescope ps1 --class chip --class_id ota42 --bytes 49432320 --md5 832209eba5124d7b988eafa9efab671c --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous

Friday : 2011-08-05

  • 12:45: Mark set quality=42 for two remaining MD10.GR0 skycell.021 reprocessed stacks that had fault=5 from bad PSF (stack_id 350546,350771).

Saturday : 2011-08-06

  • 09:30: Mark reverted few remaining fault 2,4 MD stacks and 3PI diffs. One 3PI diff (diff_id 151097) running for 9 hours on ipp026.
  • 09:45: Mark: summitcopy/registration problem with morning darks? registration down? registration/pcontrol.log last entry 05:32. Restarted (at 12:15).
  • 10:30: Mark reverted some remaining publish failures, M31 stack failures. Ran w/o -fault 2 and now czartool reporting queue to publish M31.V3test (37),SAS.footprint.123 (6), LAP.ThreePi?.20110621 (16).
  • 16:00: Mark: three M31 diffs fail not able to access template on /data/ipp026.0 from ipp006, i/o error. Around 1800 was responsive again and M31 diffs finished.
  • 18:00: Mark: two 3PI diffs failing, warp stage files missing except .log and .skycell. Regenerated the warps, 232610 finished processing okay, 232576 has error in diff with warp mask file corrupted even after re-running.
     perl ~ipp/src/ipp-20110622/tools/ --warp_id 232576 --skycell_id skycell.1837.003 --redirect-output 
     perl ~ipp/src/ipp-20110622/tools/ --warp_id 232610 --skycell_id skycell.1840.064 --redirect-output

Sunday : 2011-08-07

  • 11:00: Mark: looking at some of the failed update runs but still not sure why.