PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011-07-25

• 09:07 Serge: A lot of red in publishing. pubtool -revert -dbname gpc1 -client_id 5. Mops want to have ANW3, ANO1, and OSS asap.
• 09:24 Serge: chip.off so that ANO1 quad gets processed/reverted faster. OSS and ANW3 (but 2 exposures because of bad cam quality) all published.
• 09:34 Serge: warp.off. 8 exposures in ANO1 to go
• 09:43 Serge: pubtool -definerun -client_id 5 -label ThreePi.nightlyscience -dbname gpc1
• 09:47 Serge: All MOPS quads published. chip.on, warp.on
• 12:12 CZW: Restarted stack pantasks, as it did not appear to be properly queuing new jobs (and SAS had 900 stacks sitting idle).
• 15:52 Bills: added M31 label to stdscience and updated the input file. Did not turn on warp-stack diffs yet. We need to examine the exposures relative to the current template.
• 16:46 Mark: re-enabled MD10 for W-S diffs tonight.

Tuesday : 2011-07-26

• 09:14: Serge: Heather ran and fixed? the summitcopy problem

  summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o5768g0297o/o5768g0297o43.fits 
  --filename neb://ipp026.0/gpc1/20110726/o5768g0297o/o5768g0297o.ota43.fits --summit_id 364059 
  --exp_name o5768g0297o --inst gpc1 --telescope ps1 --class chip --class_id ota43 --bytes 49432320 
  --md5 424f69d8594ef727d7b78ce03d0a7a43 --dbname gpc1 --timeout 600 --verbose --copies 2 
  --compress --nebulous
  

• 09:36: Bill queued sts diffs for July 3.
• 10:03: Bill/Serge killed some ppSub running for ages (ipp043/diff_id 147113267983.34 seconds, ippc28/diff_id 147235/204905.41 seconds, ipp038/diff_id 147554/74371.20 seconds)
• 12:05: Serge: chip.off so that mops have OSS asap
• 12:56: Serge: warp.off
• 13:12: Serge: chip.on, warp.on
• 13:20: Serge: ippdb03 is down
• 13:40: Serge: ipp026 is down
• 15:30: CZW: paused lap processing to track down and fix a bug that was preventing SAS2.12 gri runs from queuing stacks correctly. This is resolved now.
• 17:30: Serge: By Gene's request, system stop and shut down
• 17:55: Serge: all pantasks shutdown
• 18:00: Serge: all pantasks running

  ipp@ipp004:/home/panstarrs/ipp>./check_system.sh start.server
  [...]
  ipp@ipp004:/home/panstarrs/ipp>./check_system.sh start
  pantasks server cleanup has been started (host: ippc07)
  pantasks server distribution has been started (host: ippc15)
  pantasks server pstamp has been started (host: ippc17)
  pantasks server publishing has been started (host: ippc08)
  pantasks server registration has been started (host: ipp052)
  pantasks server replication has been started (host: ippc19)
  pantasks server stack has been started (host: ippc05)
  pantasks server stdscience has been started (host: ippc16)
  pantasks server summitcopy has been started (host: ipp050)
  pantasks server update has been started (host: ippc13)
  ipp@ipp004:/home/panstarrs/ipp>./check_system.sh run
  pantasks server cleanup is running (host: ippc07)
  pantasks server distribution is running (host: ippc15)
  pantasks server pstamp is running (host: ippc17)
  pantasks server publishing is running (host: ippc08)
  pantasks server registration is running (host: ipp052)
  pantasks server replication is running (host: ippc19)
  pantasks server stack is running (host: ippc05)
  pantasks server stdscience is running (host: ippc16)
  pantasks server summitcopy is running (host: ipp050)
  pantasks server update is running (host: ippc13)
  

• 18:07: CZW: restarted replication pantasks with new input file that limits the number of hosts being used for shuffle. This does not seem to have had a significant impact on the shuffle speed.
• 18:20: Serge: fixed and restarted replication of ippdb01 onto ippdb03
• 18:20+epsilon Serge: ipp030 is down

Wednesday : 2011-07-27

• 13:36 Bill ipp030 is still down causing lots of faults. I turned magic, destreak and dist revert off to find out the extent of the errors
• 15:57 Bill turned the reverts back on since ipp030 has finished it's brain transplant.

Thursday : 2011-07-28

• 09:16 Serge: chip.off, warp.off
• 09:20 Serge: killed ppSub on ipp036 (diff 148435) and on ipp049 (148445) running for about 3 hours
• 09:21 Serge: pubtool -revert -label (OSS.nightlyscience, MD10.nightlyscience, MD08.nightlyscience)

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD