== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-07-25 ===
 * 09:07 Serge: A lot of red in publishing. {{{pubtool -revert -dbname gpc1 -client_id 5}}}. Mops want to have ANW3, ANO1, and OSS asap.
 * 09:24 Serge: {{{chip.off}}} so that ANO1 quad gets processed/reverted faster. OSS and ANW3 (but 2 exposures because of bad cam quality) all published.
 * 09:34 Serge: {{{warp.off}}}. 8 exposures in ANO1 to go
 * 09:43 Serge: {{{pubtool -definerun -client_id 5 -label ThreePi.nightlyscience -dbname gpc1}}}
 * 09:47 Serge: All MOPS quads published. chip.on, warp.on
 * 12:12 CZW: Restarted stack pantasks, as it did not appear to be properly queuing new jobs (and SAS had 900 stacks sitting idle).
 * 15:52 Bills: added M31 label to stdscience and updated the input file. Did not turn on warp-stack diffs yet. We need to examine the exposures relative to the current template.
 * 16:46 Mark: re-enabled MD10 for W-S diffs tonight.
=== Tuesday : 2011-07-26 ===
 * 09:14: Serge: Heather ran and fixed? the summitcopy problem
{{{
summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o5768g0297o/o5768g0297o43.fits 
--filename neb://ipp026.0/gpc1/20110726/o5768g0297o/o5768g0297o.ota43.fits --summit_id 364059 
--exp_name o5768g0297o --inst gpc1 --telescope ps1 --class chip --class_id ota43 --bytes 49432320 
--md5 424f69d8594ef727d7b78ce03d0a7a43 --dbname gpc1 --timeout 600 --verbose --copies 2 
--compress --nebulous
}}}
 * 09:36: Bill queued sts diffs for July 3.
 * 10:03: Bill/Serge killed some ppSub running for ages (ipp043/diff_id 147113//267983.34 seconds, ippc28/diff_id 147235/204905.41 seconds, ipp038/diff_id 147554/74371.20 seconds)
 * 12:05: Serge: chip.off so that mops have OSS asap
 * 12:56: Serge: warp.off
 * 13:12: Serge: chip.on, warp.on
 * 13:20: Serge: ippdb03 is down
 * 13:40: Serge: ipp026 is down
 * 15:30: CZW: paused lap processing to track down and fix a bug that was preventing SAS2.12 gri runs from queuing stacks correctly.  This is resolved now.
 * 17:30: Serge: By Gene's request, system stop and shut down
 * 17:55: Serge: all pantasks shutdown
 * 18:00: Serge: all pantasks running
{{{
ipp@ipp004:/home/panstarrs/ipp>./check_system.sh start.server
[...]
ipp@ipp004:/home/panstarrs/ipp>./check_system.sh start
pantasks server cleanup has been started (host: ippc07)
pantasks server distribution has been started (host: ippc15)
pantasks server pstamp has been started (host: ippc17)
pantasks server publishing has been started (host: ippc08)
pantasks server registration has been started (host: ipp052)
pantasks server replication has been started (host: ippc19)
pantasks server stack has been started (host: ippc05)
pantasks server stdscience has been started (host: ippc16)
pantasks server summitcopy has been started (host: ipp050)
pantasks server update has been started (host: ippc13)
ipp@ipp004:/home/panstarrs/ipp>./check_system.sh run
pantasks server cleanup is running (host: ippc07)
pantasks server distribution is running (host: ippc15)
pantasks server pstamp is running (host: ippc17)
pantasks server publishing is running (host: ippc08)
pantasks server registration is running (host: ipp052)
pantasks server replication is running (host: ippc19)
pantasks server stack is running (host: ippc05)
pantasks server stdscience is running (host: ippc16)
pantasks server summitcopy is running (host: ipp050)
pantasks server update is running (host: ippc13)
}}}
 * 18:07: CZW: restarted replication pantasks with new input file that limits the number of hosts being used for shuffle.  This does not seem to have had a significant impact on the shuffle speed. 
 * 18:20: Serge: fixed and restarted replication of ippdb01 onto ippdb03
 * 18:20+epsilon Serge: ipp030 is down

=== Wednesday : 2011-07-27 ===
 * 13:36 Bill ipp030 is still down causing lots of faults. I turned magic, destreak and dist revert off to find out the extent of the errors
 * 15:57 Bill turned the reverts back on since ipp030 has finished it's brain transplant.
=== Thursday : 2011-07-28 ===
 * 09:16 Serge: chip.off, warp.off
 * 09:20 Serge: killed ppSub on ipp036 (diff 148435) and on ipp049 (148445) running for about 3 hours
 * 09:21 Serge: pubtool -revert -label (OSS.nightlyscience, MD10.nightlyscience, MD08.nightlyscience)



=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===