== PS1 IPP Czar Logs for the week 2013.09.23 - 2013.09.29 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.09.23 ===
mark is czar
 * 07:30 MEH: nightly done, regular stdsci restart
 * 09:30 also doing restart of summitcopy, registration, distribution, publishing, stack for the week
 * 20:44 MEH: looks like 2 exposures of STD obs_mode in OPEN filter.. (o6559g0084o, o6559g0085o) -- chip stage has no FLAT detrend and will set to drop in a bit
=== Tuesday : 2013.09.24 ===
mark is czar
 * 07:4 MEH: nightly finished, regular restart of stdsci
 * 10:00 MEH: cleaning up som eLAP stuff.. 
{{{
-- fault 5
 stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 5
-- fault 4 after several revert etc
 stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 4
-- fault 2 -- broken warps.. some required chip updates
perl ~ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 742361 --skycell_id skycell.0740.096 --redirect-output --dbname gpc1

chiptool -setimfiletoupdate -chip_id 791999 -set_label LAP.ThreePi.20130717 -dbname gpc1
/home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 748664 --skycell_id skycell.1442.095 --redirect-output
chiptool -setimfiletoupdate -chip_id 795325 -set_label LAP.ThreePi.20130717 -dbname gpc1
/home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 751839 --skycell_id skycell.2247.076 --redirect-output
chiptool -setimfiletoupdate -chip_id 783099 -set_label LAP.ThreePi.20130717 -dbname gpc1
/home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 739674 --skycell_id skycell.0739.019 --redirect-output
--> chip_id are >> than currently updating id, may want to use a different label to get them updated -- then will manually clean chips 

}}}
 * 12:05 MEH: ps2iq overloading ippc18 with outputs, all processing and such stalled. we need a policy of not running intensive programs involving ippc18 (on ippc18 and/or to the homedir). '''if found, will be killed'''
 * 12:50 MEH: cleanup counters wacky, doing a restart
 * 17:40 MEH: chip.off to catch up warps and push stacks before restarting stdsci and nightly data
=== Wednesday : 2013.09.25 ===

=== Thursday : 2013.09.26 ===
 * 07:38 Bill: Lots of timeouts for postage stamp tasks. Looks like a database problem. Also quite a few in the update pantasks. Restarting them both
 * 14:22 Bill: stdscience could use a restart. Set to stop
=== Friday : 2013.09.27 ===

 * heather 9am: sine waves on ganglia, restarting stdsci
 * heather 12pm: something happened overnight to ipptopsps? Any idea? someone reboot a machine or something?
=== Saturday : 2013.09.28 === 
 * 07:05: Serge reports that registration is backed up. 
  * Bill looked at the registration pantasks and saw that the status page was a mess. Many many job timeouts. Restarted pantasks and burntool started running. 
  * At about the same time Heather repaired the fault. 
 * 07:18 Bill: stdscience needs to be restarted. Set to stop
 * 07:34: restart complete. Didn't wait for some LAP camera runs to finish. Removed lap label for a bit to avoid running those jobs twice.
 * 07:55: all back to normal
 * 10:06: Bill dropped a warp skyfile which repeatedly pops assertion failure with warptool -updateskyfile -skycell_id skycell.2008.031 -warp_id 842911 -set_quality 4  -fault 0
 * 14:00 MEH: stsci13 appears down around 1200.. forgot should've taken out of neb when noticed so can at least do nightly science tonight if no one around to reboot with the special raid issues
  * probably could restart stack, pstamp to clean stalled jobs -- will need to clean up hanging jobs and mounts.. MOPS will need PSS in morning so needs to be done if stsci13 not rebooted before then
 * 14:30 MEH: stacks cleared, but stack.revert.off since many faults from warps only on stsci13...
 * 15:50 MEH: clearing enough update+pstamp so that MOPS can get their stamps for TODAY, some QUB ones still hanging around
 * 16:30 MEH: dumped and restarted cleanup.. so little space, need any we can recover at this point.. there will be stalled ipp_cleanup.pl, won't kill and don't want to do -9 right now..
 * 17:10 MEH: in case no stsci13 before tonight, still trying to get things running. registration stuck on dark imfile -- cleaned and restarted registration and summitcopy 
 * 17:15 MEH: will leave stdsci last as it will be a major PITA to clean out
 * 16:10 MEH: neb having all kinds of fun timouts/faults in summitcopy, 900+ connections in nebdb.. time to start clearing all out before nightly starts..
=== Sunday : 2013.09.29 ===