== PS1 IPP Czar Logs for the week 2013.09.23 - 2013.09.29 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.09.23 ===
mark is czar
 * 07:30 MEH: nightly done, regular stdsci restart
 * 09:30 also doing restart of summitcopy, registration, distribution, publishing, stack for the week
 * 20:44 MEH: looks like 2 exposures of STD obs_mode in OPEN filter.. (o6559g0084o, o6559g0085o) -- chip stage has no FLAT detrend and will set to drop in a bit
=== Tuesday : 2013.09.24 ===
mark is czar
 * 07:4 MEH: nightly finished, regular restart of stdsci
 * 10:00 MEH: cleaning up som eLAP stuff.. 
{{{
-- fault 5
 stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 5
-- fault 4 after several revert etc
 stacktool -dbname gpc1 -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 4
-- fault 2 -- broken warps.. some required chip updates
perl ~ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 742361 --skycell_id skycell.0740.096 --redirect-output --dbname gpc1

chiptool -setimfiletoupdate -chip_id 791999 -set_label LAP.ThreePi.20130717 -dbname gpc1
/home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 748664 --skycell_id skycell.1442.095 --redirect-output
chiptool -setimfiletoupdate -chip_id 795325 -set_label LAP.ThreePi.20130717 -dbname gpc1
/home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 751839 --skycell_id skycell.2247.076 --redirect-output
chiptool -setimfiletoupdate -chip_id 783099 -set_label LAP.ThreePi.20130717 -dbname gpc1
/home/panstarrs/ipp/src/ipp-20130712/tools/runwarpskycell.pl --warp_id 739674 --skycell_id skycell.0739.019 --redirect-output
--> chip_id are >> than currently updating id, may want to use a different label to get them updated -- then will manually clean chips 

}}}
 * 12:05 MEH: ps2iq overloading ippc18 with outputs, all processing and such stalled. we need a policy of not running intensive programs involving ippc18 (on ippc18 and/or to the homedir). '''if found, will be killed'''
 * 12:50 MEH: cleanup counters wacky, doing a restart
 * 17:40 MEH: chip.off to catch up warps and push stacks before restarting stdsci and nightly data
=== Wednesday : 2013.09.25 ===

=== Thursday : 2013.09.26 ===
 * 07:38 Bill: Lots of timeouts for postage stamp tasks. Looks like a database problem. Also quite a few in the update pantasks. Restarting them both
 * 14:22 Bill: stdscience could use a restart. Set to stop
=== Friday : 2013.09.27 ===

 * heather 9am: sine waves on ganglia, restarting stdsci
 * heather 12pm: something happened overnight to ipptopsps? Any idea? someone reboot a machine or something?
=== Saturday : 2013.09.28 ===

=== Sunday : 2013.09.29 ===