== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-08-15 ===
 * 15:39 Serge. Dropped 157 exposures set to be published by mistake:
{{{
mysql> SELECT count(*) FROM publishRun where state='new' AND label='ThreePi.nightlyscience';
157
mysql> UPDATE publishRun SET state='drop' WHERE state='new' AND label='ThreePi.nightlyscience';
mysql> SELECT count(*) FROM publishRun where state='new' AND label='ThreePi.nightlyscience';
0
}}}

Bill is czar today

 * investigated why a publishRuns were stuck (before Serge dropped them). It is related to the code to do "off night diffs". 
 * Investigated a faulted MD10.GR0 publishRun. Bug in destreak script. Dropped the run.
 * 17:18: No faults on czartool M31 reference stacks are taking > 8000 seconds per skycell.

=== Tuesday : 2011-08-16 ===
Bill is czar today
 * Gene reports that some of the M31 stacks are using huge amounts of memory. These are i band skycells near the M31 core. We've killed off a few
 * turned hosts off in stdscience for ippc22 ippc24 ippc27 ippc28 to dedicate them to stacking
 * added stare nodes to stack
 * 11:05 fixed corrupted chip --chip_id 271038 --class_id XY53
 * 11:16 fixed corrupted diff --diff_id 154025 --skycell_id skycell.7.41
 * 13:15 removed M31.refstack.20110812 label from stack. We're going to run the 4 bloated skycells in a separate pantasks with only the stare nodes enabled.
 * 13:39 set stare hosts to off in stack
 * 14:23 remove stare hosts from stack. last m31 refstack process is swapping in it's 19GB of memory to check for leaks
 * 15:27 M31 stacking moved to new pantasks ~ipp/stack.m31. only the 4 stare nodes are enabled.


=== Wednesday : 2011-08-17 ===
 * 09:05 Serge End of IPP-MOPS-TEST datastore cleaning: 107GB recovered, 19344 objects deleted
=== Thursday : 2011-08-18 ===

=== Friday : 2011-08-19 ===

=== Saturday : 2011-08-20 ===

=== Sunday : 2011-08-21 ===