PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011-08-15

• 15:39 Serge. Dropped 157 exposures set to be published by mistake:

  mysql> SELECT count(*) FROM publishRun where state='new' AND label='ThreePi.nightlyscience';
  157
  mysql> UPDATE publishRun SET state='drop' WHERE state='new' AND label='ThreePi.nightlyscience';
  mysql> SELECT count(*) FROM publishRun where state='new' AND label='ThreePi.nightlyscience';
  0
  

Bill is czar today

• investigated why a publishRuns were stuck (before Serge dropped them). It is related to the code to do "off night diffs".
• Investigated a faulted MD10.GR0 publishRun. Bug in destreak script. Dropped the run.
• 17:18: No faults on czartool M31 reference stacks are taking > 8000 seconds per skycell.

Tuesday : 2011-08-16

Bill is czar today

• Gene reports that some of the M31 stacks are using huge amounts of memory. These are i band skycells near the M31 core. We've killed off a few
• turned hosts off in stdscience for ippc22 ippc24 ippc27 ippc28 to dedicate them to stacking
• added stare nodes to stack
• 11:05 fixed corrupted chip --chip_id 271038 --class_id XY53
• 11:16 fixed corrupted diff --diff_id 154025 --skycell_id skycell.7.41
• 13:15 removed M31.refstack.20110812 label from stack. We're going to run the 4 bloated skycells in a separate pantasks with only the stare nodes enabled.
• 13:39 set stare hosts to off in stack
• 14:23 remove stare hosts from stack. last m31 refstack process is swapping in it's 19GB of memory to check for leaks

Wednesday : 2011-08-17

Thursday : 2011-08-18

Friday : 2011-08-19

Saturday : 2011-08-20

Sunday : 2011-08-21