== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.10.31 ===
 * 00:30 another LAP chip 
{{{
neb://ipp025.0/gpc1/20100522/o5338g0475o/o5338g0475o.ota76.fits
      0                     NON-EXISTANT file:///data/ipp053.0/nebulous/14/50/286984446.gpc1:20100522:o5338g0475o:o5338g0475o.ota76.fits
      1 c5d2ec4d851524456fb48a1bd770d6bf file:///data/ippb01.1/nebulous/14/50/1154222719.gpc1:20100522:o5338g0475o:o5338g0475o.ota76.fits

cp /data/ippb01.1/nebulous/14/50/1154222719.gpc1:20100522:o5338g0475o:o5338g0475o.ota76.fits /data/ipp053.0/nebulous/14/50/286984446.gpc1:20100522:o5338g0475o:o5338g0475o.ota76.fits

--> neb-repair should have been able to fix this case but didn't in the chip stage if problem. if doesn't then need to know why.
}}}
 * Ingestion of nebulous on ippdb02 completed  during the week end (after 10 days: {{{Wed Oct 19 16:12:33 HST 2011 - Sat Oct 29 00:09:27 HST 2011}}}). Replication started
{{{
CHANGE MASTER TO MASTER_HOST='ippdb00', MASTER_USER='repl_neb', MASTER_PASSWORD='xxx', MASTER_LOG_FILE='mysqld-bin.003352', MASTER_LOG_POS=277806591;
}}}
The slave is 1011144 seconds behind the master.
 * 11:15 Mark: returning the compute2 group from deepstack back to distribution for now.
 * 14:17 Bill: dropped two chips whose files have been lost

{{{
regtool -updateprocessedimfile -set_ignored -exp_id 232263 -class_id XY15
regtool -updateprocessedimfile -set_ignored -exp_id 231044 -class_id XY16
}}}

=== Tuesday : 2011-11-01 ===
Heather is czar today
 * 10:31 Bill had Gavin change the automount parameters to use hard instead of soft mounts. Pantasks' restarted
 * 12:45 or so Found two more lost files

{{{
regtool -updateprocessedimfile -set_ignored -exp_id 187628 -class_id XY50
regtool -updateprocessedimfile -set_ignored -exp_id 187601 -class_id XY50
}}}
 * chip 335562 XY67 has been faulting repeatedly. To debug Bill attempted to run it by hand and it succeeded. Strange
=== Wednesday : 2011-11-02 ===
 * Bill queued 20 i band M31 exposures for chip through warp processing for a demonstration of how magic makes M31.2011 data unusable.

=== Thursday : 2011.11.03 ===
 * 10:00 Mark adding the condor_MD07.V3_01 label to pantasks to finish the chip-warp processing for refstacks. 
 * 10:10 stdscience seems lagging, restarting. LAP gets held up on ipp053 jobs. Moving condor_MD07.V3_01 priority above LAP. MD07 also being held up on issues preventing jobs from completing, but is still processing other MD07 jobs.
=== Friday : 2011.11.04 ===
Mark czar
 * 01:05 ippc13 down, cycling power. no info on console.
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===