[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.11.12 ===
 Mark is czar
 * 08:00 MEH: last of nightly science finishing up. LAP and other reprocessing chip faults from permissions still. 
  * chip.off for a while to push more LAP warps
 * 11:30 still babysitting a large number of OSS diffims slowly making their way through..
 * 14:30 OSS and 3PI diffims finished, regular restart of stdscience done.
 * 21:00 MEH: poor weather so pushing more MD05 reprocessing through
 * 21:30 repaired lost files, replaced with found orphans
{{{
neb://ipp027.0/gpc1/20101102/o5502g0217o/o5502g0217o.ota26.fits
neb://ipp051.0/gpc1/20100605/o5352g0039o/o5352g0039o.ota64.fits
}}}
=== Tuesday : 2012-11-13 ===
Serge is czar
 * 06:15: No observation last night. However ipp001 mysql server crashed: Nothing in its log though. Replication is only 10 minutes behind so it should not become a major incident.
 * 14:00: Stopped all processing
 * 14:45: Restarting all rpc.statd on all storage nodes:
{{{
schastel@ippc11:~/dev/SshSudo$ ./ssh_sudo.py -H `cat tmp/storageNodes` 'sudo /etc/init.d/rpc.statd restart'
}}}
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===