[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012-10-15 ===
 * 07:30 Bill: ipp020 is having nfs problems which has clogged stdscience and registration. Fixed the registration problem (we are a couple of hundred exposures behind).  Working on ipp020 now.
 * 07:50 Restarted stdscience and summitcopy. A stuck job in summitcopy was blocking burntool processing. We have 274 exposures copied but not registered. ipp020 is set to repair and set to off in registration and summit copy.
 * 08:00 removed LAP label from stdscience to give nightlyscience all of the horespower
 * 11:36 added lap label back. the database replication problem happened again and czartool is behind the times
 * 14:14 Bill ran stop slave; set global sql_slave_skip_counter=1; start slave; on ippdb03 and ipp001 to fix the replication problem
 * 22:10 MEH: looks like ipp016 has stalling registration and processing.. trying to isolate + clean up
  * restarted registration w/o ipp016 - ok
  * restarted stdscience w/o ipp016 - nightly_science.pl --queue_diffs hanging again.. will probably cause replication error
  * restarted pstamp w/o ipp016 - ok
  * neb-host ipp016 repair -- dont want to reboot tonight if don't have too, can still access /data/ipp016.0 just cannot ssh into (like ipp023, ipp013 etc recently)
=== Tuesday : 2012-10-16 ===

 * 06:08 EAM : ipp023 crashed, rebooted it ([wiki:Ipp023-crash-20121016])
 * 06:20 EAM : set ipp023 to 'retry' in summitcopy and registration (had been 'down' due to crash).  also set ipp054 - ipp059 to 'on' in summitcopy.  we used to have 30 machines doing the download, but now we seem to be down to 23 by default.  I suspect this is because of concern about some of the wave 1 nodes.  I think we should keep enough nodes in the list to keep the download rate acceptable.
 * 07:55 Bills: ipp016 had a panic this morning just power cycled it.
{{{
<Oct/15 06:09 pm>ipp016 login: [1200542.035287] BUG: unable to handle kernel paging request at ffffffff800073d4
<Oct/15 06:09 pm>[1200542.036617] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
<Oct/15 06:09 pm>[1200542.036617] PGD 203067 PUD 207063 PMD 0 
<Oct/15 06:09 pm>[1200542.036617] Oops: 0000 [#1] SMP 
<Oct/15 06:09 pm>[1200542.036617] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
<Oct/15 06:09 pm>[1200542.036617] CPU 0 
}}}
 * 10:20 EAM : repeated failures on stack_id 1618773 : unable to find PSF.  I set quality & fault to 42

=== Wednesday : 2012-10-17 ===
 * 06:42 Bill : repaired stalled db replication on ippdb03 and ipp001
 * 09:00 EAM : things were sluggish, so I've stopped and restarted all pantasks.

=== Thursday : 2012-10-18 ===
 * 02:45 (Serge): Fixed ipp001 and ippdb03 replication
 * 09:45 (Serge): Bill rebooted sick ipp012. Serge set it to down for nebulous. {{{neb-host ipp012 down}}}
 * 09:55 (Serge): {{{neb-host ipp012 up}}}
=== Friday : 2012-10-19 ===

=== Saturday : 2012-10-20 ===

=== Sunday : 2012-10-21 ===