(Up to PS1 IPP Czar Logs)

Monday : 2012-10-15

• 07:30 Bill: ipp020 is having nfs problems which has clogged stdscience and registration. Fixed the registration problem (we are a couple of hundred exposures behind). Working on ipp020 now.
• 07:50 Restarted stdscience and summitcopy. A stuck job in summitcopy was blocking burntool processing. We have 274 exposures copied but not registered. ipp020 is set to repair and set to off in registration and summit copy.
• 08:00 removed LAP label from stdscience to give nightlyscience all of the horespower
• 11:36 added lap label back. the database replication problem happened again and czartool is behind the times
• 14:14 Bill ran stop slave; set global sql_slave_skip_counter=1; start slave; on ippdb03 and ipp001 to fix the replication problem
• 22:10 MEH: looks like ipp016 has stalling registration and processing.. trying to isolate + clean up
  • restarted registration w/o ipp016 - ok
  • restarted stdscience w/o ipp016 - nightly_science.pl --queue_diffs hanging again.. will probably cause replication error
  • restarted pstamp w/o ipp016 - ok
  • neb-host ipp016 repair -- dont want to reboot tonight if don't have too, can still access /data/ipp016.0 just cannot ssh into (like ipp023, ipp013 etc recently)

Tuesday : 2012-10-16

• 06:08 EAM : ipp023 crashed, rebooted it (Ipp023-crash-20121016)
• 06:20 EAM : set ipp023 to 'retry' in summitcopy and registration (had been 'down' due to crash). also set ipp054 - ipp059 to 'on' in summitcopy. we used to have 30 machines doing the download, but now we seem to be down to 23 by default. I suspect this is because of concern about some of the wave 1 nodes. I think we should keep enough nodes in the list to keep the download rate acceptable.
• 07:55 Bills: ipp016 had a panic this morning just power cycled it.

  <Oct/15 06:09 pm>ipp016 login: [1200542.035287] BUG: unable to handle kernel paging request at ffffffff800073d4
  <Oct/15 06:09 pm>[1200542.036617] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Oct/15 06:09 pm>[1200542.036617] PGD 203067 PUD 207063 PMD 0 
  <Oct/15 06:09 pm>[1200542.036617] Oops: 0000 [#1] SMP 
  <Oct/15 06:09 pm>[1200542.036617] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
  <Oct/15 06:09 pm>[1200542.036617] CPU 0 
  

• 10:20 EAM : repeated failures on stack_id 1618773 : unable to find PSF. I set quality & fault to 42

Wednesday : 2012-10-17

• 06:42 Bill : repaired stalled db replication on ippdb03 and ipp001
• 09:00 EAM : things were sluggish, so I've stopped and restarted all pantasks.

Thursday : 2012-10-18

• 02:45 (Serge): Fixed ipp001 and ippdb03 replication
• 09:45 (Serge): Bill rebooted sick ipp012. Serge set it to down for nebulous. neb-host ipp012 down
• 09:55 (Serge): neb-host ipp012 up

Friday : 2012-10-19

Saturday : 2012-10-20

Sunday : 2012-10-21