PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

• 11:30 MEH: ipp056 dumping regular messages, setting to repair until more info -- Haydn reports disk on way to failing, swapped out and raid rebuilding

  Feb 17 11:18:38 ipp056 MR_MONITOR[15606]: <MRMON113> Controller ID: 0  Unexpected sense: PD = --:--:10 - Unrecovered read error, CDB =  0x28  0x00  0x5b  0xc7  0xe8  0x8f  0x00  0x00  0x04  0x00 , Sense =  0xf0  0x00  0x03  0x5b  0xc7  0xe8  0x91  0x0a  0x00  0x00  0x00  0x00  0x11  0x00  0x00  0x00  0x00  0x00 
  Feb 17 11:18:39 ipp056 MR_MONITOR[15606]: <MRMON057> Controller ID: 0  Consistency Check corrected medium error: (VD 0 Location 0x5bc7e834, PD --:--:10 Location 0x5bc7e834)
  Feb 17 11:18:39 ipp056 MR_MONITOR[15606]: <MRMON110> Controller ID: 0  Corrected medium error during recovery: PD --:--:10 Location 0x5bc7e834
  
  

• 12:20 MEH: also looks like the ippdb02 mysql backup to ippc63.1 encountered odd state and aborted at 09:31.. will need to fix so can start mysql again on ippdb02..
  • script gets week number and of course 08,09 are problems as bash interprets 0x as octal notation..
  • modified script and crontab, will restart backup process 13:31 and mysql normally restarted when finished..

Tuesday : 2014-02-18

• 08:00 MEH: ippdb02 backup to ippc63 still running, should finish later today
• 11:33 Bill: reverted staticsky runs with fault 2 (33 runs) and 3 (11 runs) since the information in the logs indicates that the failures were due to cluster problems.
• 14:00 MEH: odd pstamp/update faults w/ data_group M31.rp.2013.20110102, chip.revert.off for a bit
• 14:11 CZW: repointed the LAP queue file to the active one. This has been off since the timeout issue last week (still unresolved). Sudden thought while typing this out: the point where it ran into problems was skycell.2211/i, which is the 3664th entry in the old queue file. Since it needs to check the previous entries to ensure they haven't been done yet, this can probably stretch out to 10 minutes just in checking old entries. I've cut the currently active queue down to a three hour slice, to see if that keeps the number of things to check in check.
• 18:00 MEH: ipp064 volume degraded, already in neb-host repair for dvo work
• 20:50 MEH: ipp023 is unresponsive.. looks like crashed little after 20:20, neb-host down.. -- nothing on console, like ipp026, another weak node with history of crashing..
  • 22:00 looks like ipp023 down has pissed off ippc01 as well -- display says general protection fault occurred, will need to power cycle as well..
• 23:35 MEH: ippc01 power cycled and back up, mess of stalled jobs and mounts cleared. restarting pantasks slowly
• 00:20 MEH: there is a mix of undocumented manual setting/changes not tracked for normal loading using the ~ipp/ippconfig/pantasks_hosts.input.... have to untangle before stdsci+stk+staticsky can be started..
• 00:30 MEH: summitcopy 500 read timeouts ongoing..

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD