| 4 | | *ipp004 |
| 5 | | * 2008-01-10T06:45:08 IDE Channel 12 Device Failed |
| 6 | | * 2008-03-20T00:07:08 IDE Channel 12 Device Failed |
| 7 | | * 2008-04-07T11:23:04 IDE Channel 12 Device Removed |
| 8 | | * 2008-04-07T11:38:09 IDE Channel 12 Device Failed |
| 9 | | * 2008-04-07T15:09:22 IDE Channel 12 Device Failed |
| 10 | | * 2008-05-08T06:32:55 IDE Channel 6 Device Failed |
| 11 | | * 2008-05-29T05:44:36 IDE Channel 12 Device Removed |
| 12 | | * 2008-05-29T11:34:28 IDE Channel 12 Device Removed |
| 13 | | * 2008-05-29T14:22:27 IDE Channel 12 Device Removed |
| 14 | | * 2008-05-29T14:43:31 IDE Channel 12 Device Failed |
| 15 | | * 2008-05-29T14:54:44 IDE Channel 12 Device Removed |
| 16 | | * 2008-05-29T15:10:57 IDE Channel 12 Device Failed |
| 17 | | * 2008-05-29T19:59:58 IDE Channel 12 Device Failed |
| 18 | | * (disk slot 12 has been determined to be broken) |
| 19 | | * 2008-08-20T18:20 (approx) network became unresponsive. notified by nagios. Power cycled by gavin |
| 20 | | * 2008-08-25T20:00 (approx) system became unresponsive. Power cycled by bills |
| 21 | | * 2008-08-30T15:00 (approx) sshd stopped responding. [wiki:Ipp004-debug-notes-20080830 debug notes] |
| 22 | | * 2008-08-31T22:00 (approx) sshd stopped responding. rebooted 2008-09-01T06:30 |
| 23 | | * 2008-09-02T18:04:33 network became unresponsive (eth0: too many iterations (6) in nv_nic_irq), notified by nagios. power cycled by gavin |
| 24 | | * 2008-09-02T19:00 (approx) ipp004 power cycled due to forcedeth failure |
| 25 | | * 2009-06-15T17:02:53 IDE Channel 11 Reading Error |
| 26 | | |
| 27 | | *ipp005 |
| 28 | | * 2008-07-15T16:20:00 ethernet became unresponsive in manner consistent with forcedeth. problem power cycled by bills |
| 29 | | * 2008-07-15T17:15-30 (approx) system became unresponsive. power cycled by gavin |
| 30 | | * 2008-07-16T15:00 system became unresponsive power cycled by bills |
| 31 | | * 2008-07-17T20:45:32 IDE Channel 9 Device Removed |
| 32 | | * 2008-07-17T21:08:59 IDE Channel 9 Device Failed |
| 33 | | * 2008-07-17T23:31:08 IDE Channel 9 Device Removed |
| 34 | | * 2008-07-28T10:10:08 system became unresponsive. power cycled by gavin |
| 35 | | * 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin |
| 36 | | * 2008-08-18T17:15:00 system lost network connection. power cycled by EAM |
| 37 | | * 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam |
| 38 | | * 2008-09-12T16:55:00 total deadlock, power cycled |
| 39 | | * 2008-09-15T07:04 system unresponsive all night. console unresponsive. power cycled by bills |
| 40 | | * 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin |
| 41 | | * 2008-09-16T16:40 system deadlocked during fsck, power cycled |
| 42 | | * 2008-09-30T11:28 [wiki:Ipp005-2008-09-30-bug kernel BUG] in dmesg |
| 43 | | * 2008-10-14T06:00 system hung, no response on console, power cycled. |
| 44 | | * 2008-10-14T15:46 system hung, no response on console, power cycled. |
| 45 | | * 2008-10-15T09:30 system hung, no response on console, power cycled. |
| 46 | | * 2008-10-19T10:46 system hung, no response on console, no output power cycled by bills |
| 47 | | * 2008-12-11T12:17 system hung, no response on console, power cycled by gavin |
| 48 | | * 2009-04-16T14:08 system hung, no response on console, power cycled by bills |
| 49 | | * 2009-05-21T06:50 system hung, no response on console, power cycled by bills |
| 50 | | |
| 51 | | *ipp006 |
| 52 | | * 2008-07-31 early AM system unresponsive power cycled by eugene |
| 53 | | * 2008-08-01T21:20 approx system unresponsive. attempt to log into console got yp error power cycled by bills |
| 54 | | * 2008-08-18T12:00 kernel x87 math error? : [wiki:Ipp006-crash-20080818 kernel stack dump] |
| 55 | | * 2008-08-19T11:33 network hung, typical forcedeth.c YP/RPC errors on console |
| 56 | | * 2008-08-26T11:00 forcedeth |
| 57 | | * 2008-08-03AM network hung, console uresposive assume forcedeth power cycled by bills |
| 58 | | * 2008-08-03T11:01 network hung, console uresposive assume forcedeth power cycled by bills |
| 59 | | * 2008-09-03T12:06:26 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin |
| 60 | | * 2008-09-16T07:45? system unresponsive, power cycled by price |
| 61 | | * 2008-10-15T18:18 system became unresponsive console frozen last message [1071427.660233] do_IRQ: 1.179 No irq handler for vector. Power cycled by bills |
| 62 | | * 2008-10-16T08:18 nagios email messages about total processes and root partition |
| 63 | | * 2008-10-26T09:18 Host Down Alert. No response from console. No message. power cycled by bills |
| 64 | | * 2008-10-26T17:07 Host Down Alert. [wiki:Ipp006-crash-20081026 ipp006-crash-20081026] power cycled by eugene |
| 65 | | * 2008-11-28T09:22 Host Down Alert. error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin |
| 66 | | * 2008-12-18T16:16 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp006-crash-20081218 panic]; power cycled by gavin |
| 67 | | * 2009-0611T06:53 panic do_IRQ: 2.179 No irq handler for vector; power cycled by bills |
| 68 | | |
| 69 | | *ipp007 |
| 70 | | * 2008-07-24T20:50 system became unresponsive (nagios notification) power cycled by gavin |
| 71 | | * 2008-07-28 early AM system became unresponsive power cycled by eugene |
| 72 | | * 2008-08-10T12:30 (approx) forcedeth? unresponsive, power cycled by eugene |
| | 4 | * [wiki:ipp004_log ipp004] |
| | 5 | * [wiki:ipp005_log ipp005] |
| | 6 | * [wiki:ipp006_log ipp006] |
| | 7 | * [wiki:ipp007_log ipp007] |
| | 8 | |
