*Disk reports *[wiki:2008-08-22 2008-08-22] *ipp004 * 2008-01-10T06:45:08 IDE Channel 12 Device Failed * 2008-03-20T00:07:08 IDE Channel 12 Device Failed * 2008-04-07T11:23:04 IDE Channel 12 Device Removed * 2008-04-07T11:38:09 IDE Channel 12 Device Failed * 2008-04-07T15:09:22 IDE Channel 12 Device Failed * 2008-05-08T06:32:55 IDE Channel 6 Device Failed * 2008-05-29T05:44:36 IDE Channel 12 Device Removed * 2008-05-29T11:34:28 IDE Channel 12 Device Removed * 2008-05-29T14:22:27 IDE Channel 12 Device Removed * 2008-05-29T14:43:31 IDE Channel 12 Device Failed * 2008-05-29T14:54:44 IDE Channel 12 Device Removed * 2008-05-29T15:10:57 IDE Channel 12 Device Failed * 2008-05-29T19:59:58 IDE Channel 12 Device Failed * (disk slot 12 has been determined to be broken) * 2008-08-20T18:20 (approx) network became unresponsive. notified by nagios. Power cycled by gavin * 2008-08-25T20:00 (approx) system became unresponsive. Power cycled by bills * 2008-08-30T15:00 (approx) sshd stopped responding. [wiki:Ipp004-debug-notes-20080830 debug notes] * 2008-08-31T22:00 (approx) sshd stopped responding. rebooted 2008-09-01T06:30 * 2008-09-02T18:04:33 network became unresponsive (eth0: too many iterations (6) in nv_nic_irq), notified by nagios. power cycled by gavin * 2008-09-02T19:00 (approx) ipp004 power cycled due to forcedeth failure *ipp005 * 2008-07-15T16:20:00 ethernet became unresponsive in manner consistent with forcedeth. problem power cycled by bills * 2008-07-15T17:15-30 (approx) system became unresponsive. power cycled by gavin * 2008-07-16T15:00 system became unresponsive power cycled by bills * 2008-07-17T20:45:32 IDE Channel 9 Device Removed * 2008-07-17T21:08:59 IDE Channel 9 Device Failed * 2008-07-17T23:31:08 IDE Channel 9 Device Removed * 2008-07-28T10:10:08 system became unresponsive. power cycled by gavin * 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin * 2008-08-18T17:15:00 system lost network connection. power cycled by EAM * 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam * 2008-09-12T16:55:00 total deadlock, power cycled * 2008-09-15T07:04 system unresponsive all night. console unresponsive. power cycled by bills * 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin * 2008-09-16T16:40 system deadlocked during fsck, power cycled * 2008-09-30T11:28 [wiki:Ipp005-2008-09-30-bug kernel BUG] in dmesg * 2008-10-14T06:00 system hung, no response on console, power cycled. * 2008-10-14T15:46 system hung, no response on console, power cycled. * 2008-10-15T09:30 system hung, no response on console, power cycled. * 2008-10-19T10:46 system hung, no response on console, no output power cycled by bills * 2008-12-11T12:17 system hung, no response on console, power cycled by gavin * 2009-04-16T14:08 system hung, no response on console, power cycled by bills * 2009-05-21T06:50 system hung, no response on console, power cycled by bills *ipp006 * 2008-07-31 early AM system unresponsive power cycled by eugene * 2008-08-01T21:20 approx system unresponsive. attempt to log into console got yp error power cycled by bills * 2008-08-18T12:00 kernel x87 math error? : [wiki:Ipp006-crash-20080818 kernel stack dump] * 2008-08-19T11:33 network hung, typical forcedeth.c YP/RPC errors on console * 2008-08-26T11:00 forcedeth * 2008-08-03AM network hung, console uresposive assume forcedeth power cycled by bills * 2008-08-03T11:01 network hung, console uresposive assume forcedeth power cycled by bills * 2008-09-03T12:06:26 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin * 2008-09-16T07:45? system unresponsive, power cycled by price * 2008-10-15T18:18 system became unresponsive console frozen last message [1071427.660233] do_IRQ: 1.179 No irq handler for vector. Power cycled by bills * 2008-10-16T08:18 nagios email messages about total processes and root partition * 2008-10-26T09:18 Host Down Alert. No response from console. No message. power cycled by bills * 2008-10-26T17:07 Host Down Alert. [wiki:Ipp006-crash-20081026 ipp006-crash-20081026] power cycled by eugene * 2008-11-28T09:22 Host Down Alert. error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin * 2008-12-18T16:16 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp006-crash-20081218 panic]; power cycled by gavin *ipp007 * 2008-07-24T20:50 system became unresponsive (nagios notification) power cycled by gavin * 2008-07-28 early AM system became unresponsive power cycled by eugene * 2008-08-10T12:30 (approx) forcedeth? unresponsive, power cycled by eugene *ipp008 * 2007-03-23T05:39:42 IDE Channel 1 Device Removed * 2008-08-09T06:38:00 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam) * 2008-08-10T02:29:00 system became unresponsive: error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam) * 2008-08-11T10:53:34 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin * 2008-08-11T14:46:55 system became unresponsive: notified by kaiser power cycled by gavin * 2008-08-11T15:28:28 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin * 2008-09-03T07:40 power cycled by bills due to network hang * 2008-09-05T05:04 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (many times, also 0.189, 2.189) (power cycled by eam) * 2008-10-19T12:30 system became unresponsive power cycled * 2008-10-20T22:00 system became unresponsive power cycled (nothing on console) * 2008-10-21T12:53:00 system became unresponsive power cycled by gavin (nothing on console) * 2008-10-21T14:05:00 system became unresponsive power cycled by gavin (nothing on console) * 2008-10-22T10:35:00 system became unresponsive power cycled by bills (nothing on console) * 2008-10-22T10:50:00 failed pretty quickly. (warps were running) power cycled by bills * 2008-10-30T12:22:00 system became unresponsive power cycled by gavin (nothing on console) * 2008-11-03T11:20 system became unresponsive power cycled by bills (nothing on console) Paul was processing on it. * 2008-12-13T17:13 system became unresponsive (time is nagios email) power cycled by bills 12-14T09:55 (nothing on console) * 2008-12-18T08:00 system became unresponsive power cycled by gavin (nothing on console) * 2008-12-19T14:39:00 crashed, no message on console, power cycled by eam & rodney * 2008-12-19T18:39 system unresponsive, no message on console, power cycled by gavin * 2008-12-20T17:33 crashed, no message on console, power cycled by rodney * 2008-12-21T16:40 crashed, no message on console, power cycled by rodney * 2009-03-01T14:00 system became unresponsive power cycled (nothing on console) * 2009-03-03T11:00 system became unresponsive power cycled (nothing on console) * 2009-03-05T13:40 system became unresponsive power cycled (nothing on console) *ipp009 * 2008-02-07T17:43:55 IDE Channel 5 Device Failed * 2008-07-12 time unknown forcedeth problem power cycled by bills * 2008-07-03T00:00 (approx) forcedeth problem power cycled around 0600 by bills * 2008-10-30T11:07:00 console hung, spinlock [wiki:Ipp009-crash-20081030 ipp009-crash-20081030] * 2008-11-15T15:22 no response nothing on console power cycled by bills *ipp010 * 2008-02-08T10:34:33 IDE Channel 12 Device Removed * 2008-02-08T10:44:12 IDE Channel 12 Device Removed * 2008-02-08T10:48:01 IDE Channel 12 Device Removed * 2008-07-21T11:45 system became unresponsive over network power cycled by bills * 2008-07-31T07:35 system became unresponsive over network power cycled by bills * 2008-09-10T16:24 unresponsive, power cycled by price * 2008-10-16T10:05 [wiki:Ipp010-crash-2008-10-16 kernel panic] * 2008-12-11T00:02 unresponsive no messages power cycled by bills around 08:55 * 2009-01-04T00:12:11 IDE Channel 1 Device Failed *ipp011 * 2007-03-23T05:51:14 IDE Channel 24 Device Failed * 2007-03-23T05:51:14 IDE Channel 22 Device Removed * 2007-03-23T05:51:14 IDE Channel 23 Device Removed * 2007-03-23T05:54:44 IDE Channel 1 Device Removed * 2008-07-10T10:15:00 - forcedeth NIC failure - RPC errors on console * 2008-07-15T07:00:00 approx forecdeth problem power cycled by bills * 2008-08-11T17:23:32 forecdeth problem power cycled by gavin *ipp012 * 2007-03-22T14:36:28 IDE Channel 15 Device Removed *ipp013 * 2006-01-01T00:56:09 IDE Channel 22 Device Removed * 2008-06-23T02:33:49 IDE Channel 11 Device Failed(SMART) * 2008-07-09T10:41:59 IDE Channel 11 Device Removed *ipp014 * 2008-07-10T00:42:03 IDE Channel 10 Device Failed(SMART) * 2008-07-10T02:23:17 IDE Channel 10 Device Removed * 2008-10-06T05:31 Machine became unresponsive. No message on console. Power cycled by bills 06:07 * 2008-10-16T05:51 kernel panic [wiki:Ipp014_panic_1016 ipp014_panic_1016] * 2008-10-15T22:30 machine became unresponsive nothing on console power cyled by bills 8:52 10/16 * 2008-10-20T20:49 kernel panic unable to handle kernel paging request [wiki:Ipp014_panic_1020 ipp014_panic_1020] * 2008-10-22T05:45 machine became unresponsive nothing on console power cycled by bills around 07:20 * 2008-10-25T11:55:39 machine hung (spinlock) [wiki:Ipp014-crash-20081025 ipp014-crash-20081025] * 2008-10-25T11:55:39 machine hung (spinlock) [wiki:Ipp014-crash-20081026 ipp014-crash-20081026] * 2008-10-31T00:00:00 machine hung, no stack trace (a few segfault warnings only) * 2008-11-05T14:10:00 machine hung (spinlock) * 2008-11-26T11:50 (approx) console unresponsive; [wiki:Ipp014-crash-2008-11-26 general protection fault] on console; power cycled by pap * 2008-11-26T11:00 (approx) console unresponsive; [wiki:Ipp014-crash-20081127 ipp014-crash-20081127]; power cycled by eam * 2008-12-05 crashed overnight, [wiki:Ipp014-crash-2008-12-04 multiple traces produced]; power cycled by pap * 2008-12-11T01:54 unresponsive no messages power cycled by bills around 08:55 * 2009-03-02T17:00 (approx) console unresponsive; [wiki:ipp014-crash-20090302 ipp014-crash-20090302]; power cycled by eam * 2009-03-05T18:23 kernel panic [wiki:ipp014_crash-20090305 ipp014_crash-20090305]; power cycled by bills * 2009-03-11T06:00 kernel panic [wiki:ipp014_crash-20090311 ipp014_crash-20090311]; power cycled by eam * 2009-03-11T16:00 kernel panic [wiki:ipp014_crash-20090311b ipp014_crash-20090311b]; power cycled by eam * 2009-04-07T10:29 kernel panic [wiki:ipp014_crash20090407 ipp014_crash-20090407]; power cycled by bills *ipp015 * 2008-07-03T23:37:05 IDE Channel 21 Device Removed * 2008-07-04T01:27:23 IDE Channel 3 Device Failed * 2008-07-04T01:40:07 IDE Channel 3 Device Removed * 2008-07-04T01:43:16 IDE Channel 4 Device Failed * 2008-07-09T16:31:25 IDE Channel 7 Device Removed * 2008-03-22T23:47:35 IDE Channel 7 Device Removed * 2008-03-22T23:57:31 IDE Channel 7 Device Inserted * 2008-3-22T23:47:35 IDE Channel 7 Device Removed * 2008-3-22T23:57:31 IDE Channel 7 Device Inserted * 2008-3-23T12:27:16 ARC-1280-VOL#00 Complete Rebuild 012:29:44 * 2008-07-09 ipp015 disk set moved into a wave #2 chassis (ipp025) * 2009-04-24T08:00 ipp015 cannot access /data/ipp032.0 [[BR]] no umount command in ipp032:/var/log/messages until attempt by bills to umount by hand around 7:58 am [[BR]] last mount from ipp015 Apr 22. [[BR]] tried to reboot around 11am hung during shutdown trying to unmount /data/ipp032.0[[BR]] power cycled by bills *ipp016 * 2008-02-26T10:35:50 IDE Channel 13 Device Removed * 2008-09-10T17:48 Looks like [wiki:Ipp016-crash-20080910 forcedeth], rebooted by price * 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin * i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills * 2008-10-08T15:53 [wiki:Ipp016-crash-20081008 soft lockup] * 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin * 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin * 2008-11-03T18:04 system unresponsive while processing stacks; [wiki:Ipp016-crash-20081103 panic] on console (looks like spinlock problem); power cycled by pap * 2008-11-12T17:00 ; power cycled [768330.447278] do_IRQ: 2.187 No irq handler for vector [768402.246842] do_IRQ: 0.179 No irq handler for vector * 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia). Error message: [1629646.647189] do_IRQ: 2.179 No irq handler for vector. Power cycled by pap. * 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp016-crash-20081218 panic]; power cycled by gavin * 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin * 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin * 2008-12-20T17:30 crashed, no message on console, power cycled by rodney * 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin * 2009-01-12T13:32:31] system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin * 2009-01-13T12:24 system became unresponsive No error messages power cycled by bills * 2009-01-13 system went down two more times. The second time a kernel panic occured due to a null pointer dereference [wiki:Ipp016-crash090113 ipp016-crash090113]. * 2009-01-20 [wiki:Ipp016-crash-2009-01-20 kernel oops] kernel panic, power cycled. * 2009-01-23 [wiki:Ipp016-crash-2009-01-23 kernel BUG] dead lock, power cycled. * 2009-01-23T14:17 [wiki:Ipp016-crash-2009-01-23T14:17 kernel panic] kernel panic, power cycled. * 2009-01-28T07:07 [wiki:Ipp016-crash-2009-01-28T07:07 kernel panic] kernel panic, power cycled. * 2009-01-28T11:21 [wiki:Ipp016-crash-2009-01-28T11:21 do_IRQ] do_IRQ on console, YP/RPC errors being printed on console, power cycled. * 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin * 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00 * 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene * 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled. * 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled * 2009-03-05T13:50 do_IRQ: 2.179 No irq handler for vector; power cycled by EAM * 2009-03-12T11:10 no console messages; power cycled by EAM * 2009-04-22T11:37 system unable to mount /data/ipp004.0 bill hung console trying so power cycled. * 2009-05-15T12:08 do_IRQ error; power cycled by PAP *ipp017 *ipp018 * 2008-01-09T07:52:30 IDE Channel 23 Device Removed * 2008-01-09T07:52:59 IDE Channel 3 Device Removed * 2008-01-09T07:52:59 IDE Channel 4 Device Removed * 2008-01-09T07:53:39 IDE Channel 3 Device Removed * 2008-01-09T07:53:42 IDE Channel 4 Device Removed * 2008-01-09T07:59:26 IDE Channel 23 Device Removed * 2008-08-07T00:00 (approx) ipp018 network became unresponsive. Power cycled by bills * 2008-08-13T16:54 (~) classical forcedeth network lockup. Power cycled * 2008-08-15T06:00 forcedeth : [wiki:Ipp008-crash-20080815 kernel stack dump] * 2008-08-17T21:05 forcedeth : [wiki:Ipp008-crash-20080817 kernel stack dump] * 2008-08-19T21:41 notified by kaiser to reboot, required fsck /dev/sda3 by gavin * 2008-09-11T1335 [wiki:Ipp018-crash-2008-09-11 kernel oops] * 2008-09-15T11:00 [wiki:Ipp018-crash-2008-09-15 kernel oops]; ssh was dead but was able to login and attempt to reboot; reboot hung forever... had to power cycle * 2008-12-19T14:45 crashed with kernel panic : stack trace [wiki:Ipp018-crash-20081219 ipp018-crash-20081219] * 2009-04-20T01:00 (approx) crash with stack trace [wiki:ipp018-crash-20090420 ipp018-crash-20090420] * 2009-04-23T11:10 panic with stack trace [wiki:ipp018-crash-20090424 ipp018-crash-20090423] * 2009-04-24T16:30 panic with stack trace [wiki:ipp018-crash-20090424b ipp018-crash-20090424] * 2009-04-26T04:30 panic with stack trace [wiki:ipp018-crash-20090426 ipp018-crash-20090426] * 2009-05-07T15:15 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin * 2009-05-15T12:12 crashed [wiki:ipp018-crash-20090515]; power cycled by PAP * 2009-05-18T11:41:14 crashed [wiki:ipp018-crash-20090518]; power cycled by gavin *ipp019 * 2008-09-03T22:10:27 found ipp019@ippcon booted from livecd, notified by nagios. unable to boot from raidset *ipp020 * 2007-06-21T07:02:03 IDE Channel 19 Device Failed * 2007-06-21T07:02:09 IDE Channel 19 Device Failed * 2007-06-21T07:02:16 IDE Channel 20 Device Removed * 2008-07-15:12:00:00 (approx) system unresponsive *** power cycled * 2008-08-20T17:51:57 forcdeth network death * 2008-09-10T12:50:00 (approx) system unresponsive *** power cycled * 2008-09-16T21:58 kernel panic power cycled by bills around 2008-09-17T07:00 * 2008-09-19T04:49 [172920.752681] Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills at 06:53 failure occured while iqanalysis was starting up processing *ipp021 * 2008-08-26T17:00 forcedeth hang power cycled by bills * 2008-09-10T18:48 forcedeth problem power cycled by gavin @ ~19:31 * 2008-11-05T18:49 load went to 11, console hung. power cycled by eam *ipp023 * 2009-02-20T13:03:49 IDE Channel 6 Reading Error - Reseated 2009-02-20T13:34:50 *ipp025 * 2008-10-21T11:36 power cycled by gavin [wiki:Ipp025-nfs-2008-10-20 BUG: soft lockup] * 2008-10-22T12:17 console hung message: [85568.599373] BUG: spinlock lockup on CPU#3, pswarp/14919, ffff88013624a100 power cycled by bills * 2008-10-22T14:16:36 console messages: [wiki:Ipp025-crash-2008-10-22 BUG: spinlock on CPU#2] * 2008-10-24T16:29:00 console hung: [wiki:Ipp025-crash-2008-10-24 ipp025-crash-2008-10-24] * 2008-10-24T16:29:00 console hung: [wiki:Ipp025-crash-20081026 ipp025-crash-20081026] * 2008-10-26T15:45 kernel panic [wiki:Ipp025-crash-20081026b ipp025-crash-20081026b] power cycled by bills * 2008-10-30T10:59:00 console hung, spinlock: [wiki:Ipp025-crash-20081030 ipp025-crash-20081030] * 2008-11-03T10:07 down (do_IRQ: 0.179 No irq handler for vector) during pap MD processing (ppStack 4 threads), power cycled by pap * Suspect [wiki:Ipp025-20081103 memory problems] * 2009-05-10T12:00 down [wiki:ipp025-crash-20090510 ipp025-crash-20090510] *ipp027 * 2008-09-21T21:00 (approx) [wiki:Ipp027-crash-2008-09-21 kernel oops] * 2008-10-04T09:30 machine non-responsive. Console frozen with no messages. Power cycled by bills. * 2008-11-15T15:57 no response nothing on console power cycled by bills * 2008-11-15T22:20 no response nothing on console power cycled by bills * 2008-11-20T00 (approx) no response on console, power cycled by pap * 2008-11-26T11:06 no response on console, power cycled by pap; was under heavy load * 2008-11-27T08:30 no response on console, power cycled by bills; was under heavy load * 2009-01-23T14:15 no response on console; was under heavy load * 2009-02-19T08:35:05 IDE Channel 20 Device Failed * 2009-03-23T10:22:50 IDE Channel 20 Device Failed *ipp028 * 2008-09-22T16:30 [wiki:Ipp028-oops-2008-09-22 ipp028-oops-2008-09-22] * 2008-09-26T10:40 [wiki:Ipp028-oops-2008-09-26 ipp028-oops-2008-09-26] * 2008-10-23T05:50:09 IDE Channel 21 Device Failed *ipp030 * 2009-02-20T10:03:49 IDE Channel 17 Reading Error - Reseated 2009-02-20T10:34:16 *ipp033 * 2008-09-22T04:00 (approx) [wiki:Ipp033-crash-2008-09-22 kernel oops] *ipp036 * 2008-10-29T06:34:58 IDE Channel 3 Device Failed *ipp039 * 2009-05-04T14:52:43 IDE Channel 17 Device Failed * ASA serial<->node name map 
Note all nums start with 190314xx (where "xx" is the two digits).

ipp004: 07
ipp005: 10
ipp006: 12
ipp007: 11
ipp008: 01
ipp009: 13
ipp010: 09
ipp011: 08
ipp012: 02
ipp013: 04
ipp014: 03
ipp016: 05
ipp018: 14
ipp025: 06
ipp037: 00