* 2008-02-26T10:35:50 IDE Channel 13 Device Removed * 2008-09-10T17:48 Looks like [wiki:Ipp016-crash-20080910 forcedeth], rebooted by price * 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin * i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills * 2008-10-08T15:53 [wiki:Ipp016-crash-20081008 soft lockup] * 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin * 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin * 2008-11-03T18:04 system unresponsive while processing stacks; [wiki:Ipp016-crash-20081103 panic] on console (looks like spinlock problem); power cycled by pap * 2008-11-12T17:00 ; power cycled [768330.447278] do_IRQ: 2.187 No irq handler for vector [768402.246842] do_IRQ: 0.179 No irq handler for vector * 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia). Error message: [1629646.647189] do_IRQ: 2.179 No irq handler for vector. Power cycled by pap. * 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp016-crash-20081218 panic]; power cycled by gavin * 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin * 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin * 2008-12-20T17:30 crashed, no message on console, power cycled by rodney * 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin * 2009-01-12T13:32:31 system unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin * 2010-05-10T19:45:39 system unresponsive;. console message [ 4519.328798] do_IRQ: 2.189 No irq handler for vector; power cycled by gavin * 2009-01-13T12:24 system became unresponsive No error messages power cycled by bills * 2009-01-13 system went down two more times. The second time a kernel panic occured due to a null pointer dereference [wiki:Ipp016-crash090113 ipp016-crash090113]. * 2009-01-20 [wiki:Ipp016-crash-2009-01-20 kernel oops] kernel panic, power cycled. * 2009-01-23 [wiki:Ipp016-crash-2009-01-23 kernel BUG] dead lock, power cycled. * 2009-01-23T14:17 [wiki:Ipp016-crash-2009-01-23T14:17 kernel panic] kernel panic, power cycled. * 2009-01-28T07:07 [wiki:Ipp016-crash-2009-01-28T07:07 kernel panic] kernel panic, power cycled. * 2009-01-28T11:21 [wiki:Ipp016-crash-2009-01-28T11:21 do_IRQ] do_IRQ on console, YP/RPC errors being printed on console, power cycled. * 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin * 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00 * 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene * 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled. * 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled * 2009-03-05T13:50 do_IRQ: 2.179 No irq handler for vector; power cycled by EAM * 2009-03-12T11:10 no console messages; power cycled by EAM * 2009-04-22T11:37 system unable to mount /data/ipp004.0 bill hung console trying so power cycled. * 2009-05-15T12:08 do_IRQ error; power cycled by PAP * 2009-06-08T23 do_IRQ: 1.189 No irq handler for vector; power cycled by bills * 2009-12-08T05:36:06 node down not responding no console or log message power cycle cindy * 2009-12-29T13:13:51 IDE Channel Device 10 failed (replaced 2009-12-30T16:04:53) * 2010-05-10T12:46 power cycled by bills * ipp016 login: [12451489.477939] do_IRQ: 1.179 No irq handler for vector * [12451563.482352] BUG: spinlock lockup on CPU#1, ppSub/16197, ffff880160fec948 * [12453912.155470] do_IRQ: 0.189 No irq handler for vector * 2010-05-10T17:24:19 system down, unresponsive from the console, [ 5250.746281] do_IRQ: 2.179 No irq handler for vector, power cycled by gavin * 2010-05-10T19:45:39 system down, console message [ 4519.328798] do_IRQ: 2.189 No irq handler for vector; power cycled by gavin * 2010-05-11T14:06:30 system down, [wiki:Ipp016-crash-2010-05-11T14:06 do_IRQ]: 0.179 No irq handler for vector ; power cycled by gavin * 2010-05-28T07:44:16 IDE Channel !#12 Device Failed (replaced 2010-05-28T09:34:08) * 2010-05-28T10:00 Memory swap * 2010-06-26T11:01:14 IDE Channel 18 Device Failed (replaced 2010-06-28T09:28:54) * 2010-07-27T09:42:38 system unresponsive, [wiki:Ipp016-crash-20100727 ipp016-crash-20100727], power cycled by gavin * 2010-08-01T15:29:45 IDE Channel 8 Device Failed (replaced 2010-08-02T11:11:46) * 2010-08-08T04:43:43 IDE Channel 1 Device Failed (replaced 2010-08-09T09:37:40) * 2010-09-03T10:00 number of processes blew up system became unresponsive nothing on console power cycled by bills * 2010-09-30T17:00:30 IDE Channel 15 Device Failed (replaced 2010-09-30T18:00:30) * 2010-10-01T06:32:06 IDE Channel 23 Device Failed (replaced 2010-10-01T16:44:01) * 2010-10-3T9:52:32 DISK# 2 Device Failed replaced 2010-10-04 * 2010-10-06T23:41:12 system unresponsive, [wiki:Ipp016-crash-20101006 ipp016-crash-20101006], power cycled by gavin * 2010-10-07T09:58:12 system unresponsive, console [wiki:Ipp016-crash-20101007 messages]: [ 8757.412781] do_IRQ: 2.179 No irq handler for vector, power cycled by gavin * 2010-10-19 6:32:6 Drive # 7 Device Failed replaced rebuilt * 2010-11-05T06:34:04 IDE Channel 17 Device Failed (replaced 2010-11-05T09:41:21) * 2010-11-21 Channel # 20 Device Failed replaced rebuilt * 2010-11-25 4:4:32 Volume Degraded Channel # 9 Device Failed * 2010-12-15T08:10:02 IDE Channel 5 Device Failed (replaced 2010-12-15T09:18:56) * 2010-12-21T06:47:28 IDE Channel 14 Device Failed (replaced 2010-12-21T09:25:28) * 2010-12-31T20:42:51 IDE Channel 16 Device Failed (replaced 2011-01-03T11:27:36) * 2011-01-08T08:00:00 System down for over 7 hours [wiki:Ipp016-crash-2011-01-08] * 2011-08-04 10:11:1 Controller # 1(PCI) IDE Channel !#16 Reading Error replaced * 2011-11-10 9:5:9 Controller#1(PCI) IDE Channel !#2 Reading Error NOT REPLACED * 2011-11-16 19:18:42 Controller#1(PCI) IDE Channel !#2 Reading Error REPLACED * 2012-02-16 17:37:26 Controller#1(PCI) IDE Channel !#8 Reading Error * 2012-03-04 13:44:17 Controller#1(PCI) IDE Channel !#8 Reading Error REPLACED * 2012-07-10 17:29:20 Controller#1(PCI) IDE Channel !#21 Reading Error REPLACED * 2012-07-26T17:24:47 IDE Channel 24 Reading Error (replaced 2012-07-27T10:12:30) * 2012-08-28T15:43:56 IDE Channel 3 Reading Error (replaced 2012-08-28T17:06:44) * 2012-09-03 12:40 rebooted to clear stalled mounts (see [wiki:PS1_IPP_Czarlog_20120903 * 2012-09-11 04:47:15 Controller#1(PCI) IDE Channel !#15 Reading Error (replaced: 2012-9-11 9:22:29 Controller# 1(PCI) IDE Channel # 15 Device Inserted) * 2012-10-01 20:30 power cycled after not recovering mounts (see czar log [wiki:PS1_IPP_Czarlog_20121001] * 2012-10-06 07:32:20 Controller#1(PCI) IDE Channel !#19 Reading Error (replaced: 2012-10-06 08:48:45) * Note: The time reported by the Areca RAID controller is off by ~3.5 hours! 08:48:55 vs. 12:19:16 * 2012-10-16 07:55 Bills: ipp016 had a panic this morning just power cycled it. (from czar log [wiki:PS1_IPP_Czarlog_20121015]) {{{ ipp016 login: [1200542.035287] BUG: unable to handle kernel paging request at ffffffff800073d4 [1200542.036617] IP: [] xprt_autoclose+0x19/0x4c [1200542.036617] PGD 203067 PUD 207063 PMD 0 [1200542.036617] Oops: 0000 [#1] SMP [1200542.036617] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm [1200542.036617] CPU 0 }}} * 2012-10-29 22:30 GPF and couldn't ssh into, power cycled [wiki:Ipp016-crash-20121029]