IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:Production_Cluster

Version 46 (modified by jhoblitt, 17 years ago) ( diff )

--

*Disk reports

*2008-08-22

*ipp008

  • 2007-03-23T05:39:42 IDE Channel 1 Device Removed
  • 2008-08-09T06:38:00 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
  • 2008-08-10T02:29:00 system became unresponsive: error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
  • 2008-08-11T10:53:34 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
  • 2008-08-11T14:46:55 system became unresponsive: notified by kaiser power cycled by gavin
  • 2008-08-11T15:28:28 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
  • 2008-09-03T07:40 power cycled by bills due to network hang
  • 2008-09-05T05:04 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (many times, also 0.189, 2.189) (power cycled by eam)
  • 2008-10-19T12:30 system became unresponsive power cycled
  • 2008-10-20T22:00 system became unresponsive power cycled (nothing on console)
  • 2008-10-21T12:53:00 system became unresponsive power cycled by gavin (nothing on console)
  • 2008-10-21T14:05:00 system became unresponsive power cycled by gavin (nothing on console)
  • 2008-10-22T10:35:00 system became unresponsive power cycled by bills (nothing on console)
  • 2008-10-22T10:50:00 failed pretty quickly. (warps were running) power cycled by bills
  • 2008-10-30T12:22:00 system became unresponsive power cycled by gavin (nothing on console)
  • 2008-11-03T11:20 system became unresponsive power cycled by bills (nothing on console) Paul was processing on it.
  • 2008-12-13T17:13 system became unresponsive (time is nagios email) power cycled by bills 12-14T09:55 (nothing on console)
  • 2008-12-18T08:00 system became unresponsive power cycled by gavin (nothing on console)
  • 2008-12-19T14:39:00 crashed, no message on console, power cycled by eam & rodney
  • 2008-12-19T18:39 system unresponsive, no message on console, power cycled by gavin
  • 2008-12-20T17:33 crashed, no message on console, power cycled by rodney
  • 2008-12-21T16:40 crashed, no message on console, power cycled by rodney
  • 2009-03-01T14:00 system became unresponsive power cycled (nothing on console)
  • 2009-03-03T11:00 system became unresponsive power cycled (nothing on console)
  • 2009-03-05T13:40 system became unresponsive power cycled (nothing on console)
  • 2009-06-15T17:13:13 IDE Channel 23 Device Removed

*ipp009

  • 2008-02-07T17:43:55 IDE Channel 5 Device Failed
  • 2008-07-12 time unknown forcedeth problem power cycled by bills
  • 2008-07-03T00:00 (approx) forcedeth problem power cycled around 0600 by bills
  • 2008-10-30T11:07:00 console hung, spinlock ipp009-crash-20081030
  • 2008-11-15T15:22 no response nothing on console power cycled by bills
  • 2009-06-11T08:39 system became unresponsive connected to console do_IRQ error; power cycled by bills

*ipp010

  • 2008-02-08T10:34:33 IDE Channel 12 Device Removed
  • 2008-02-08T10:44:12 IDE Channel 12 Device Removed
  • 2008-02-08T10:48:01 IDE Channel 12 Device Removed
  • 2008-07-21T11:45 system became unresponsive over network power cycled by bills
  • 2008-07-31T07:35 system became unresponsive over network power cycled by bills
  • 2008-09-10T16:24 unresponsive, power cycled by price
  • 2008-10-16T10:05 kernel panic
  • 2008-12-11T00:02 unresponsive no messages power cycled by bills around 08:55
  • 2009-01-04T00:12:11 IDE Channel 1 Device Failed

*ipp011

  • 2007-03-23T05:51:14 IDE Channel 24 Device Failed
  • 2007-03-23T05:51:14 IDE Channel 22 Device Removed
  • 2007-03-23T05:51:14 IDE Channel 23 Device Removed
  • 2007-03-23T05:54:44 IDE Channel 1 Device Removed
  • 2008-07-10T10:15:00 - forcedeth NIC failure - RPC errors on console
  • 2008-07-15T07:00:00 approx forecdeth problem power cycled by bills
  • 2008-08-11T17:23:32 forecdeth problem power cycled by gavin

*ipp012

  • 2007-03-22T14:36:28 IDE Channel 15 Device Removed

*ipp013

  • 2006-01-01T00:56:09 IDE Channel 22 Device Removed
  • 2008-06-23T02:33:49 IDE Channel 11 Device Failed(SMART)
  • 2008-07-09T10:41:59 IDE Channel 11 Device Removed

*ipp014

  • 2008-07-10T00:42:03 IDE Channel 10 Device Failed(SMART)
  • 2008-07-10T02:23:17 IDE Channel 10 Device Removed
  • 2008-10-06T05:31 Machine became unresponsive. No message on console. Power cycled by bills 06:07
  • 2008-10-16T05:51 kernel panic ipp014_panic_1016
  • 2008-10-15T22:30 machine became unresponsive nothing on console power cyled by bills 8:52 10/16
  • 2008-10-20T20:49 kernel panic unable to handle kernel paging request ipp014_panic_1020
  • 2008-10-22T05:45 machine became unresponsive nothing on console power cycled by bills around 07:20
  • 2008-10-25T11:55:39 machine hung (spinlock) ipp014-crash-20081025
  • 2008-10-25T11:55:39 machine hung (spinlock) ipp014-crash-20081026
  • 2008-10-31T00:00:00 machine hung, no stack trace (a few segfault warnings only)
  • 2008-11-05T14:10:00 machine hung (spinlock)
  • 2008-11-26T11:50 (approx) console unresponsive; general protection fault on console; power cycled by pap
  • 2008-11-26T11:00 (approx) console unresponsive; ipp014-crash-20081127; power cycled by eam
  • 2008-12-05 crashed overnight, multiple traces produced; power cycled by pap
  • 2008-12-11T01:54 unresponsive no messages power cycled by bills around 08:55
  • 2009-03-02T17:00 (approx) console unresponsive; ipp014-crash-20090302; power cycled by eam
  • 2009-03-05T18:23 kernel panic ipp014_crash-20090305; power cycled by bills
  • 2009-03-11T06:00 kernel panic ipp014_crash-20090311; power cycled by eam
  • 2009-03-11T16:00 kernel panic ipp014_crash-20090311b; power cycled by eam
  • 2009-04-07T10:29 kernel panic ipp014_crash-20090407; power cycled by bills

*ipp015

  • 2008-07-03T23:37:05 IDE Channel 21 Device Removed
  • 2008-07-04T01:27:23 IDE Channel 3 Device Failed
  • 2008-07-04T01:40:07 IDE Channel 3 Device Removed
  • 2008-07-04T01:43:16 IDE Channel 4 Device Failed
  • 2008-07-09T16:31:25 IDE Channel 7 Device Removed
  • 2008-03-22T23:47:35 IDE Channel 7 Device Removed
  • 2008-03-22T23:57:31 IDE Channel 7 Device Inserted
  • 2008-3-22T23:47:35 IDE Channel 7 Device Removed
  • 2008-3-22T23:57:31 IDE Channel 7 Device Inserted
  • 2008-3-23T12:27:16 ARC-1280-VOL#00 Complete Rebuild 012:29:44
  • 2008-07-09 ipp015 disk set moved into a wave #2 chassis (ipp025)
  • 2009-04-24T08:00 ipp015 cannot access /data/ipp032.0

no umount command in ipp032:/var/log/messages until attempt by bills to umount by hand around 7:58 am
last mount from ipp015 Apr 22.
tried to reboot around 11am hung during shutdown trying to unmount /data/ipp032.0
power cycled by bills

*ipp016

  • 2008-02-26T10:35:50 IDE Channel 13 Device Removed
  • 2008-09-10T17:48 Looks like forcedeth, rebooted by price
  • 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
  • <Sep/27 03:10 pm>i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills
  • 2008-10-08T15:53 soft lockup
  • 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
  • 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
  • 2008-11-03T18:04 system unresponsive while processing stacks; panic on console (looks like spinlock problem); power cycled by pap
  • 2008-11-12T17:00 ; power cycled

<Nov/12 02:06 pm>[768330.447278] do_IRQ: 2.187 No irq handler for vector <Nov/12 02:07 pm>[768402.246842] do_IRQ: 0.179 No irq handler for vector

  • 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia). Error message: <code>[1629646.647189] do_IRQ: 2.179 No irq handler for vector</code>. Power cycled by pap.
  • 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; panic; power cycled by gavin
  • 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin
  • 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin
  • 2008-12-20T17:30 crashed, no message on console, power cycled by rodney
  • 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin
  • 2009-01-12T13:32:31] system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
  • 2009-01-13T12:24 system became unresponsive No error messages power cycled by bills
  • 2009-01-13 system went down two more times. The second time a kernel panic occured due to a null pointer dereference ipp016-crash090113.
  • 2009-01-20 kernel oops kernel panic, power cycled.
  • 2009-01-23 kernel BUG dead lock, power cycled.
  • 2009-01-23T14:17 kernel panic kernel panic, power cycled.
  • 2009-01-28T07:07 kernel panic kernel panic, power cycled.
  • 2009-01-28T11:21 do_IRQ do_IRQ on console, YP/RPC errors being printed on console, power cycled.
  • 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin
  • 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00
  • 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene
  • 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled.
  • 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled
  • 2009-03-05T13:50 do_IRQ: 2.179 No irq handler for vector; power cycled by EAM
  • 2009-03-12T11:10 no console messages; power cycled by EAM
  • 2009-04-22T11:37 system unable to mount /data/ipp004.0 bill hung console trying so power cycled.
  • 2009-05-15T12:08 do_IRQ error; power cycled by PAP
  • 2009-06-08T23 do_IRQ: 1.189 No irq handler for vector; power cycled by bills

*ipp017

*ipp018

*ipp019

  • 2008-09-03T22:10:27 found ipp019@ippcon booted from livecd, notified by nagios. unable to boot from raidset

*ipp020

  • 2007-06-21T07:02:03 IDE Channel 19 Device Failed
  • 2007-06-21T07:02:09 IDE Channel 19 Device Failed
  • 2007-06-21T07:02:16 IDE Channel 20 Device Removed
  • 2008-07-15:12:00:00 (approx) system unresponsive * power cycled
  • 2008-08-20T17:51:57 forcdeth network death
  • 2008-09-10T12:50:00 (approx) system unresponsive * power cycled
  • 2008-09-16T21:58 kernel panic power cycled by bills around 2008-09-17T07:00
  • 2008-09-19T04:49 [172920.752681] Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills at 06:53 failure occured while iqanalysis was starting up processing

*ipp021

  • 2008-08-26T17:00 forcedeth hang power cycled by bills
  • 2008-09-10T18:48 forcedeth problem power cycled by gavin @ ~19:31
  • 2008-11-05T18:49 load went to 11, console hung. power cycled by eam

*ipp023

  • 2009-02-20T13:03:49 IDE Channel 6 Reading Error - Reseated 2009-02-20T13:34:50
  • 2009-06-01T11:40:13 IDE Channel 10 Device Failed

*ipp025

*ipp027

  • 2008-09-21T21:00 (approx) kernel oops
  • 2008-10-04T09:30 machine non-responsive. Console frozen with no messages. Power cycled by bills.
  • 2008-11-15T15:57 no response nothing on console power cycled by bills
  • 2008-11-15T22:20 no response nothing on console power cycled by bills
  • 2008-11-20T00 (approx) no response on console, power cycled by pap
  • 2008-11-26T11:06 no response on console, power cycled by pap; was under heavy load
  • 2008-11-27T08:30 no response on console, power cycled by bills; was under heavy load
  • 2009-01-23T14:15 no response on console; was under heavy load
  • 2009-02-19T08:35:05 IDE Channel 20 Device Failed
  • 2009-03-23T10:22:50 IDE Channel 20 Device Failed
  • 2009-06-01T06:23:53 Degraded unit: unit=0, port=1 - disk#2 failed

*ipp028

*ipp030

  • 2009-02-20T10:03:49 IDE Channel 17 Reading Error - Reseated 2009-02-20T10:34:16

*ipp033

*ipp036

  • 2008-10-29T06:34:58 IDE Channel 3 Device Failed

*ipp039

  • 2009-05-04T14:52:43 IDE Channel 17 Device Failed

*ipp045

  • 2009-06-26T15:15:00 [1559830.598447] Kernel panic - not syncing: Attempted to kill init!
  • ASA serial<->node name map

<pre> Note all nums start with 190314xx (where "xx" is the two digits).

ipp004: 07 ipp005: 10 ipp006: 12 ipp007: 11 ipp008: 01 ipp009: 13 ipp010: 09 ipp011: 08 ipp012: 02 ipp013: 04 ipp014: 03 ipp016: 05 ipp018: 14 ipp025: 06 ipp037: 00 </pre>

Note: See TracWiki for help on using the wiki.