IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of Production_Cluster


Ignore:
Timestamp:
Feb 24, 2009, 4:23:57 PM (17 years ago)
Author:
trac
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Production_Cluster

    v1 v1  
     1*Disk reports
     2  *[wiki:2008-08-22 2008-08-22]
     3
     4 *ipp004
     5  * 2008-01-10T06:45:08 IDE Channel 12 Device Failed 
     6  * 2008-03-20T00:07:08 IDE Channel 12 Device Failed 
     7  * 2008-04-07T11:23:04 IDE Channel 12 Device Removed 
     8  * 2008-04-07T11:38:09 IDE Channel 12 Device Failed 
     9  * 2008-04-07T15:09:22 IDE Channel 12 Device Failed 
     10  * 2008-05-08T06:32:55 IDE Channel 6 Device Failed 
     11  * 2008-05-29T05:44:36 IDE Channel 12 Device Removed 
     12  * 2008-05-29T11:34:28 IDE Channel 12 Device Removed 
     13  * 2008-05-29T14:22:27 IDE Channel 12 Device Removed 
     14  * 2008-05-29T14:43:31 IDE Channel 12 Device Failed 
     15  * 2008-05-29T14:54:44 IDE Channel 12 Device Removed 
     16  * 2008-05-29T15:10:57 IDE Channel 12 Device Failed 
     17  * 2008-05-29T19:59:58 IDE Channel 12 Device Failed 
     18  * (disk slot 12 has been determined to be broken)
     19  * 2008-08-20T18:20 (approx) network became unresponsive. notified by nagios. Power cycled by gavin
     20  * 2008-08-25T20:00 (approx) system became unresponsive. Power cycled by bills
     21  * 2008-08-30T15:00 (approx) sshd stopped responding.  [wiki:Ipp004-debug-notes-20080830 debug notes]
     22  * 2008-08-31T22:00 (approx) sshd stopped responding.  rebooted 2008-09-01T06:30
     23  * 2008-09-02T18:04:33 network became unresponsive (eth0: too many iterations (6) in nv_nic_irq), notified by nagios. power cycled by gavin
     24  * 2008-09-02T19:00 (approx) ipp004 power cycled due to forcedeth failure
     25
     26 *ipp005
     27  * 2008-07-15T16:20:00 ethernet became unresponsive in  manner consistent with forcedeth. problem power cycled by bills
     28  * 2008-07-15T17:15-30 (approx) system became unresponsive.  power cycled by gavin
     29  * 2008-07-16T15:00   system became unresponsive power cycled by bills
     30  * 2008-07-17T20:45:32 IDE Channel 9 Device Removed
     31  * 2008-07-17T21:08:59 IDE Channel 9 Device Failed
     32  * 2008-07-17T23:31:08 IDE Channel 9 Device Removed
     33  * 2008-07-28T10:10:08 system became unresponsive.  power cycled by gavin
     34  * 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin
     35  * 2008-08-18T17:15:00 system lost network connection.  power cycled by EAM
     36  * 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam
     37  * 2008-09-12T16:55:00 total deadlock, power cycled
     38  * 2008-09-15T07:04    system unresponsive all night. console unresponsive.  power cycled by bills
     39  * 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin
     40  * 2008-09-16T16:40    system deadlocked during fsck, power cycled
     41  * 2008-09-30T11:28    [wiki:Ipp005-2008-09-30-bug kernel BUG] in dmesg
     42  * 2008-10-14T06:00    system hung, no response on console, power cycled.
     43  * 2008-10-14T15:46    system hung, no response on console, power cycled.
     44  * 2008-10-15T09:30    system hung, no response on console, power cycled.
     45  * 2008-10-19T10:46    system hung, no response on console, no output power cycled by bills
     46  * 2008-12-11T12:17    system hung, no response on console, power cycled by gavin
     47
     48 *ipp006
     49  * 2008-07-31 early AM system unresponsive power cycled by eugene
     50  * 2008-08-01T21:20 approx system unresponsive. attempt to log into console got yp error power cycled by bills
     51  * 2008-08-18T12:00 kernel x87 math error? : [wiki:Ipp006-crash-20080818 kernel stack dump]
     52  * 2008-08-19T11:33 network hung, typical forcedeth.c YP/RPC errors on console
     53  * 2008-08-26T11:00 forcedeth
     54  * 2008-08-03AM     network hung, console uresposive assume forcedeth power cycled by bills
     55  * 2008-08-03T11:01 network hung, console uresposive assume forcedeth power cycled by bills
     56  * 2008-09-03T12:06:26 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
     57  * 2008-09-16T07:45? system unresponsive, power cycled by price
     58  * 2008-10-15T18:18 system became unresponsive console frozen last message [1071427.660233] do_IRQ: 1.179 No irq handler for vector. Power cycled by bills
     59  * 2008-10-16T08:18 nagios email messages about total processes and root partition
     60  * 2008-10-26T09:18 Host Down Alert. No response from console. No message. power cycled by bills
     61  * 2008-10-26T17:07 Host Down Alert. [wiki:Ipp006-crash-20081026 ipp006-crash-20081026] power cycled by eugene
     62  * 2008-11-28T09:22 Host Down Alert. error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
     63  * 2008-12-18T16:16 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp006-crash-20081218 panic]; power cycled by gavin
     64
     65 *ipp007
     66  * 2008-07-24T20:50   system became unresponsive (nagios notification) power cycled by gavin
     67  * 2008-07-28 early AM system became unresponsive power cycled by eugene
     68  * 2008-08-10T12:30 (approx) forcedeth?  unresponsive, power cycled by eugene
     69
     70 *ipp008
     71  * 2007-03-23T05:39:42 IDE Channel 1 Device Removed 
     72  * 2008-08-09T06:38:00 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
     73  * 2008-08-10T02:29:00 system became unresponsive: error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
     74  * 2008-08-11T10:53:34 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
     75  * 2008-08-11T14:46:55 system became unresponsive: notified by kaiser power cycled by gavin
     76  * 2008-08-11T15:28:28 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
     77  * 2008-09-03T07:40     power cycled by bills due to network hang
     78  * 2008-09-05T05:04    system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (many times, also 0.189, 2.189) (power cycled by eam)
     79  * 2008-10-19T12:30 system became unresponsive power cycled
     80  * 2008-10-20T22:00 system became unresponsive power cycled (nothing on console)
     81  * 2008-10-21T12:53:00 system became unresponsive power cycled by gavin (nothing on console)
     82  * 2008-10-21T14:05:00 system became unresponsive power cycled by gavin (nothing on console)
     83  * 2008-10-22T10:35:00 system became unresponsive power cycled by bills (nothing on console)
     84  * 2008-10-22T10:50:00 failed pretty quickly. (warps were running) power cycled by bills
     85  * 2008-10-30T12:22:00 system became unresponsive power cycled by gavin (nothing on console)
     86  * 2008-11-03T11:20 system became unresponsive power cycled by bills (nothing on console) Paul was processing on it.
     87  * 2008-12-13T17:13 system became unresponsive (time is nagios email) power cycled by bills 12-14T09:55 (nothing on console)
     88  * 2008-12-18T08:00 system became unresponsive power cycled by gavin (nothing on console)
     89  * 2008-12-19T14:39:00 crashed, no message on console, power cycled by eam & rodney
     90  * 2008-12-19T18:39 system unresponsive, no message on console, power cycled by gavin
     91  * 2008-12-20T17:33 crashed, no message on console, power cycled by rodney
     92  * 2008-12-21T16:40 crashed, no message on console, power cycled by rodney
     93
     94 *ipp009
     95  * 2008-02-07T17:43:55 IDE Channel 5 Device Failed 
     96  * 2008-07-12 time unknown forcedeth problem power cycled by bills
     97  * 2008-07-03T00:00 (approx) forcedeth problem power cycled around 0600 by bills
     98  * 2008-10-30T11:07:00 console hung, spinlock [wiki:Ipp009-crash-20081030 ipp009-crash-20081030]
     99  * 2008-11-15T15:22  no response nothing on console power cycled by bills
     100
     101 *ipp010
     102  * 2008-02-08T10:34:33 IDE Channel 12 Device Removed 
     103  * 2008-02-08T10:44:12 IDE Channel 12 Device Removed 
     104  * 2008-02-08T10:48:01 IDE Channel 12 Device Removed 
     105  * 2008-07-21T11:45   system became unresponsive over network power cycled by bills
     106  * 2008-07-31T07:35   system became unresponsive over network power cycled by bills
     107  * 2008-09-10T16:24 unresponsive, power cycled by price
     108  * 2008-10-16T10:05 [wiki:Ipp010-crash-2008-10-16 kernel panic]
     109  * 2008-12-11T00:02 unresponsive no messages power cycled by bills around 08:55
     110  * 2009-01-04T00:12:11 IDE Channel 1 Device Failed
     111
     112 *ipp011
     113  * 2007-03-23T05:51:14 IDE Channel 24 Device Failed 
     114  * 2007-03-23T05:51:14 IDE Channel 22 Device Removed 
     115  * 2007-03-23T05:51:14 IDE Channel 23 Device Removed 
     116  * 2007-03-23T05:54:44 IDE Channel 1 Device Removed 
     117  * 2008-07-10T10:15:00 - forcedeth NIC failure - RPC errors on console
     118  * 2008-07-15T07:00:00 approx forecdeth problem power cycled by bills
     119  * 2008-08-11T17:23:32 forecdeth problem power cycled by gavin
     120
     121 *ipp012
     122  * 2007-03-22T14:36:28 IDE Channel 15 Device Removed 
     123
     124 *ipp013
     125  * 2006-01-01T00:56:09 IDE Channel 22 Device Removed 
     126  * 2008-06-23T02:33:49 IDE Channel 11 Device Failed(SMART) 
     127  * 2008-07-09T10:41:59 IDE Channel 11 Device Removed 
     128
     129 *ipp014
     130  * 2008-07-10T00:42:03 IDE Channel 10 Device Failed(SMART) 
     131  * 2008-07-10T02:23:17 IDE Channel 10 Device Removed 
     132  * 2008-10-06T05:31    Machine became unresponsive. No message on console. Power cycled by bills 06:07
     133  * 2008-10-16T05:51 kernel panic [wiki:Ipp014_panic_1016 ipp014_panic_1016]
     134  * 2008-10-15T22:30 machine became unresponsive nothing on console power cyled by bills 8:52 10/16
     135  * 2008-10-20T20:49 kernel panic unable to handle kernel paging request [wiki:Ipp014_panic_1020 ipp014_panic_1020]
     136  * 2008-10-22T05:45 machine became unresponsive nothing on console power cycled by bills around 07:20
     137  * 2008-10-25T11:55:39 machine hung (spinlock) [wiki:Ipp014-crash-20081025 ipp014-crash-20081025]
     138  * 2008-10-25T11:55:39 machine hung (spinlock) [wiki:Ipp014-crash-20081026 ipp014-crash-20081026]
     139  * 2008-10-31T00:00:00 machine hung, no stack trace (a few segfault warnings only)
     140  * 2008-11-05T14:10:00 machine hung (spinlock)
     141  * 2008-11-26T11:50 (approx) console unresponsive; [wiki:Ipp014-crash-2008-11-26 general protection fault] on console; power cycled by pap
     142  * 2008-11-26T11:00 (approx) console unresponsive; [wiki:Ipp014-crash-20081127 ipp014-crash-20081127]; power cycled by eam
     143  * 2008-12-05 crashed overnight, [wiki:Ipp014-crash-2008-12-04 multiple traces produced]; power cycled by pap
     144  * 2008-12-11T01:54 unresponsive no messages power cycled by bills around 08:55
     145
     146 *ipp015
     147  * 2008-07-03T23:37:05 IDE Channel 21 Device Removed 
     148  * 2008-07-04T01:27:23 IDE Channel 3 Device Failed 
     149  * 2008-07-04T01:40:07 IDE Channel 3 Device Removed 
     150  * 2008-07-04T01:43:16 IDE Channel 4 Device Failed 
     151  * 2008-07-09T16:31:25 IDE Channel 7 Device Removed 
     152  * 2008-03-22T23:47:35 IDE Channel 7 Device Removed
     153  * 2008-03-22T23:57:31 IDE Channel 7 Device Inserted
     154  * 2008-3-22T23:47:35 IDE Channel 7 Device Removed
     155  * 2008-3-22T23:57:31 IDE Channel 7 Device Inserted
     156  * 2008-3-23T12:27:16 ARC-1280-VOL#00 Complete Rebuild 012:29:44
     157  * 2008-07-09 ipp015 disk set moved into a wave #2 chassis (ipp025)
     158
     159 *ipp016
     160  * 2008-02-26T10:35:50 IDE Channel 13 Device Removed 
     161  * 2008-09-10T17:48 Looks like [wiki:Ipp016-crash-20080910 forcedeth], rebooted by price
     162  * 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
     163  * <Sep/27 03:10 pm>i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills
     164  * 2008-10-08T15:53 [wiki:Ipp016-crash-20081008 soft lockup]
     165  * 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
     166  * 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
     167  * 2008-11-03T18:04 system unresponsive while processing stacks; [wiki:Ipp016-crash-20081103 panic] on console (looks like spinlock problem); power cycled by pap
     168  * 2008-11-12T17:00 ; power cycled
     169<Nov/12 02:06 pm>[768330.447278] do_IRQ: 2.187 No irq handler for vector
     170<Nov/12 02:07 pm>[768402.246842] do_IRQ: 0.179 No irq handler for vector
     171  * 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia).  Error message: <code>[1629646.647189] do_IRQ: 2.179 No irq handler for vector</code>.  Power cycled by pap.
     172  * 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp016-crash-20081218 panic]; power cycled by gavin
     173  * 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin
     174  * 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin
     175  * 2008-12-20T17:30 crashed, no message on console, power cycled by rodney
     176  * 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin
     177  * 2009-01-12T13:32:31] system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
     178  * 2009-01-13T12:24  system became unresponsive No error messages power cycled by bills
     179  * 2009-01-13   system went down two more times. The second time a kernel panic occured due to a null pointer dereference [wiki:Ipp016-crash090113 ipp016-crash090113].
     180  * 2009-01-20 [wiki:Ipp016-crash-2009-01-20 kernel oops] kernel panic, power cycled.
     181  * 2009-01-23 [wiki:Ipp016-crash-2009-01-23 kernel BUG] dead lock, power cycled.
     182  * 2009-01-23T14:17 [wiki:Ipp016-crash-2009-01-23T14:17 kernel panic] kernel panic, power cycled.
     183  * 2009-01-28T07:07 [wiki:Ipp016-crash-2009-01-28T07:07 kernel panic] kernel panic, power cycled.
     184  * 2009-01-28T11:21 [wiki:Ipp016-crash-2009-01-28T11:21 do_IRQ] do_IRQ on console, YP/RPC errors being printed on console, power cycled.
     185  * 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin
     186  * 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector  BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00
     187  * 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene
     188  * 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled.
     189  * 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled
     190
     191 *ipp017
     192
     193 *ipp018
     194  * 2008-01-09T07:52:30 IDE Channel 23 Device Removed 
     195  * 2008-01-09T07:52:59 IDE Channel 3 Device Removed 
     196  * 2008-01-09T07:52:59 IDE Channel 4 Device Removed 
     197  * 2008-01-09T07:53:39 IDE Channel 3 Device Removed 
     198  * 2008-01-09T07:53:42 IDE Channel 4 Device Removed 
     199  * 2008-01-09T07:59:26 IDE Channel 23 Device Removed 
     200  * 2008-08-07T00:00 (approx) ipp018 network became unresponsive. Power cycled by bills
     201  * 2008-08-13T16:54 (~) classical forcedeth network lockup.  Power cycled
     202  * 2008-08-15T06:00 forcedeth : [wiki:Ipp008-crash-20080815 kernel stack dump]
     203  * 2008-08-17T21:05 forcedeth : [wiki:Ipp008-crash-20080817 kernel stack dump]
     204  * 2008-08-19T21:41 notified by kaiser to reboot, required fsck /dev/sda3 by gavin
     205  * 2008-09-11T1335 [wiki:Ipp018-crash-2008-09-11 kernel oops]
     206  * 2008-09-15T11:00 [wiki:Ipp018-crash-2008-09-15 kernel oops]; ssh was dead but was able to login and attempt to reboot; reboot hung forever... had to power cycle
     207  * 2008-12-19T14:45 crashed with kernel panic : stack trace [wiki:Ipp018-crash-20081219 ipp018-crash-20081219]
     208
     209 *ipp019
     210  * 2008-09-03T22:10:27 found ipp019@ippcon booted from livecd, notified by nagios. unable to boot from raidset
     211
     212 *ipp020
     213  * 2007-06-21T07:02:03 IDE Channel 19 Device Failed 
     214  * 2007-06-21T07:02:09 IDE Channel 19 Device Failed 
     215  * 2007-06-21T07:02:16 IDE Channel 20 Device Removed
     216  * 2008-07-15:12:00:00 (approx) system unresponsive *** power cycled
     217  * 2008-08-20T17:51:57 forcdeth network death
     218  * 2008-09-10T12:50:00 (approx) system unresponsive *** power cycled
     219  * 2008-09-16T21:58    kernel panic power cycled by bills around 2008-09-17T07:00
     220  * 2008-09-19T04:49    [172920.752681] Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills at 06:53 failure occured while iqanalysis was starting up processing
     221
     222 *ipp021
     223  * 2008-08-26T17:00 forcedeth hang power cycled by bills
     224  * 2008-09-10T18:48 forcedeth problem power cycled by gavin @ ~19:31
     225  * 2008-11-05T18:49 load went to 11, console hung. power cycled by eam
     226
     227 *ipp023
     228  *  2009-02-20T13:03:49 IDE Channel 6 Reading Error - Reseated 2009-02-20T13:34:50
     229
     230 *ipp025
     231  * 2008-10-21T11:36 power cycled by gavin [wiki:Ipp025-nfs-2008-10-20 BUG: soft lockup]
     232  * 2008-10-22T12:17 console hung message: [85568.599373] BUG: spinlock lockup on CPU#3, pswarp/14919, ffff88013624a100 power cycled by bills
     233  * 2008-10-22T14:16:36 console messages: [wiki:Ipp025-crash-2008-10-22 BUG: spinlock on CPU#2]
     234  * 2008-10-24T16:29:00 console hung: [wiki:Ipp025-crash-2008-10-24 ipp025-crash-2008-10-24]
     235  * 2008-10-24T16:29:00 console hung: [wiki:Ipp025-crash-20081026 ipp025-crash-20081026]
     236  * 2008-10-26T15:45    kernel panic [wiki:Ipp025-crash-20081026b ipp025-crash-20081026b] power cycled by bills
     237  * 2008-10-30T10:59:00 console hung, spinlock: [wiki:Ipp025-crash-20081030 ipp025-crash-20081030]
     238  * 2008-11-03T10:07 down (do_IRQ: 0.179 No irq handler for vector) during pap MD processing (ppStack 4 threads), power cycled by pap
     239  * Suspect [wiki:Ipp025-20081103 memory problems]
     240
     241 *ipp027
     242  * 2008-09-21T21:00 (approx) [wiki:Ipp027-crash-2008-09-21 kernel oops]
     243  * 2008-10-04T09:30 machine non-responsive. Console frozen with no messages. Power cycled by bills.
     244  * 2008-11-15T15:57  no response nothing on console power cycled by bills
     245  * 2008-11-15T22:20  no response nothing on console power cycled by bills
     246  * 2008-11-20T00 (approx) no response on console, power cycled by pap
     247  * 2008-11-26T11:06 no response on console, power cycled by pap; was under heavy load
     248  * 2008-11-27T08:30 no response on console, power cycled by bills; was under heavy load
     249  * 2009-01-23T14:15 no response on console; was under heavy load
     250  * 2009-02-19T08:35:05 IDE Channel 20 Device Failed
     251
     252 *ipp028
     253  * 2008-09-22T16:30 [wiki:Ipp028-oops-2008-09-22 ipp028-oops-2008-09-22]
     254  * 2008-09-26T10:40 [wiki:Ipp028-oops-2008-09-26 ipp028-oops-2008-09-26]
     255  * 2008-10-23T05:50:09 IDE Channel 21 Device Failed
     256
     257 *ipp030
     258  * 2009-02-20T10:03:49 IDE Channel 17 Reading Error - Reseated 2009-02-20T10:34:16
     259
     260 *ipp033
     261  * 2008-09-22T04:00 (approx) [wiki:Ipp033-crash-2008-09-22 kernel oops]
     262
     263 *ipp036
     264  *2008-10-29T06:34:58 IDE Channel 3 Device Failed
     265
     266 * ASA serial<->node name map
     267<pre>
     268Note all nums start with 190314xx (where "xx" is the two digits).
     269
     270ipp004: 07
     271ipp005: 10
     272ipp006: 12
     273ipp007: 11
     274ipp008: 01
     275ipp009: 13
     276ipp010: 09
     277ipp011: 08
     278ipp012: 02
     279ipp013: 04
     280ipp014: 03
     281ipp016: 05
     282ipp018: 14
     283ipp025: 06
     284ipp037: 00
     285</pre>