IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 46 and Version 47 of Production_Cluster


Ignore:
Timestamp:
Jul 2, 2009, 11:31:50 AM (17 years ago)
Author:
jhoblitt
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Production_Cluster

    v46 v47  
    1 *Disk reports
     1'''Storage Nodes'''
     2 * [wiki:ipp004_log ipp004]
     3 * [wiki:ipp005_log ipp005]
     4 * [wiki:ipp006_log ipp006]
     5 * [wiki:ipp007_log ipp007]
     6 * [wiki:ipp008_log ipp008]
     7 * [wiki:ipp009_log ipp009]
     8 * [wiki:ipp010_log ipp010]
     9 * [wiki:ipp011_log ipp011]
     10 * [wiki:ipp012_log ipp012]
     11 * [wiki:ipp013_log ipp013]
     12 * [wiki:ipp014_log ipp014]
     13 * [wiki:ipp015_log ipp015]
     14 * [wiki:ipp016_log ipp016]
     15 * [wiki:ipp017_log ipp017]
     16 * [wiki:ipp018_log ipp018]
     17 * [wiki:ipp019_log ipp019]
     18 * [wiki:ipp020_log ipp020]
     19 * [wiki:ipp021_log ipp021]
     20 * [wiki:ipp022_log ipp022]
     21 * [wiki:ipp023_log ipp023]
     22 * [wiki:ipp024_log ipp024]
     23 * [wiki:ipp025_log ipp025]
     24 * [wiki:ipp026_log ipp026]
     25 * [wiki:ipp027_log ipp027]
     26 * [wiki:ipp028_log ipp028]
     27 * [wiki:ipp029_log ipp029]
     28 * [wiki:ipp030_log ipp030]
     29 * [wiki:ipp031_log ipp031]
     30 * [wiki:ipp032_log ipp032]
     31 * [wiki:ipp033_log ipp033]
     32 * [wiki:ipp034_log ipp034]
     33 * [wiki:ipp035_log ipp035]
     34 * [wiki:ipp036_log ipp036]
     35 * [wiki:ipp037_log ipp037]
     36 * [wiki:ipp038_log ipp038]
     37 * [wiki:ipp039_log ipp039]
     38 * [wiki:ipp040_log ipp040]
     39 * [wiki:ipp041_log ipp041]
     40 * [wiki:ipp042_log ipp042]
     41 * [wiki:ipp043_log ipp043]
     42 * [wiki:ipp044_log ipp044]
     43 * [wiki:ipp045_log ipp045]
     44 * [wiki:ipp046_log ipp046]
     45 * [wiki:ipp047_log ipp047]
     46 * [wiki:ipp048_log ipp048]
     47 * [wiki:ipp049_log ipp049]
     48 * [wiki:ipp050_log ipp050]
     49 * [wiki:ipp051_log ipp051]
     50 * [wiki:ipp052_log ipp052]
     51 * [wiki:ipp053_log ipp053]
     52
     53'''DB Nodes'''
     54 * [wiki:ippdb00_log ippdb00]
     55 * [wiki:ippdb01_log ippdb01]
     56 * [wiki:ippdb02_log ippdb02]
     57
     58'''Compute Nodes'''
     59 * [wiki:ippc00_log ippc00]
     60 * [wiki:ippc01_log ippc01]
     61 * [wiki:ippc02_log ippc02]
     62 * [wiki:ippc03_log ippc03]
     63 * [wiki:ippc04_log ippc04]
     64 * [wiki:ippc05_log ippc05]
     65 * [wiki:ippc06_log ippc06]
     66 * [wiki:ippc07_log ippc07]
     67 * [wiki:ippc08_log ippc08]
     68 * [wiki:ippc09_log ippc09]
     69 * [wiki:ippc10_log ippc10]
     70 * [wiki:ippc11_log ippc11]
     71 * [wiki:ippc12_log ippc12]
     72 * [wiki:ippc13_log ippc13]
     73 * [wiki:ippc14_log ippc14]
     74 * [wiki:ippc15_log ippc15]
     75 * [wiki:ippc16_log ippc16]
     76 * [wiki:ippc17_log ippc17]
     77 * [wiki:ippc18_log ippc18]
     78 * [wiki:ippc19_log ippc19]
     79
     80 * [wiki:asaSerialMap ASA serial<->node name map]
     81
     82Disk reports
    283  *[wiki:2008-08-22 2008-08-22]
    3 
    4   * [wiki:ipp004_log ipp004]
    5   * [wiki:ipp005_log ipp005]
    6   * [wiki:ipp006_log ipp006]
    7   * [wiki:ipp007_log ipp007]
    8 
    9 
    10  *ipp008
    11   * 2007-03-23T05:39:42 IDE Channel 1 Device Removed 
    12   * 2008-08-09T06:38:00 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
    13   * 2008-08-10T02:29:00 system became unresponsive: error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
    14   * 2008-08-11T10:53:34 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
    15   * 2008-08-11T14:46:55 system became unresponsive: notified by kaiser power cycled by gavin
    16   * 2008-08-11T15:28:28 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
    17   * 2008-09-03T07:40     power cycled by bills due to network hang
    18   * 2008-09-05T05:04    system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (many times, also 0.189, 2.189) (power cycled by eam)
    19   * 2008-10-19T12:30 system became unresponsive power cycled
    20   * 2008-10-20T22:00 system became unresponsive power cycled (nothing on console)
    21   * 2008-10-21T12:53:00 system became unresponsive power cycled by gavin (nothing on console)
    22   * 2008-10-21T14:05:00 system became unresponsive power cycled by gavin (nothing on console)
    23   * 2008-10-22T10:35:00 system became unresponsive power cycled by bills (nothing on console)
    24   * 2008-10-22T10:50:00 failed pretty quickly. (warps were running) power cycled by bills
    25   * 2008-10-30T12:22:00 system became unresponsive power cycled by gavin (nothing on console)
    26   * 2008-11-03T11:20 system became unresponsive power cycled by bills (nothing on console) Paul was processing on it.
    27   * 2008-12-13T17:13 system became unresponsive (time is nagios email) power cycled by bills 12-14T09:55 (nothing on console)
    28   * 2008-12-18T08:00 system became unresponsive power cycled by gavin (nothing on console)
    29   * 2008-12-19T14:39:00 crashed, no message on console, power cycled by eam & rodney
    30   * 2008-12-19T18:39 system unresponsive, no message on console, power cycled by gavin
    31   * 2008-12-20T17:33 crashed, no message on console, power cycled by rodney
    32   * 2008-12-21T16:40 crashed, no message on console, power cycled by rodney
    33   * 2009-03-01T14:00 system became unresponsive power cycled (nothing on console)
    34   * 2009-03-03T11:00 system became unresponsive power cycled (nothing on console)
    35   * 2009-03-05T13:40 system became unresponsive power cycled (nothing on console)
    36   * 2009-06-15T17:13:13 IDE Channel 23 Device Removed
    37 
    38  *ipp009
    39   * 2008-02-07T17:43:55 IDE Channel 5 Device Failed 
    40   * 2008-07-12 time unknown forcedeth problem power cycled by bills
    41   * 2008-07-03T00:00 (approx) forcedeth problem power cycled around 0600 by bills
    42   * 2008-10-30T11:07:00 console hung, spinlock [wiki:Ipp009-crash-20081030 ipp009-crash-20081030]
    43   * 2008-11-15T15:22  no response nothing on console power cycled by bills
    44   * 2009-06-11T08:39 system became unresponsive connected to console do_IRQ error; power cycled by bills
    45 
    46  *ipp010
    47   * 2008-02-08T10:34:33 IDE Channel 12 Device Removed 
    48   * 2008-02-08T10:44:12 IDE Channel 12 Device Removed 
    49   * 2008-02-08T10:48:01 IDE Channel 12 Device Removed 
    50   * 2008-07-21T11:45   system became unresponsive over network power cycled by bills
    51   * 2008-07-31T07:35   system became unresponsive over network power cycled by bills
    52   * 2008-09-10T16:24 unresponsive, power cycled by price
    53   * 2008-10-16T10:05 [wiki:Ipp010-crash-2008-10-16 kernel panic]
    54   * 2008-12-11T00:02 unresponsive no messages power cycled by bills around 08:55
    55   * 2009-01-04T00:12:11 IDE Channel 1 Device Failed
    56 
    57  *ipp011
    58   * 2007-03-23T05:51:14 IDE Channel 24 Device Failed 
    59   * 2007-03-23T05:51:14 IDE Channel 22 Device Removed 
    60   * 2007-03-23T05:51:14 IDE Channel 23 Device Removed 
    61   * 2007-03-23T05:54:44 IDE Channel 1 Device Removed 
    62   * 2008-07-10T10:15:00 - forcedeth NIC failure - RPC errors on console
    63   * 2008-07-15T07:00:00 approx forecdeth problem power cycled by bills
    64   * 2008-08-11T17:23:32 forecdeth problem power cycled by gavin
    65 
    66  *ipp012
    67   * 2007-03-22T14:36:28 IDE Channel 15 Device Removed 
    68 
    69  *ipp013
    70   * 2006-01-01T00:56:09 IDE Channel 22 Device Removed 
    71   * 2008-06-23T02:33:49 IDE Channel 11 Device Failed(SMART) 
    72   * 2008-07-09T10:41:59 IDE Channel 11 Device Removed 
    73 
    74  *ipp014
    75   * 2008-07-10T00:42:03 IDE Channel 10 Device Failed(SMART) 
    76   * 2008-07-10T02:23:17 IDE Channel 10 Device Removed 
    77   * 2008-10-06T05:31    Machine became unresponsive. No message on console. Power cycled by bills 06:07
    78   * 2008-10-16T05:51 kernel panic [wiki:Ipp014_panic_1016 ipp014_panic_1016]
    79   * 2008-10-15T22:30 machine became unresponsive nothing on console power cyled by bills 8:52 10/16
    80   * 2008-10-20T20:49 kernel panic unable to handle kernel paging request [wiki:Ipp014_panic_1020 ipp014_panic_1020]
    81   * 2008-10-22T05:45 machine became unresponsive nothing on console power cycled by bills around 07:20
    82   * 2008-10-25T11:55:39 machine hung (spinlock) [wiki:Ipp014-crash-20081025 ipp014-crash-20081025]
    83   * 2008-10-25T11:55:39 machine hung (spinlock) [wiki:Ipp014-crash-20081026 ipp014-crash-20081026]
    84   * 2008-10-31T00:00:00 machine hung, no stack trace (a few segfault warnings only)
    85   * 2008-11-05T14:10:00 machine hung (spinlock)
    86   * 2008-11-26T11:50 (approx) console unresponsive; [wiki:Ipp014-crash-2008-11-26 general protection fault] on console; power cycled by pap
    87   * 2008-11-26T11:00 (approx) console unresponsive; [wiki:Ipp014-crash-20081127 ipp014-crash-20081127]; power cycled by eam
    88   * 2008-12-05 crashed overnight, [wiki:Ipp014-crash-2008-12-04 multiple traces produced]; power cycled by pap
    89   * 2008-12-11T01:54 unresponsive no messages power cycled by bills around 08:55
    90   * 2009-03-02T17:00 (approx) console unresponsive; [wiki:ipp014-crash-20090302 ipp014-crash-20090302]; power cycled by eam
    91   * 2009-03-05T18:23 kernel panic [wiki:ipp014_crash-20090305 ipp014_crash-20090305]; power cycled by bills
    92   * 2009-03-11T06:00 kernel panic [wiki:ipp014_crash-20090311 ipp014_crash-20090311]; power cycled by eam
    93   * 2009-03-11T16:00 kernel panic [wiki:ipp014_crash-20090311b ipp014_crash-20090311b]; power cycled by eam
    94   * 2009-04-07T10:29 kernel panic [wiki:ipp014_crash20090407 ipp014_crash-20090407]; power cycled by bills
    95 
    96  *ipp015
    97   * 2008-07-03T23:37:05 IDE Channel 21 Device Removed 
    98   * 2008-07-04T01:27:23 IDE Channel 3 Device Failed 
    99   * 2008-07-04T01:40:07 IDE Channel 3 Device Removed 
    100   * 2008-07-04T01:43:16 IDE Channel 4 Device Failed 
    101   * 2008-07-09T16:31:25 IDE Channel 7 Device Removed 
    102   * 2008-03-22T23:47:35 IDE Channel 7 Device Removed
    103   * 2008-03-22T23:57:31 IDE Channel 7 Device Inserted
    104   * 2008-3-22T23:47:35 IDE Channel 7 Device Removed
    105   * 2008-3-22T23:57:31 IDE Channel 7 Device Inserted
    106   * 2008-3-23T12:27:16 ARC-1280-VOL#00 Complete Rebuild 012:29:44
    107   * 2008-07-09 ipp015 disk set moved into a wave #2 chassis (ipp025)
    108   * 2009-04-24T08:00  ipp015 cannot access /data/ipp032.0 [[BR]]
    109         no umount command in ipp032:/var/log/messages until attempt by bills to umount by hand around 7:58 am [[BR]]
    110         last mount from ipp015 Apr 22. [[BR]]
    111         tried to reboot around 11am hung during shutdown trying to unmount /data/ipp032.0[[BR]]
    112         power cycled by bills
    113 
    114  *ipp016
    115   * 2008-02-26T10:35:50 IDE Channel 13 Device Removed 
    116   * 2008-09-10T17:48 Looks like [wiki:Ipp016-crash-20080910 forcedeth], rebooted by price
    117   * 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
    118   * <Sep/27 03:10 pm>i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills
    119   * 2008-10-08T15:53 [wiki:Ipp016-crash-20081008 soft lockup]
    120   * 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
    121   * 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
    122   * 2008-11-03T18:04 system unresponsive while processing stacks; [wiki:Ipp016-crash-20081103 panic] on console (looks like spinlock problem); power cycled by pap
    123   * 2008-11-12T17:00 ; power cycled
    124 <Nov/12 02:06 pm>[768330.447278] do_IRQ: 2.187 No irq handler for vector
    125 <Nov/12 02:07 pm>[768402.246842] do_IRQ: 0.179 No irq handler for vector
    126   * 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia).  Error message: <code>[1629646.647189] do_IRQ: 2.179 No irq handler for vector</code>.  Power cycled by pap.
    127   * 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; [wiki:Ipp016-crash-20081218 panic]; power cycled by gavin
    128   * 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin
    129   * 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin
    130   * 2008-12-20T17:30 crashed, no message on console, power cycled by rodney
    131   * 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin
    132   * 2009-01-12T13:32:31] system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
    133   * 2009-01-13T12:24  system became unresponsive No error messages power cycled by bills
    134   * 2009-01-13   system went down two more times. The second time a kernel panic occured due to a null pointer dereference [wiki:Ipp016-crash090113 ipp016-crash090113].
    135   * 2009-01-20 [wiki:Ipp016-crash-2009-01-20 kernel oops] kernel panic, power cycled.
    136   * 2009-01-23 [wiki:Ipp016-crash-2009-01-23 kernel BUG] dead lock, power cycled.
    137   * 2009-01-23T14:17 [wiki:Ipp016-crash-2009-01-23T14:17 kernel panic] kernel panic, power cycled.
    138   * 2009-01-28T07:07 [wiki:Ipp016-crash-2009-01-28T07:07 kernel panic] kernel panic, power cycled.
    139   * 2009-01-28T11:21 [wiki:Ipp016-crash-2009-01-28T11:21 do_IRQ] do_IRQ on console, YP/RPC errors being printed on console, power cycled.
    140   * 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin
    141   * 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector  BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00
    142   * 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene
    143   * 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled.
    144   * 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled
    145   * 2009-03-05T13:50 do_IRQ: 2.179 No irq handler for vector; power cycled by EAM
    146   * 2009-03-12T11:10 no console messages; power cycled by EAM
    147   * 2009-04-22T11:37 system unable to mount /data/ipp004.0 bill hung console trying so power cycled.
    148   * 2009-05-15T12:08 do_IRQ error; power cycled by PAP
    149   * 2009-06-08T23 do_IRQ: 1.189 No irq handler for vector; power cycled by bills
    150 
    151  *ipp017
    152 
    153  *[wiki:ipp018_log ipp018]
    154 
    155  *ipp019
    156   * 2008-09-03T22:10:27 found ipp019@ippcon booted from livecd, notified by nagios. unable to boot from raidset
    157 
    158  *ipp020
    159   * 2007-06-21T07:02:03 IDE Channel 19 Device Failed 
    160   * 2007-06-21T07:02:09 IDE Channel 19 Device Failed 
    161   * 2007-06-21T07:02:16 IDE Channel 20 Device Removed
    162   * 2008-07-15:12:00:00 (approx) system unresponsive *** power cycled
    163   * 2008-08-20T17:51:57 forcdeth network death
    164   * 2008-09-10T12:50:00 (approx) system unresponsive *** power cycled
    165   * 2008-09-16T21:58    kernel panic power cycled by bills around 2008-09-17T07:00
    166   * 2008-09-19T04:49    [172920.752681] Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills at 06:53 failure occured while iqanalysis was starting up processing
    167 
    168  *ipp021
    169   * 2008-08-26T17:00 forcedeth hang power cycled by bills
    170   * 2008-09-10T18:48 forcedeth problem power cycled by gavin @ ~19:31
    171   * 2008-11-05T18:49 load went to 11, console hung. power cycled by eam
    172 
    173  *ipp023
    174   * 2009-02-20T13:03:49 IDE Channel 6 Reading Error - Reseated 2009-02-20T13:34:50
    175   * 2009-06-01T11:40:13 IDE Channel 10 Device Failed
    176 
    177  *ipp025
    178   * 2008-10-21T11:36 power cycled by gavin [wiki:Ipp025-nfs-2008-10-20 BUG: soft lockup]
    179   * 2008-10-22T12:17 console hung message: [85568.599373] BUG: spinlock lockup on CPU#3, pswarp/14919, ffff88013624a100 power cycled by bills
    180   * 2008-10-22T14:16:36 console messages: [wiki:Ipp025-crash-2008-10-22 BUG: spinlock on CPU#2]
    181   * 2008-10-24T16:29:00 console hung: [wiki:Ipp025-crash-2008-10-24 ipp025-crash-2008-10-24]
    182   * 2008-10-24T16:29:00 console hung: [wiki:Ipp025-crash-20081026 ipp025-crash-20081026]
    183   * 2008-10-26T15:45    kernel panic [wiki:Ipp025-crash-20081026b ipp025-crash-20081026b] power cycled by bills
    184   * 2008-10-30T10:59:00 console hung, spinlock: [wiki:Ipp025-crash-20081030 ipp025-crash-20081030]
    185   * 2008-11-03T10:07 down (do_IRQ: 0.179 No irq handler for vector) during pap MD processing (ppStack 4 threads), power cycled by pap
    186   * Suspect [wiki:Ipp025-20081103 memory problems]
    187   * 2009-05-10T12:00 down [wiki:ipp025-crash-20090510 ipp025-crash-20090510]
    188   * 2009-06-07T15:49 went down while processing magic data. No response on console. power cycled by bills
    189 
    190  *ipp027
    191   * 2008-09-21T21:00 (approx) [wiki:Ipp027-crash-2008-09-21 kernel oops]
    192   * 2008-10-04T09:30 machine non-responsive. Console frozen with no messages. Power cycled by bills.
    193   * 2008-11-15T15:57  no response nothing on console power cycled by bills
    194   * 2008-11-15T22:20  no response nothing on console power cycled by bills
    195   * 2008-11-20T00 (approx) no response on console, power cycled by pap
    196   * 2008-11-26T11:06 no response on console, power cycled by pap; was under heavy load
    197   * 2008-11-27T08:30 no response on console, power cycled by bills; was under heavy load
    198   * 2009-01-23T14:15 no response on console; was under heavy load
    199   * 2009-02-19T08:35:05 IDE Channel 20 Device Failed
    200   * 2009-03-23T10:22:50 IDE Channel 20 Device Failed
    201   * 2009-06-01T06:23:53 Degraded unit: unit=0, port=1 - disk#2 failed
    202 
    203  *ipp028
    204   * 2008-09-22T16:30 [wiki:Ipp028-oops-2008-09-22 ipp028-oops-2008-09-22]
    205   * 2008-09-26T10:40 [wiki:Ipp028-oops-2008-09-26 ipp028-oops-2008-09-26]
    206   * 2008-10-23T05:50:09 IDE Channel 21 Device Failed
    207 
    208  *ipp030
    209   * 2009-02-20T10:03:49 IDE Channel 17 Reading Error - Reseated 2009-02-20T10:34:16
    210 
    211  *ipp033
    212   * 2008-09-22T04:00 (approx) [wiki:Ipp033-crash-2008-09-22 kernel oops]
    213 
    214  *ipp036
    215   * 2008-10-29T06:34:58 IDE Channel 3 Device Failed
    216 
    217  *ipp039
    218   * 2009-05-04T14:52:43 IDE Channel 17 Device Failed
    219 
    220  *ipp045
    221   * 2009-06-26T15:15:00 [1559830.598447] Kernel panic - not syncing: Attempted to kill init!
    222 
    223  * ASA serial<->node name map
    224 <pre>
    225 Note all nums start with 190314xx (where "xx" is the two digits).
    226 
    227 ipp004: 07
    228 ipp005: 10
    229 ipp006: 12
    230 ipp007: 11
    231 ipp008: 01
    232 ipp009: 13
    233 ipp010: 09
    234 ipp011: 08
    235 ipp012: 02
    236 ipp013: 04
    237 ipp014: 03
    238 ipp016: 05
    239 ipp018: 14
    240 ipp025: 06
    241 ipp037: 00
    242 </pre>