IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 34 and Version 35 of Production_Work_Log


Ignore:
Timestamp:
Feb 9, 2012, 10:43:45 PM (14 years ago)
Author:
huntley
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Production_Work_Log

    v34 v35  
    22= 2012 Production Node Work =
    33
    4 = FEB 2 =
    5   - Checked 4 servers for possible fan failures - Haydn and Rita
    6     - ippc08 - larger fan on power supply not working -- frozen stuck. We did not have the replacement fan with us and will replace next week.
    7     - ipp015 - we found two fans in the front (that pull the cold air in) that weren't working. We discovered two power connectors to power the fans were not connected.  After connecting power all fans now working.
    8     - ipp025 - no fan failures found.
    9     - ipp017 - no fan failures found.
    10 
    11   - Rita - While I was there working Gavin called for support to check out ipp027, which seemed to have crashed. After power cycling a few times, waiting 1 hr., and power cycling again, the system rebooted and came back online.
     4== FEB 9 ==
     5  * ipp027 -- began displaying the BIOS message today -- we have no idea why, but we were very thankful!  Used the CD to update its BIOS.  This cleared the old settings, so I changed the ones which we always change.  Now it works fine again.
     6  * ipp028 -- still catatonic.  Displayed "E4" POST code.  Cindy recommended removing everything from the machine, so we removed the RAID card.  This caused the two beeps it was making to change to a different (undocumented in the manual) pattern.  The POST code changed to "E1".  Next we removed all of the RAM.  The POST code changed to "60" after going through a bunch of different values.  We started reinstalling the RAM, first 1 stick, then 4, then 8.  With one stick the BIOS screen said that the BIOS settings were invalid, so I reset them to our usual values and saved it.  Apparently this machine's BIOS settings spontaneously got corrupted.  If this happens again, we should replace the CR2032 battery with a fresh one, and/or check that the CMOS reset jumper is in the default position. Reinstalled the RAID card.  Oddly, this machine recognizes that it has an ATA DVD/RW drive, however it cannot read the CD in the drive.
     7
     8== FEB 8 ==
     9  * ipp030 - one small fan in back wasn't working, but didn't bother to replace it.  It has miniscule impact.
     10  * ipp028 - no fan failures found.
     11  * ipp031 - no fan failures found.
     12  * ipp023 - no fan failures found.
     13  * ipp014 - no fan failures found, however one of the CPU fans is installed backwards -- it is pushing air toward all of the other fans.  Fixing it requires removing the heat sink and rotating it 180 degrees, which we didn't want to do at that time, but we should consider doing in the future.
     14  * ipp016 - missing one fan (on the left), had 2 broken fans.  One fan was actually broken, and the other wouldn't work because of a short/open circuit on the litte board which the fans plug into.  Replaced the broken fan, and connected the other fan directly to the headers behind the disk drives.
     15  * ippc08 - the large power supply fan was actually ok.
     16  * ipp025 - moved the power cables on the PDU to where they should have gone.
     17  * ipp027 - the power cables weren't looped through the power supply handles, so one by one I moved them.  Tried to update the BIOS with the CD which Gavin sent me the ISO for.  Unfortunately, the machine turned catatonic, and we couldn't see anything on the video and the caps lock light wouldn't come on on the keyboard.
     18  * ipp028 - wouldn't come back up.  Gavin recommended using the motherboard jumper to clear the CMOS.  Behaved just like ipp027 -- catatonic.  Displayed "E4" POST code.
     19
     20== FEB 2 ==
     21  * Checked 4 servers for possible fan failures - Haydn and Rita
     22    * ippc08 - larger fan on power supply not working -- frozen stuck. We did not have the replacement fan with us and will replace next week.
     23    * ipp015 - we found two fans in the front (that pull the cold air in) that weren't working. We discovered two power connectors to power the fans were not connected.  After connecting power all fans now working.
     24    * ipp025 - no fan failures found.
     25    * ipp017 - no fan failures found.
     26  * Rita - While I was there working Gavin called for support to check out ipp027, which seemed to have crashed. After power cycling a few times, waiting 1 hr., and power cycling again, the system rebooted and came back online.
     27
    1228== JAN 30  ==
    1329 * ipp027 crashing and cpu overheated Opened ipp027 to find CPU1 fan had come unplugged. - reconnected it - running ok since then
    1430 * remounted ippc12 with new MOBO installed by Haydn
    1531 * Unboxed and racked WAVE 5 34 1U's in cab 10 and 11. 
     32
    1633== JAN 31  ==
    1734 * Configured the bios on all 34 WAVE 5 machines and began the labeling of the nodes.
     35
    1836= 2011 Production Node Work =
     37
    1938== DEC 19 ==
    2039 * Replaced the bad RAM in ipp063.
     
    2342 * Found two small fans.  I'll try them in the power supply for ippc12.
    2443 * ipp064 sometimes shows 40 GB RAM when it boots, and other times 48 GB.  It contains some inconsistent RAM.  I'll try to find it and swap it out next time I'm there.
     44
    2545== DEC 12 ==
    2646 * Temporarily swapped two known good motherboards into the chassis for ippc11 and ippc12.  They both worked fine, so this indicates that the motherboards in ippc11 and ippc12 are both broken.  Perhaps the CPU's and RAM are okay, but we'll need a known good LGA771 motherboard to test them with.
     47
    2748== DEC 8 ==
    2849 * Swapped the RAM between ipp065, ipp066, and ipp063.  ipp065 and ipp066 either sometimes or always indicated only 40GB, instead of 48GB.  When we moved the RAM between the different machines, the good RAM from ipp063 always worked, the bad RAM always failed, and the inconsistent RAM continued to behave inconsistently, so the problem is that we have at least two bad sticks of RAM -- one which always fails and one which fails intermittently.
     50
    2951== DEC 7 ==
    3052 * Tested the power supplies on ippc11 and ippc12.  ippc11's power supply's -12v line wasn't working.  ippc12's power supply produces the right voltages, but the exit fan doesn't spin, which decreases the airflow to about 25% of what it should be.  Found a spare power supply for ippc11. Replaced the motherboard in ippc11, but the "new" motherboard doesn't work.
     53
    3154== NOV 21 ==
    3255 * Cindy Bill and Rita swapped out motherboard and CPU's from ippc12 into ipp029
     56
    3357== NOV 15 ==
    3458 * added auto-monthly consistency checks for ippb machines
    3559 * switched the names for ippb00/03 machines - auto-mount configuration ippb00.0 -> ippb00.2 to reflect physical array swaps done.
     60
    3661== Nov 10 ==
    3762 * Haydn visited MHPCC.