https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Production_Cluster_Status

2012 Production Node Work

FEB 2

• Checked 4 servers for possible fan failures - Haydn and Rita
  • ippc08 - fan on power supply not working. We did not have the replacement fan with us and will replace next week.
  • ipp015 - we found 3 fans in the front (that pull the cold air in) that weren't working. One fan was bad, but we discovered two power connectors to power the fans not connected. After connecting power and replacing the bad fan, all fans now working.
  • ipp025 - no fan failures found.
  • ipp017 - no fan failures found.

• Rita - While I was there working Gavin called for support to check out ipp027, which seemed to have crashed. After power cycling a few times, waiting 1 hr., and power cycling again, the system rebooted and came back online.

JAN 30

• ipp027 crashing and cpu overheated Opened ipp027 to find CPU1 fan had come unplugged. - reconnected it - running ok since then
• remounted ippc12 with new MOBO installed by Haydn
• Unboxed and racked WAVE 5 34 1U's in cab 10 and 11.

JAN 31

• Configured the bios on all 34 WAVE 5 machines and began the labeling of the nodes.

2011 Production Node Work

DEC 19

• Replaced the bad RAM in ipp063.
• Rebuilt the RAID in ipp064, however I didn't have the documentation, couldn't get help from Cindy or Gavin, and one parameter wasn't correct, so it will have to be redone. Maybe Wednesday...
• Found an old Tyan LGA771 motherboard and brought it back. I'll try populating it to see if it works in ippc11.
• Found two small fans. I'll try them in the power supply for ippc12.
• ipp064 sometimes shows 40 GB RAM when it boots, and other times 48 GB. It contains some inconsistent RAM. I'll try to find it and swap it out next time I'm there.

DEC 12

• Temporarily swapped two known good motherboards into the chassis for ippc11 and ippc12. They both worked fine, so this indicates that the motherboards in ippc11 and ippc12 are both broken. Perhaps the CPU's and RAM are okay, but we'll need a known good LGA771 motherboard to test them with.

DEC 8

• Swapped the RAM between ipp065, ipp066, and ipp063. ipp065 and ipp066 either sometimes or always indicated only 40GB, instead of 48GB. When we moved the RAM between the different machines, the good RAM from ipp063 always worked, the bad RAM always failed, and the inconsistent RAM continued to behave inconsistently, so the problem is that we have at least two bad sticks of RAM -- one which always fails and one which fails intermittently.

DEC 7

• Tested the power supplies on ippc11 and ippc12. ippc11's power supply's -12v line wasn't working. ippc12's power supply produces the right voltages, but the exit fan doesn't spin, which decreases the airflow to about 25% of what it should be. Found a spare power supply for ippc11. Replaced the motherboard in ippc11, but the "new" motherboard doesn't work.

NOV 21

• Cindy Bill and Rita swapped out motherboard and CPU's from ippc12 into ipp029

NOV 15

• added auto-monthly consistency checks for ippb machines
• switched the names for ippb00/03 machines - auto-mount configuration ippb00.0 -> ippb00.2 to reflect physical array swaps done.

Nov 10

• Haydn visited MHPCC.
• Installed a new BBU in ippdb02.
• Checked ipp029, because it was reported to crash quickly under load. The CPU heat sinks had not been installed correctly. Heat sink compound covered only ~50% of the CPU's. Removed the old heat sink compound and used Artic Silver 5. We should probably do this for all of the machines in this series -- just because they aren't crashing, doesn't mean they were installed correctly. They are probably simply throttling themselves to avoid thermal failure.
• Checked ippc11, because it was reported to crash under load every few days. Burned myself slightly when my bare forearm touched the bottom of the case. This is a 1U machine, and the power supply is incredibly hot (~55 deg C). Removed the CPU heat sinks, but they had originally been installed correctly. Re-racked the machine, but other machines with the same design have power supplies which are much cooler. Perhaps the power supply is failing, or one of the fans in it has failed. Because this is a 1U machine, the CPU heat sinks depend on the cover being on in order to get good air flow across them, but next time I'm there, there is a spare power supply which might work in this machine, and I can try running it with the top off for a few seconds to check if the power supply fans are working or not.
• Reseated the memory in ipp065, but it still reports only 40GB instead of 48GB.

Nov 7

• team racking and installation of 13 5 U's
• put cpu back into ipp036
• put ippc10 back in rack

Nov 4

• ippb00 Haydn replaced back plane for ippb00 a0 - Drive 3 detected. All drives put back in unit - parts sent back to YMI.

Nov 1

• Installed ASUS MOBO into ippc10

Oct 27

• Installed spare ASUS MOBO into ippc10 - cpu heat sinks do not install properly - Brought ippc10 back to ATRC -ordered proper CPU carriage frame from YMI

Oct 25

• ippb00 replaced cable,raid card and sled - still not detecting drive A0 3 - ordered back plane

Oct 24

• ipp036 has dead CPU0 fan unable to boot.
• Swapped MOBO out of ippc10 into ipp036 (still in rack - unable to remove it - only me working ) Left ipp036 with 1 processor - not sure if cpu damaged - did want to hurt MOBO of running system
• Installed 6 of the 8 3TB drives in STARE nodes. 2 drive DOA sent for RMA

Oct 19

• replaced ippb00 A0 drive 3 but still not being detected. worked with LSI to de-bug - needs hardware fix Ordered parts from YMI

Oct 10

• Swapped CPU0 memory with CPU1 memory

Sept 30

• installed 3TB disks in slots 2&3

Sept 20

• Bill and Haydn Installed the new MOBO on ipp020

Sept 13

• Bill and Haydn Installed the new MOBO on ipp021

Aug 31

• ipp010, ipp017, ipp020 (disk upgrades) and ipp021 (mobo

upgrade).

Aug 30

• Replaced IPP014's raid controller.

Aug 23 2011

• Haydn Bill and I were able upgrade ipp004,ipp013 and ipp018 MB and disks without incident.
• We corrected the existing error in the RAID card cabling in ipp004 so now the drives are seated in the same order as the other machines ( funnily we discovered and remembered this problem because we had missed swapping a drive and put a 500GB back in which was showing in the wrong slot )

• ipp037 has a new MB.

• Gavin installed the OS and building the RAID - he was instrumental once again in supporting us while at MHPCC.

• Haydn did all the drive replacements

Aug 16

• Installed the replacement MOBO for ippc10

Aug 15

• ipp011
  • Upgraded to new MB and 2 TB drives. Had to move CPU0 memory stick from outer blue slot to middle black slot.
  • Suppose to fill blue slots first. The memory was not detected in the outer black slot. Same weirdness seen in ipp009 ( according to Gavin ASA chassis)

• ipp019
  • Upgraded to new MB and 2 TB drives. Memory all in blue slots and detected
  • The 1st new MB installed did not detect any memory in the cpu0 no matter where we put the sticks. Tried a second new MB using all the same memory and CPU's - with good results. Will RMA the 1st new MB.

• ipp015
  • Upgraded to 2TB drives. On reboot saw only 12GB of memory. For this machine memory is suppose to be an all black slots. However we noted that the memory was in a mixture of black and blue slots. Moved to all blacks. All memory detected.

Aug 1

• Swapped all memory out of ippdb03 - formally ippc00 - using the memory from ippc10 - waiting for MB
• Added 2 2TB disks to ippc10 to become the newest rendition of ippdb03

July 29

• Added new memory to ippb00 1nd 01 - now have 32GB each.

July 27

• Spent AM diagnosing the non-boot problem in ipp030 - failed MB
• After lunch
• Took failed MB out of ipp030 and replaced it with MB from ippc10.
• ippc10 remains out of the rack.

July 15

• Swapped the 1st 3 mem sticks out of ippdb00

May 16

• replace a raid card in ipp005

Feb 9

• Bill and I replaced the motherboards of ipp005, ipp006, ipp007, ipp025

Notables: ipp006 & ipp025 memory counts were missing 4GB. We identified which slot and put the stick in a black slot next to a blue working slot. All 24GB are now showing on both machines. The malfunctioning slots on the 2 machines were not the same. We tested the power cables and while we found some odd readings they were not consistent between the machines or relative to the cpu's with the malfunctioning slots.

The boards installed on the back of the CD drive are not working. We still have external CD drives hanging off the machines. We need to come up with a solution.

At some point in time - this fix should work for ipp009 so we have all 24 GB available.

ipp006: Voltage readings for CPU0 8pin power connector: +5v +12v +3.3V 5.2 12.5 3.4

-12V +12V 5vSB P6 11.9 12.2* 5.4* 410ms

• blinking

Voltage readings for CPU1 8pin power connector: same as cpu0 except for pg was 590ms and blinking too.

CPU1 slot closest to the cpu was not working.

ipp025: Voltage readings for CPU0 8pin power connector: Same display for ipp006 24 pin and 8 pin cpu0

Voltage readings for CPU0 8pin power connector: Same for CPU1 8pin. pg 430ms

Feb 4

We installed the new RAID card in ipp012 without problems.

However, we did not have success with the IDE->SATA converter board for the CD/DVD. The drive was not being detected on boot. We tried all SATA ports, different card, different CD/DVD, different cable to no avail. We checked the bios and there was note anything we could see that would prevent it from working. The external drive has been reattached. We brought a card and cd/dvd player back to test it and get it working here. It may need a jumper set to make sure it is not in slave mode - however we did not have a manual or a jumper - the boards did not come with one.

We also had the usual rack problems - ipp012 is particularly difficult to put back in - it is at the bottom of the rack and ipp013 is weighting down on it a bit.

Jan 31

disk upgrades on ipp008, ipp016, and ipp021, and motherboard upgrades on ipp008 and ipp016.

Jan 24

We put the power supplies from ipp009 in and it booted normally. However there are still fan related alarms sounding though all the fans are working. We swapped out the led/alarm board out with the same results.

ipp009 is up and ready for use but still short 4GB of memory. After many tests we can conclude that the problems lies in something other than the motherboard or memory. We had the same problem with a known good motherboard and memory. We narrowed it down to a set of black and blue memory slots associated with CPU0. Not the slots themselves, something that serves the slots. We know there is a problem with 2 slots because 12 x4GB memory sticks only showed 40 GB of memory. Also: The power supplies from ipp014 were put into ipp09 and there didn't seem to be any issues. Therefore we have no idea why ipp014 started booting normally with a power supply swap. Note: the ipp009 memory problem occurred with the good power supplies as well.

Jan 21

replaced motherboard hardware and disks for ipp009, ipp012, ipp014.

ipp012 new disk 11 had to be swapped out. ipp009: 6 4BG sticks were installed but only 5 are showing. All the memory has been swapped out for new memory with the same result. ipp014: We hear continuous beeps on power on changing too 2 short beeps and a long one after something appears on the screen. However the 2 beeps and a long on are not constant - they become random. We don't see any memory report on the first screen like the others. We changed the mother board and memory with identical results. There was a dead fan issue but we rewired it and all the fans are working and the fan light is off. No effect on the beep pattern. The initial quick short beeps sound like memory beeps but it weird that we get it to boot at all. Left out of the rack.

The new mother boards do not have an IDE connection for the existing optical drives. We (Bill) drove to the only 2 computer stores on the island and purchased the only 2 notebook size optical drives with sata connectors available. However - once we installed these we realized the connectors were something we had not seen - and did not have the cables for. At first glance they look like sata connections but there is a power component as well. opps. Fortunately Brad had external usb optical drives we could use. Hundreds of them, in fact. Gavin had to Mac Gyvered them in place because the cables were too short to set them anywhere.

2010 Production Node Work

(Up to Production Cluster Status)

Sept 15

• Removed cable obstructing power supply fan in ippc11

Sept 13

• Replaced CPUs (with new ones) and failed CPU fan in ipp014

Aug 23

• Swapped Memory out of IPP14

Aug 6

• Swapped Memory out of IPP12
• Upgraded IPP006 and IPP007 with 2TB drives

Jun 22

• Swapped Memory out of IPP14

April 27

• ipp018
  • swapped memory -> continuous beeps
  • Brad noticed one row of the hard drive array did not light up with red lights
  • Reseated SATA cables to backplane and RAID card
  • Swapped in set of new memory

April 26

• Swapped in larger RAID card memory module in ipp037
• ipp018
  • reseated memory -> still 2G memory
  • swapped CPUs -> still 2G memory showing
  • swapped in new set of memory (all 8 sticks) -> no boot rapid beeps

April 16

• Swapped two new CPUs into ipp037 and booted it into .31 kernel

April 8

• Swapped positions of ipp005 and ipp037 in rack
• Attached ipp037 SATA cables to the RAID card so they're mapped properly
• Noted fan warnings on nodes ipp014, 016, 008, 007, 004, 025, 024, and 045 (appear to be due to fan power board failures)

April 1

• ipp008
  • Put back into rack
  • Still missing single HD array fan

• ipp018
  • New motherboard; 2 new CPU fans (Sunon)
  • Put back in rack

• ipp037
  • New motherboard; 2 new CPU fans (Sunon)
  • Put back in rack
  • Array cables not mapped to RAID card correctly

• ipp014
  • Unsuccessfully attempted to get one of the HD array duo fans working
  • Appears to be an issue with the connector

March 22

• ipp018
  • Boots but goes to A:\>
  • Pulled CMOS battery; reset MB w/ jumper; no help

• ipp008
  • No boot
  • Rapid beeps
  • One pwr supply dead
  • Swapped in new MB
  • Replaced CPU fans with those taken from ipp037
  • Single HD array fan not working (appears to be module connector)

February 22

• Confirmed light on ippc17 drive
• Installed additional fan in ippc18 (small fan on right side)

February 16

• ipp008
  • No boot
  • rapid beeps -> then 1 sec beep

• ipp018
  • No boot
  • Both CPU fans not working - one has broken wire
  • No codes on MB
  • No lights on power supplies

• ipp037
  • No RAID card
  • CPUs installed w/o paste

February 11

• Installed an additional fan in ippc19 (small fan on right side)
• Swapped CPUs
  • ipp018 -> ipp005
• ipp037 -> ipp008

• Notes: ipp018 had excessive paste, bent pins on CPU2 (on MB), and an apparently scorched heatsink

NODE SWAPPING MAP

*Wave1 IPP17 died. Its disks were swapped into Wave2 IPP37

• So IPP17 is a Wave2 Machine.

*The broken Wave1 Machine (previously IPP17) was sent back. When it came back

• It had a different motherboard and was incomplete - missing fans.
• A newly broken IPP14 disks were swapped in.

*So IPP14 became IPP37.

• IPP14 ( Now IPP37 ) was out of warranty so we decided to swap the mother board and cpu's

• Summary:
  • IPP14 is a Wave 1 ASA Machine with a different motherboard.
  • IPP17 Is a Wave 2 YMI Machine
  • IPP37 is a Wave 1 Machine