​https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Production_Cluster_Status

2011 Production Node Work

Jan 21

replaced motherboard hardware and disks for ipp009, ipp012, ipp014.

ipp012 new disk 11 had to be swapped out. ipp009: 6 4BG sticks were installed but only 5 are showing. All the memory has been swapped out for new memory with the same result. ipp014: We hear continuous beeps on power on changing too 2 short beeps and a long one after something appears on the screen. However the 2 beeps and a long on are not constant - they become random. We don't see any memory report on the first screen like the others. We changed the mother board and memory with identical results. There was a dead fan issue but we rewired it and all the fans are working and the fan light is off. No effect on the beep pattern. The initial quick short beeps sound like memory beeps but it weird that we get it to boot at all. Left out of the rack.

The new mother boards do not have an IDE connection for the existing optical drives. We (Bill) drove to the only 2 computer stores on the island and purchased the only 2 notebook size optical drives with sata connectors available. However - once we installed these we realized the connectors were something we had not seen - and did not have the cables for. At first glance they look like sata connections but there is a power component as well. opps. Fortunately Brad had external usb optical drives we could use. Hundreds of them, in fact. Gavin had to Mac Gyvered them in place because the cables were too short to set them anywhere.

Jan 24

We put the power supplies from ipp009 in and it booted normally. However there are still fan related alarms sounding though all the fans are working. We swapped out the led/alarm board out with the same results.

ipp009 is up and ready for use but still short 4GB of memory. After many tests we can conclude that the problems lies in something other than the motherboard or memory. We had the same problem with a known good motherboard and memory. We narrowed it down to a set of black and blue memory slots associated with CPU0. Not the slots themselves, something that serves the slots. We know there is a problem with 2 slots because 12 x4GB memory sticks only showed 40 GB of memory. Also: The power supplies from ipp014 were put into ipp09 and there didn't seem to be any issues. Therefore we have no idea why ipp014 started booting normally with a power supply swap. Note: the ipp009 memory problem occurred with the good power supplies as well.

Jan 31

disk upgrades on ipp008, ipp016, and ipp021, and motherboard upgrades on ipp008 and ipp016.

Feb 4

We installed the new RAID card in ipp012 without problems.

However, we did not have success with the IDE->SATA converter board for the CD/DVD. The drive was not being detected on boot. We tried all SATA ports, different card, different CD/DVD, different cable to no avail. We checked the bios and there was note anything we could see that would prevent it from working. The external drive has been reattached. We brought a card and cd/dvd player back to test it and get it working here. It may need a jumper set to make sure it is not in slave mode - however we did not have a manual or a jumper - the boards did not come with one.

We also had the usual rack problems - ipp012 is particularly difficult to put back in - it is at the bottom of the rack and ipp013 is weighting down on it a bit.

Feb 9

Bill and I replaced the motherboards of ipp005, ipp006, ipp007, ipp025

Notables: ipp006 & ipp025 memory counts were missing 4GB. We identified which slot and put the stick in a black slot next to a blue working slot. All 24GB are now showing on both machines. The malfunctioning slots on the 2 machines were not the same. We tested the power cables and while we found some odd readings they were not consistent between the machines or relative to the cpu's with the malfunctioning slots.

The boards installed on the back of the CD drive are not working. We still have external CD drives hanging off the machines. We need to come up with a solution.

At some point in time - this fix should work for ipp009 so we have all 24 GB available.

ipp006: Voltage readings for CPU0 8pin power connector: +5v +12v +3.3V 5.2 12.5 3.4

-12V +12V 5vSB P6 11.9 12.2* 5.4* 410ms

• blinking

Voltage readings for CPU1 8pin power connector: same as cpu0 except for pg was 590ms and blinking too.

CPU1 slot closest to the cpu was not working.

ipp025: Voltage readings for CPU0 8pin power connector: Same display for ipp006 24 pin and 8 pin cpu0

Voltage readings for CPU0 8pin power connector: Same for CPU1 8pin. pg 430ms

May 16

replace a raid card in ipp005

July 15

Swapped the 1st 3 mem sticks out of ippdb00

July 27

Spent AM diagnosing the non-boot problem in ipp030 - failed MB After lunch Took failed MB out of ipp030 and replaced it with MB from ippc10. ippc10 remains out of the rack.

July 29

Added new memory to ippb00 1nd 01 - now have 32GB each.

Aug 1

Swapped all memory out of ippdb03 - formally ippc00 - using the memory from ippc10 - waiting for MB Added 2 2TB disks to ippc10 to become the newest rendition of ippdb03

Aug 15

ipp011 Upgraded to new MB and 2 TB drives. Had to move CPU0 memory stick from outer blue slot to middle black slot. Suppose to fill blue slots first. The memory was not detected in the outer black slot. Same weirdness seen in ipp009 ( according to Gavin ASA chassis)

ipp019 Upgraded to new MB and 2 TB drives. Memory all in blue slots and detected *The 1st new MB installed did not detect any memory in the cpu0 no matter where we put the sticks. Tried a second new MB using all the same memory and CPU's - with good results. Will RMA the 1st new MB.

ipp015 Upgraded to 2TB drives. On reboot saw only 12GB of memory. For this machine memory is suppose to be an all black slots. However we noted that the memory was in a mixture of black and blue slots. Moved to all blacks. All memory detected.

Aug 16

Installed the replacement MOBO for ippc10

Aug 23 2011

Haydn Bill and I were able upgrade ipp004,ipp013 and ipp018 MB and disks without incident.

We corrected the existing error in the RAID card cabling in ipp004 so now the drives are seated in the same order as the other machines ( funnily we discovered and remembered this problem because we had missed swapping a drive and put a 500GB back in which was showing in the wrong slot )

ipp037 has a new MB.

Gavin installed the OS and building the RAID - he was instrumental once again in supporting us while at MHPCC.

Haydn did all the drive replacements

Aug 30

Replaced IPP014's raid controller.

Aug 31

ipp010, ipp017, ipp020 (disk upgrades) and ipp021 (mobo upgrade).

Sept 13

Bill and Haydn Installed the new MOBO on ipp021

Sept 20

Bill and Haydn Installed the new MOBO on ipp020

Oct 10

Swapped CPU0 memory with CPU1 memory

2010 Production Node Work

(Up to Production Cluster Status)

Sept 15

• Removed cable obstructing power supply fan in ippc11

Sept 13

• Replaced CPUs (with new ones) and failed CPU fan in ipp014

Aug 23

• Swapped Memory out of IPP14

Aug 6

• Swapped Memory out of IPP12
• Upgraded IPP006 and IPP007 with 2TB drives

Jun 22

• Swapped Memory out of IPP14

April 27

• ipp018
  • swapped memory -> continuous beeps
  • Brad noticed one row of the hard drive array did not light up with red lights
  • Reseated SATA cables to backplane and RAID card
  • Swapped in set of new memory

April 26

• Swapped in larger RAID card memory module in ipp037
• ipp018
  • reseated memory -> still 2G memory
  • swapped CPUs -> still 2G memory showing
  • swapped in new set of memory (all 8 sticks) -> no boot rapid beeps

April 16

• Swapped two new CPUs into ipp037 and booted it into .31 kernel

April 8

• Swapped positions of ipp005 and ipp037 in rack
• Attached ipp037 SATA cables to the RAID card so they're mapped properly
• Noted fan warnings on nodes ipp014, 016, 008, 007, 004, 025, 024, and 045 (appear to be due to fan power board failures)

April 1

• ipp008
  • Put back into rack
  • Still missing single HD array fan

• ipp018
  • New motherboard; 2 new CPU fans (Sunon)
  • Put back in rack

• ipp037
  • New motherboard; 2 new CPU fans (Sunon)
  • Put back in rack
  • Array cables not mapped to RAID card correctly

• ipp014
  • Unsuccessfully attempted to get one of the HD array duo fans working
  • Appears to be an issue with the connector

March 22

• ipp018
  • Boots but goes to A:\>
  • Pulled CMOS battery; reset MB w/ jumper; no help

• ipp008
  • No boot
  • Rapid beeps
  • One pwr supply dead
  • Swapped in new MB
  • Replaced CPU fans with those taken from ipp037
  • Single HD array fan not working (appears to be module connector)

February 22

• Confirmed light on ippc17 drive
• Installed additional fan in ippc18 (small fan on right side)

February 16

• ipp008
  • No boot
  • rapid beeps -> then 1 sec beep

• ipp018
  • No boot
  • Both CPU fans not working - one has broken wire
  • No codes on MB
  • No lights on power supplies

• ipp037
  • No RAID card
  • CPUs installed w/o paste

February 11

• Installed an additional fan in ippc19 (small fan on right side)
• Swapped CPUs
  • ipp018 -> ipp005
• ipp037 -> ipp008

• Notes: ipp018 had excessive paste, bent pins on CPU2 (on MB), and an apparently scorched heatsink

NODE SWAPPING MAP

*Wave1 IPP17 died. Its disks were swapped into Wave2 IPP37

• So IPP17 is a Wave2 Machine.

*The broken Wave1 Machine (previously IPP17) was sent back. When it came back

• It had a different motherboard and was incomplete - missing fans.
• A newly broken IPP14 disks were swapped in.

*So IPP14 became IPP37.

• IPP14 ( Now IPP37 ) was out of warranty so we decided to swap the mother board and cpu's

• Summary:
  • IPP14 is a Wave 1 ASA Machine with a different motherboard.
  • IPP17 Is a Wave 2 YMI Machine
  • IPP37 is a Wave 1 Machine