== PS1 IPP Czar Logs for the weekend 2010.12.03 - 2010.10.05 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Friday : 2010.12.03 ===
Bill is acting czar today. We are running STS.20101202 chip - warp) and MD03.refstack.20101202 stacks.

Apparently something has changed in ppStack and it is a memory hog. Certain skycells fail with memory allocation error and in some cases crash the node running the process.

3 hosts died with this problem a bit after 3 pm. ipp015, c27 and c29. Cindy and Gavin rebooted them. Then Gavin needed to clean up some nfs problems.

As a workaround we're going to stop processing of these stacks in stdscience and use a separate pantasks with a selection of hosts that have 32GB of physical memory. This will run out of ~ipp/stackonly with pantasks_server on ipp042.

Around 16:00 I disabled the two labels (using labeltool) and I am allowing the stacks that are running to finish. We were getting lots of faults from chip-warp because nodes running the stacks are intermittently unresponsive as nfs servers.

=== Saturday : 2010.12.04 ===

=== Sunday : 2010.12.05 ===