| Version 7 (modified by , 16 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the weekend 2010.12.03 - 2010.10.05
(Up to PS1 IPP Czar Logs)
Friday : 2010.12.03
Bill is acting czar today. We are running STS.20101202 chip - warp) and MD03.refstack.20101202 stacks.
Apparently something has changed in ppStack and it is a memory hog. Certain skycells fail with memory allocation error and in some cases crash the node running the process.
3 hosts died with this problem a bit after 3 pm. ipp015, c27 and c29. Cindy and Gavin rebooted them. Then Gavin needed to clean up some nfs problems.
As a workaround we're going to stop processing of these stacks in stdscience and use a separate pantasks with a selection of hosts that have 32GB of physical memory. This will run out of ~ipp/stackonly with pantasks_server on ipp042.
Around 16:00 I disabled the two labels (using labeltool) and I am allowing the stacks that are running to finish. We were getting lots of faults from chip-warp because nodes running the stacks are intermittently unresponsive as nfs servers.
16:53 HST started up "stackonly" and restarted stdscience. I have not yet added the STS label to stdscience. Will do that after we see how the stacking progresses.
18:14 added STS.20101202 to stdscience and run. Lowered poll.limit to 64
