| Version 17 (modified by , 16 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the weekend 2010.12.03 - 2010.10.05
(Up to PS1 IPP Czar Logs)
Friday : 2010.12.03
Bill is acting czar today. We are running STS.20101202 chip - warp) and MD03.refstack.20101202 stacks.
Apparently something has changed in ppStack and it is a memory hog. Certain skycells fail with memory allocation error and in some cases crash the node running the process.
3 hosts died with this problem a bit after 3 pm. ipp015, c27 and c29. Cindy and Gavin rebooted them. Then Gavin needed to clean up some nfs problems.
As a workaround we're going to stop processing of these stacks in stdscience and use a separate pantasks with a selection of hosts that have 32GB of physical memory. This will run out of ~ipp/stackonly with pantasks_server on ipp042.
Around 16:00 I disabled the two labels (using labeltool) and I am allowing the stacks that are running to finish. We were getting lots of faults from chip-warp because nodes running the stacks are intermittently unresponsive as nfs servers.
16:53 HST started up "stackonly" and restarted stdscience. I have not yet added the STS label to stdscience. Will do that after we see how the stacking progresses.
18:14 added STS.20101202 to stdscience and run. Lowered poll.limit to 64. Later turned warp off.
Turned hosts running stacks off in stdscience ipp040 - 52 (less 42,44)
22:56 since we're getting new data on the summit tonight, I turned off stacking so as to keep things from potentially being unstable. Turned warp back on.
23:15 Lots of faults from ippc27. (wasn't that machine down for the past week? Set it off in pantasks) Also turned hosts 40 -52 back on
23:20 Lots of faults from ipp048. The problem was the nfs access to ipp006 was broken. sudo /usr/local/sbin/force.umount ipp006 seems to have fixed that.
23:13 past bedtime. No faults on the czartool dash board. set.poll 180 (the usual value) good night
Saturday : 2010.12.04
06:00 many faults. Reverted chip but they came right back. Examining pantasks.stdout.log all of the faults are for jobs dispatched to ippc07. That node was getting I/O errors trying to talk to ipp027. force.umount fixed the problem. All faults reverted. Looking at ippc07:/var/log/messages ipp027 stopped responding at 03:50. Ganglia data for ipp027 doesn't show any unusual activity.
Summit copy and registration are proceeding. Last exposure so far is o5534g0566o. We're about 33 exposures behind with downloads This is likely due to the 132 stare exposures taken.
06:18 Noticed that publishing wasn't running. Started it up.
06:35 All object exposures from last night have been copied and registered. 20 STS chipRuns still processing.
07:11 Noticed that cleanup wasn't running. Fired it up.
