PS1 IPP Czar Logs for the week 2011-05-09 - 2011-05-15

(Up to PS1 IPP Czar Logs)

Monday : 2011-05-09

Bill is czar today.

• 15:18 incomplete list of activities done so far today
  • restarted distribution with double the number of nodes (since it was way behind)
  • moved warp to it's own pantasks.
  • Now we have a few too many pclients running on some of the weaker machines. Some nodes have entered swap heck. So removed 2 of the 3 host entries in distribution for ipp013 and ipp037.
  • Created a new publishing client and associated data store IPP-MOPS-LAP
  • queued next set of runs for STS.2010.b (STS.b.20100831)
  • repaired 2 warp and one diff skycells that had corrupted files. (ipp050, ipp051, and ipp025)

Tuesday : 2011-05-10

Bill is czar today.

• Still getting lots of faults and file corruption. A couple of magicRun's got very damaged with zero sized streakMap files. Our tools aren't ready for this.
• 11:51 Stopping stdscience. The warp pantasks experiment is causing too much load on the system.
• 12:33 stdscience restarted including warp.

Wednesday : 2011-05-11

Heather is czar today

• Bill: 09:40 Still getting some file corruption. chip 229883 XY04 warp 196230.skycell.1198.051 Fixed with tools/run*
• 09:45 turned off chip processing to allow the nightly science warps to finish faster
• 11:13 chip processing turned back on.
• 19:25 ipp005 died. rebooted and set it to state down. Reverted the chipRuns that wanted files from there.

Thursday : 2011-05-12

• 07:30 ipp005 crashed again.
• 11:00 stopped processing to check in two fixes to ppStack: Remove unneeded assertion failure, and reject inputs with fhwm_major > 12.
• 11:28 Several MD08 diffs failed because they couldn't find the cmf from the reference stack. It turned out that one of the instances had zero size.
• 11:34 turned chip processing off to allow the LAP warps to catch up a bit.
• 13:25 cluster is quite unhappy. set stack poll to 20 to reduce the memory load.
• 14:50 moved stack out of stdscience into it's own pantasks using only compute nodes for processing.
• 15:19 restarted distribution because pcontrol was pegging one of ippc15's cpus at 100%
• 15ish setting up ippdb03 as the new ippdb01 replication server

Friday : 2011-05-13

Saturday : 2011-05-14

Sunday : 2011-05-15