PS1 IPP Czar Logs for the week 2014-03-17 - 2014-03-23

(Up to PS1 IPP Czar Logs)

Monday : 2014-03-17

• 09:30 Bill storage.hosts.on, storage.hosts ignore in staticsky pantasks
• 11:15 at suggestion of serge and gene, kicked a diff to have a quality code of 42

Tuesday : 2014-03-18

mark is czar

• 07:05 MEH: clearing WS diffim fault 5 (532198,532231,532259,532292 skycell.1089.062; 532381 skycell.2608.044; 533099 skycell.1084.087, skycell.1084.096)
• 07:50 MEH: nightly finished, staticsky storage.hosts.on, storage.hosts.ignore
• 08:20 MEH: after ~4 days, restarting regular pantasks for the week
• 10:30 MEH: seeing load on ipp032 for mops and ippc30 for PSS, setting off in staticsky adds manually
  • ippc30 has MOPS stamps taking ~1min each rather than ~3/min normally, running staticsky on ippc30 probably isn't helping
• 18:00 MEH: extra nodes for staticsky off in time for nightly, storage.hosts.off
• 19:30 MEH: ipp053 looks to have gone unresponsive ~19:20.. seriously..
  • nothing on console, power cycling since non-responsive for over 10 min, ganglia showing drop of cpu to only ~20% cpu_wait @1900 so something probably just crashed
  • failed to boot on first power cycle, finally kicked over on second power cycle and one of the long boot screen systems for ram check..
  • leaving in repair and out of processing for tonight

Wednesday : 2014-03-19

Thursday : 2014-03-20

Friday : 2014-03-21

Saturday : 2014-03-22

Sunday : 2014-03-23