PS1 IPP Czar Logs for the week 2014.07.07 - 2014.07.13

(Up to PS1 IPP Czar Logs)

Monday : 2014.07.07

• 12:00 Bill: turned parser and finish tasks off in pstamp in prepartion for restart and cleanup of ps_ud labels
  • 12:04 all jobs waiting for updates have finished. Set chip, warp, and diff runs with label like ps_ud% to be cleaned
  • 12:23 pstamp pantasks restarted
• 14:40 EAM : had a failing warp, set to bad quality:

  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 3006 -warp_id 976390 -skycell_id skycell.0698.007
  

• 20:40 CZW: issued "run" command to all pantasks to get them going for nightly processing.

Tuesday : 2014.07.08

• 07:50 EAM : the chiller at MRTC-B failed in the night and Haydn had to shut down the whole cluster. machines are slowly coming back up now.
• 13:00 EAM : gavin brought back up the machines which could be power-cycled from Manoa. ippdb00 had trouble starting mysql : it needed to recover from the binlogs, and this was slow, but eventually succeeded. jaws machines needed to be manually booted by Haydn with some difficulty. ipp030 also gave some trouble. I initially proposed that we move ippdb01 as well today, but it took so long for Haydn to get past security that we gave up on that plan for today. At this point, processing is running, but there are lots of fault, probably due to block nebulous files. I need to clear these out before we will be in reasonable shape.

• 13:45 EAM : ipp059 just crashed: wiki:ipp059-20140708-crash.log

Wednesday : 2014.07.09

• 02:00 Bill : stdscience pantasks has apparently not been running since 22:45 or so last night. Nothing in pantasks.stderr.log except the startup messages. panatsks.stdout.log is dated 22:45. Last like in pcontrol log is "caught parent shutdown"

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD