PS1 IPP Czar Logs for the week 2015.05.05 - 2015.05.11

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.05

Bill is the (absent minded) czar today

• 09:30 Bill: lap is stalled by missing files. Took 1 set of c2 nodes from stack and added to staticsky
• 11:10 MEH: running a batch fix of these broken burn.tbl to unstick LAP -- things moving again, huge backlog of stacks in process
• 12:30 CZW: One last burntool table seemed to be gumming things up. ipp_apply_burntool_single.pl --continue 2 --class_id XY64 --exp_id 265113 --this_uri neb://ipp051.0/gpc1/20101216/o5546g0024o/o5546g0024o.ota64.fits --previous_uri neb://ipp051.0/gpc1/20101216/o5546g0023o/o5546g0023o.ota64.fits --dbname gpc1 sorted it out (previous table was bad as well).
• 13:40 Bill: ipp051 is back up.
  • Haydn says ippc16 has an alarm and warning light. Took it down so that he could have a look at it.
• 14:20 MEH: turning ippsXX on in staticsky while preparing next deepstack run
• 14:48 Bill: after Haydn opened ippc16 up and reseated connectors, etc it stopped beeping. It is back on line.
• 15:31 Executed periodic shut down and restart of pantasks (except stack and staticsky).

Tuesday : 2015.05.06

Bill is czar today

• 06:52 removed lap label from stdscience while nightly science warps finish up
• 07:12 restarted staticsky in galactic plane with only 1 x c2 2 x m0 and 2 x m1 nodes enabled. Once I see the memory usage may add more c2s.
• 07:50 MEH: time to cycle out the ipps machines
• 13:58 staticsky memory usage is mostly staying below 25G. Trying 2 at a time.
• 14:57 stdscience restarted
• 15:37 started a script on ippc03 to delete the staticsky residual images which we don't need to save.
• 18:39 staticsky memory use is getting out of control. Set one set of the c2's to off

Wednesday : 2015.05.07

• 06:26 Bill: removed LAP label from stdscience until nightly warps finish
  • 07:36 added lap label back in
• 10:05 Bill enabled second set of c2 nodes in staticsky (nothing crashed last night and we should be getting further from the really high density regions).
• 11:00 ippc63 can be added back to the c2 group
• 16:00 Bill: 2 x c2 is growing vm use to > 80G in some cases. Back to 1x
• 18:20 Bill: cleared two diff faults with

      difftool -updatediffskyfile -set_quality 14006 -diff_id 545777 -fault 0 -skycell_id skycell.2121.090
      difftool -updatediffskyfile -set_quality 14006 -diff_id 545777 -fault 0 -skycell_id skycell.2192.037
  

Thursday : 2015.05.08

Friday : 2015.05.09

Saturday : 2015.05.10

Sunday : 2015.05.11