PS1 IPP Czar Logs for the week 2015.05.05 - 2015.05.11

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.05

Bill is the (absent minded) czar today

• 09:30 Bill: lap is stalled by missing files. Took 1 set of c2 nodes from stack and added to staticsky
• 11:10 MEH: running a batch fix of these broken burn.tbl to unstick LAP -- things moving again, huge backlog of stacks in process
• 12:30 CZW: One last burntool table seemed to be gumming things up. ipp_apply_burntool_single.pl --continue 2 --class_id XY64 --exp_id 265113 --this_uri neb://ipp051.0/gpc1/20101216/o5546g0024o/o5546g0024o.ota64.fits --previous_uri neb://ipp051.0/gpc1/20101216/o5546g0023o/o5546g0023o.ota64.fits --dbname gpc1 sorted it out (previous table was bad as well).
• 13:40 Bill: ipp051 is back up.
  • Haydn says ippc16 has an alarm and warning light. Took it down so that he could have a look at it.
• 14:20 MEH: turning ippsXX on in staticsky while preparing next deepstack run
• 14:48 Bill: after Haydn opened ippc16 up and reseated connectors, etc it stopped beeping. It is back on line.
• 15:31 Executed periodic shut down and restart of pantasks (except stack and staticsky).

Tuesday : 2015.05.06

Bill is czar today

• 06:52 removed lap label from stdscience while nightly science warps finish up
• 07:12 restarted staticsky in galactic plane with only 1 x c2 2 x m0 and 2 x m1 nodes enabled. Once I see the memory usage may add more c2s.
• 07:50 MEH: time to cycle out the ipps machines
• 13:58 staticsky memory usage is mostly staying below 25G. Trying 2 at a time.
• 14:57 stdscience restarted
• 15:37 started a script on ippc03 to delete the staticsky residual images which we don't need to save.
• 18:39 staticsky memory use is getting out of control. Set one set of the c2's to off

Wednesday : 2015.05.07

• 06:26 Bill: removed LAP label from stdscience until nightly warps finish
  • 07:36 added lap label back in
• 10:05 Bill enabled second set of c2 nodes in staticsky (nothing crashed last night and we should be getting further from the really high density regions).
• 11:00 ippc63 can be added back to the c2 group
• 16:00 Bill: 2 x c2 is growing vm use to > 80G in some cases. Back to 1x
• 18:20 Bill: cleared two diff faults with

      difftool -updatediffskyfile -set_quality 14006 -diff_id 545777 -fault 0 -skycell_id skycell.2121.090
      difftool -updatediffskyfile -set_quality 14006 -diff_id 545777 -fault 0 -skycell_id skycell.2192.037
  

Thursday : 2015.05.08

• 04:27 Bill: removed lap label from stdscience
• 05:10 set second set of c2 notes on in staticsky
• 07:56 bill: restarted stdscience pantasks
• 11:25 MEH: ipp051 has been out since 5/5 after new mobo installed... probably should be added back into processing..

Friday : 2015.05.09

mark is czar

• 08:30 MEH: large number of PSS chips fault 3, M31.rp.2013.20091223

   -> pmFPAfromFilename (pmFPAfileDefine.c:533): Problem in configure files
       Failed to determine camera format for neb://ipp021.0/gpc1/auxmask/mpg.20130506/det_969/mask_XY33_5174-5196.fits
  

  • Bill notes these cannot be updated w/ the auxmask and no alternative exists for the time being, to clear just send back to goto_cleaned
• 08:40 MEH: 2x c2 on in staticsky for single run (looks like c44,53,55 may have been fully off) + 1x c2 on in stack, once diffims finish then regular restart stdsci and pushing more power into stack
• 10:00 MEH: manual transfer power stdsci->stk since no chip->warp to do

  s3: -5 stdsci, +3 stk 
  s2: -4 stdsci, +2 stk
  s1: -3 stdsci, +2 stk
  

• 10:20 MEH: will keep stack out of c2 for a while, based on Bill's email and single run set of 2x looks ok for now -- 2x c2 for staticsky only
• 13:20 MEH: 1x c2 in stack on/off for a set while watching mem use+rate

Saturday : 2015.05.10

Sunday : 2015.05.11