PS1 IPP Czar Logs for the week 2014-04-07 - 2014-04-13

(Up to PS1 IPP Czar Logs)

Monday : 2014-04-07

Bill is czar today

• 13:00 restarted stdscience, pstamp, summitcopy, and registration pantasks
• 06:50 restarted stdscience withnew warp.pro that uses 2 x POLL_LIMIT for warp limit

Tuesday : 2014-04-08

• 03:30 Bill: we still have a warp backlog, but it may not be growing as quickly as usual. Removed LAP and ps_ud labels from stdscience.
  • definitely better balance between chip and warp than before. warp rate seems to be limited by number of jobs requiring stsci nodes.
• 05:30 status near end of night
  • 36 incomplete downloads
  • 21 new chip runs
  • 16 new camRuns
  • 19 new warpRuns
• 22:15 MEH: looks like stdsci crashed or was shutdown ~10 mins ago.. no note, so assuming crashed and restarting now

Wednesday : 2014-04-09

• 09:35 Bill: turned off chip stage to allow 3pi warps and diffs to finish up
  • 09:38 MOPS needs some updates. Turned chip on but set LAP label to inactive
• 10:02 nightly warps are done setting lap label back to active
• 12:10 MEH: MD observations made monday night 4/7 -- adding labels and such back into stdsci for processing
• 13:09 bill: removed LAP label from stdscience and stack. This afternoon we are going to use the horsepower for SAS.
  • reduced poll limit for sas 20 150 while stdscience finishes MD diffs from Monday.
  • 14:00 poll limit back up
• 14:04 added MD labels to distribution pantasks
• 15:06 bill : neb-host ippb02 down :(
• 16:43 bill : sas warps are done. Will queue stacks and run them now using hosts.stack. Since lap is off and we aren't getting much MD data the stack pantasks shouldn't be doing much.

Thursday : 2014-04-10

mark is czar

• 06:55 bill: nightly science warps are done. Adding LAP label to stdscience.
• 07:15 MEH: ipp060.0 has only 75M space available.. -- all systems are low, another night of data will fill at this rate even w/ cleanup

Friday : 2014-04-11

Saturday : 2014-04-12

Sunday : 2014-04-13