PS1 IPP Czar Logs for the week 2013.02.18 - 2013.02.24

(Up to PS1 IPP Czar Logs)

Monday : 2013.02.18

• 08:00 MEH: regular restart of stdscience to keep LAP rate up
• 22:00 MEH: running low on chip/warp/diff (pile-up of stacks in GP) so adding a few more runs 43->50

Tuesday : 2013.02.19

mark is czar

• 08:00 MEH: again restarting stdscience to keep the LAP rate up
• 08:40 MEH: ipp008-ipp010 taken out of processing in ppconfig/pantasks_hosts.input for Gene to use with ipptopsps runs
• 08:50 MEH: chip.revert.off while fixing LAP chips
• 09:30 MEH: giving stack an extra allocation of compute3 to try and more quickly push the pile of 3000 through -- without nightly science to process stacks never caught up over the weekend
• 10:50 MEH: out of chips/warps.. adding more LAP runs -- should've used the ~ipp/lap/trickle_add.pl script to slowly add 1x1, but was manually doing similarly so should be okay. in the past Chris notes some exposures get dupe processed.
• 10:57 Bill: ran 'queuessky.lap --ra_min 60 --ra_max 90 --go' which queued about 5800 five filter staticsky runs
  • MEH: set poll 30->10 until some of the large backlog of stacks move through since staticsky running in stack now
• 16:30 MEH: ippc03 has had warnings about disk space, there is/was an old old 2011 nebulous_server.old.log hanging onto 24G of disk
• 16:40 turning skycal.on back and in the stdscience/input file
• 17:50 Chris setup next RA hour slice for LAP
• 18:30 MEH: ippdb01 went into heavy WAIT_CPU state and all processing piled-up. stopping processing until worked out
• 19:40 MEH: things settling down, slowly restarting

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD