PS1 IPP Czar Logs for the week 2013.02.18 - 2013.02.24

(Up to PS1 IPP Czar Logs)

Monday : 2013.02.18

• 08:00 MEH: regular restart of stdscience to keep LAP rate up
• 22:00 MEH: running low on chip/warp/diff (pile-up of stacks in GP) so adding a few more runs 43->50

Tuesday : 2013.02.19

mark is czar

• 08:00 MEH: again restarting stdscience to keep the LAP rate up
• 08:40 MEH: ipp008-ipp010 taken out of processing in ppconfig/pantasks_hosts.input for Gene to use with ipptopsps runs
• 08:50 MEH: chip.revert.off while fixing LAP chips
• 09:30 MEH: giving stack an extra allocation of compute3 to try and more quickly push the pile of 3000 through -- without nightly science to process stacks never caught up over the weekend
• 10:50 MEH: out of chips/warps.. adding more LAP runs -- should've used the ~ipp/lap/trickle_add.pl script to slowly add 1x1, but was manually doing similarly so should be okay. in the past Chris notes some exposures get dupe processed.
• 10:57 Bill: ran 'queuessky.lap --ra_min 60 --ra_max 90 --go' which queued about 5800 five filter staticsky runs
  • MEH: set poll 30->10 until some of the large backlog of stacks move through since staticsky running in stack now
• 16:30 MEH: ippc03 has had warnings about disk space, there is/was an old old 2011 nebulous_server.old.log hanging onto 24G of disk
• 16:40 turning skycal.on back and in the stdscience/input file
• 17:50 Chris setup next RA hour slice for LAP
• 18:30 MEH: ippdb01 went into heavy WAIT_CPU state and all processing piled-up. stopping processing until worked out
• 19:40 MEH: things settling down, slowly restarting
• 23:25 MEH: LAP stacks mostly caught up so reallocating all back to default

Wednesday : 2013.02.20

heather is czar. woke up this morning and stdscience was down. I'm restarting it.

• 13:30 Bill: regenerated 2 missing burntool tables and recovered o5520g0235o.ota26.fits

Thursday : 2013.02.21

• 11:32 Bill: set stack pantasks to stop in preparation for a rebuild of ippTools and a database change.
• 12:00 Bill: all ~ipp pantasks shut down. rebuilding ipp-20121218
• 12:12 ippc17 has become unresponsive. Ganglia says that it happend about 900 seconds ago. Nothing on the console. Power cycling.
• 12:24 Bill: restarted all pantasks servers except cleanup and deepstack (which is not being used)
• 12:35 Bill: recovered o5499g0668o.ota26.fits and o5498g0681o.ota26.fits regenerated burntool tables for o5255g0082o.ota02.fits and o5255g0079o.ota11.fits
• 12:37 Bill: ippc17 can't see /data/ippb02.1

Friday : 2013.02.22

Saturday : 2013.02.23

Sunday : 2013.02.24