PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2015.04.27

• 8:12 Bill restarted postage stamp pantasks. Increased the poll limit for dependency checking to 160 to speed up the image updates.
• 11:00 EAM: stopped ~ipp pantasks for Bill's pstamp upgrade
• 13:10 EAM: restarted ~ipp pantasks post-Bill's pstamp upgrade
• 17:25 EAM: Chris has generated separate left and right queues for pv3 full-force (left is running from 11h west, right is continuing from about 1hr going east). I've moved the pv3 full-force processing to ~ippsky/pv3ffrt and ~ippsky/pv3fflt. I've also added a new check nightly queue script which lives in ~ipp/check.nightly.queue.sh. ~ipp/stdscience and both fforce pantasks execute this script. the fforce pantasks turn off storage nodes if the outstanding nightly jobs are > 200, and shut off compute nodes if jobs > 1000. Finally, I've updated the ippMonitor code to know about fullforce.

Tuesday : 2015.04.28

• 08:20 EAM: we are having stsci problems again. last night, stsci06 crashed (it took gavin's intervention to reboot as the console was mis-mapped). this morning, stsci15 crashed around 3am; after I booted it, stsci17 crashed within 30 min (around 6:00).
• 11:20 MEH: ipps08 non-responsive on console, power cycle --
  • ipps00 also been behaving strangely and rebooting

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD