== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.04.27 ===
  * 8:12 Bill restarted postage stamp pantasks. Increased the poll limit for dependency checking to 160 to speed up the image updates.
  * 11:00 EAM: stopped ~ipp pantasks for Bill's pstamp upgrade
  * 13:10 EAM: restarted ~ipp pantasks post-Bill's pstamp upgrade
  * 17:25 EAM: Chris has generated separate left and right queues for pv3 full-force (left is running from 11h west, right is continuing from about 1hr going east).  I've moved the pv3 full-force processing to ~ippsky/pv3ffrt and ~ippsky/pv3fflt.  I've also added a new check nightly queue script which lives in ~ipp/check.nightly.queue.sh.  ~ipp/stdscience and both fforce pantasks execute this script.  the fforce pantasks turn off storage nodes if the outstanding nightly jobs are > 200, and shut off compute nodes if jobs > 1000.  Finally, I've updated the ippMonitor code to know about fullforce.

=== Tuesday : 2015.04.28 ===
 * 08:20 EAM: we are having stsci problems again.  last night, stsci06 crashed (it took gavin's intervention to reboot as the console was mis-mapped).  this morning, stsci15 crashed around 3am; after I booted it,  stsci17 crashed within 30 min (around 6:00). 
 * 11:20 MEH: ipps08 non-responsive on console, power cycle -- 
  * ipps00 also been behaving strangely and rebooting
 * 13:00 MEH: ipp063 disk warnings, put neb-host repair
 * 17:45 CZW: restarting ippsky pv3ffrt/pv3fflt.
=== Wednesday : 2015.04.30 ===
 * 10:46 MEH: ~ipp/raidstatus/ipp063 reports raid ok after disk fail yesterday. neb-host up again
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===