== PS1 IPP Czar Logs for the week 2016.12.05 - 2016.12.11 ==
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.12.05 ===
 * MEH: large set QUB/MOPS warp updates/reprocessing continues and targeting ipp100+, running ~ippmops/stdscience using ipps and c2 (normally used for QUB WS and stacks) -- if pantasks is stopped, then you must notify me directly as it can block MOPS+QUB stamps 
=== Tuesday : 2016.12.06 ===

=== Wednesday : 2016.12.07 ===
 * MEH: ipp118-122.x were not completely setup and/or checked for ipp use, link for /local/ipp broken -- fixed, adding as group s7 in pantasks_hosts.input, adding to nightly science targeting and heavily loading CPU and IO with MOPS/QUB processing until weather clears
 * MEH: ippcXX nodes lately have been having NFS hangs that stall MOPS pstamp request (and QUB if there had been any data) -- restarting nfs/rpc.statd seems to clear -- czar needs to keep eye on 
 * MEH: stamps/updates for LAP.PV3 showing ongoing faults due missing files from stsci shuffle -- some reporting ipp118 but looks like due to IP being reused
 * MEH: gpc1 dumps to ipp001 have been 0 size for past couple days it appears due to /data/ipp001.0 being full -- adding an email warning crontab in ipp@ipp001, running cleanup script and rm'ing incomplete dumps so not to contaminate archive...
  * will set one up in ipp@ippc18 to also monitor homedir 
 * MEH: Haydn to replace ipp122 10G card and wanted to replace ipp083 BBU -- out of processing and neb-host down
  * ipp122 back up but cannot log into while disks mount ok -- leave out of processing and in repair -- then later was able to log into again, gmond restart and neb-host up
  * ipp083 unclear, leave in repair until morning -- BBU will be replaced in next few days, neb-host repair when nightly starts as it is in WT state
=== Thursday : 2016.12.08 ===
 * MEH: no nightly data tonight, leaving MOPS/QUB processing at day rate
=== Friday : 2016.12.09 ===
 * MEH: adding MOPS/QUB processing 1x to all ippx nodes to test stability for heavy processing
 * 16:00 CZW: Restarting IPP pantasks.
=== Saturday : 2016.12.10 ===

=== Sunday : 2016.12.11 ===