== PS1 IPP Czar Logs for the week 2016.01.04 - 2016.01.10 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.01.04 ===
 * 13:01 MEH: will restart pantasks for night once publish finishes
 * 18:16 MEH: ipp026 and ipp039 datanodes also only 1 cpu but used like other nodes w/ 2.. taking out for now -- pantasks hosts should probably have weak group re-established after last re-org
=== Tuesday : 2016.01.05 ===
 * 05:32 MEH: most of new datanodes are now red... there is data to clean up after iteration with MOPS but things are getting thin..
 * 11:01 MEH: restarting pantasks
 * 13:06 MEH: boosting pstamp for QUB requests
=== Wednesday : 2016.01.06 ===
 * 08:53 MEH: bumping pstamp again for QUB
 * 15:27 MEH: starting some special reprocessing for MOPS in ippqub:stdscience_ws -- please do NOT restart the pantasks -- finished, all clear
=== Thursday : 2016.01.07 ===
 * 08:06 MEH: more special MOPS processing, do not restart stdscience_ws pantasks -- finished, all clear
 * 20:30 EAM: ipp028 is down for the night: crashed and haydn could not reboot it remotely.  i've put it down in nebulous

=== Friday : 2016.01.08 ===
 * 12:45 EAM : stopping and restarting pantasks for the night
 * 14:36 MEH: more special MOPS processing, do not restart stdscience_ws pantasks -- finished
=== Saturday : 2016.01.09 ===
 * 10:55 MEH: boosting pstamp for QUB
 * 14:45 MEH: more special MOPS processing, do not restart stdscience_ws pantasks 
 * 15:13 MEH: ipp028 has been up couple days now but only with 1 cpu -- should've be taken out of processing until balanced in ippconfig/pantasks_hosts.input.. also put to neb-host repair so data can be accessed..
=== Sunday : 2016.01.10 ===
 * 07:14 MEH: boosting pstamp for QUB
 * 12:59 MEH: restarting all pantasks for tonight, boosting pstamp again for QUB (poll over 100k again)