PS1 IPP Czar Logs for the week 2016.01.04 - 2016.01.10

(Up to PS1 IPP Czar Logs)

Monday : 2016.01.04

• 13:01 MEH: will restart pantasks for night once publish finishes
• 18:16 MEH: ipp026 and ipp039 datanodes also only 1 cpu but used like other nodes w/ 2.. taking out for now -- pantasks hosts should probably have weak group re-established after last re-org

Tuesday : 2016.01.05

• 05:32 MEH: most of new datanodes are now red... there is data to clean up after iteration with MOPS but things are getting thin..
• 11:01 MEH: restarting pantasks
• 13:06 MEH: boosting pstamp for QUB requests

Wednesday : 2016.01.06

• 08:53 MEH: bumping pstamp again for QUB
• 15:27 MEH: starting some special reprocessing for MOPS in ippqub:stdscience_ws -- please do NOT restart the pantasks -- finished, all clear

Thursday : 2016.01.07

• 08:06 MEH: more special MOPS processing, do not restart stdscience_ws pantasks -- finished, all clear
• 20:30 EAM: ipp028 is down for the night: crashed and haydn could not reboot it remotely. i've put it down in nebulous

Friday : 2016.01.08

• 12:45 EAM : stopping and restarting pantasks for the night
• 14:36 MEH: more special MOPS processing, do not restart stdscience_ws pantasks -- finished

Saturday : 2016.01.09

• 10:55 MEH: boosting pstamp for QUB
• 14:45 MEH: more special MOPS processing, do not restart stdscience_ws pantasks
• 15:13 MEH: ipp028 has been up couple days now but only with 1 cpu -- should've be taken out of processing until balanced in ippconfig/pantasks_hosts.input.. also put to neb-host repair so data can be accessed..

Sunday : 2016.01.10

• 07:14 MEH: boosting pstamp for QUB
• 12:59 MEH: restarting all pantasks for tonight, boosting pstamp again for QUB (poll over 100k again)