PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.01.23

• MEH: QUB targeted followup needs ippx, ipps nodes as well as nightly nodes in AM -- finished
• 11:45 CZW: Increasing replication job count to 300, with the goal of pushing this back to 450 by the end of the day.
• MEH: MOPS moving from ipp028 to ipp027 as replacement while ipp032 is down -- neb-host repair and out of processing
• 14:57 CZW: Set ipp118 and ipp120 to down in nebulous to allow Haydn to scan the file system after the weekend outage.
• 15:00 CZW: Mounting issues are hanging up postage stamp generation. It looks like a number of hosts still have bad mounts, and that is preventing the jobs from completing. I've stopped stdscience while I try to sort out the problem.
• 15:09 CZW: The last two items appear to be somewhat related. ipp075 had jobs not running, and had a 'mount ipp118' job hanging. Killing this (which had been running since ~12:00) allowed all other mount related issues to clear (df being the obvious example), and jobs immediately started running. I'm going to search the cluster for similar jobs, kill them, and hope that ipp118 responds better once the file systems have been checked.
• 15:20 CZW: The hung pstamp jobs have cleared, without me completing my scan. I'm going to let stdscience continue, and troubleshoot anything that looks stuck.
• 17:13 CZW: ipp118 has been repaired, and I am setting it back to "repair" status in nebulous.
• 17:53 CZW: ipp120 has been repaired as well, and I have set both it and ipp118 to "up" in nebulous.

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD