PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.01.30

• 11:15 CZW: Restarted ITC raw data shuffle to work on failures from previous iterations.
• 16:15 CZW: Restarting raw shuffle with randomized list to attempt to prevent overloading (~ipptest/replication on stare04).
• MEH: ipp004-031 neb-host repair so new data goes to them in preparation for the move to ITC/decomissioning
  • ipp056, ipp058 also neb-host repair since appears raid is using emergency spare and don't want nightly science on it in case problem while Haydn in Manoa for week
• MEH: data targeting to remove ipp004-031, nightly pantasks normally restarted to include change
• MEH: Haydn setting up new ippc64-ippc127 nodes at ITC using decommissioned machine IPs, so some warnings to ps-ipp-ops may pop up
• 21:30 CZW: Restarted shuffle pantasks pointing at permanent shuffle products (starting with camera). I've cut the number of parallel jobs significantly (to 30), to ensure this doesn't interact badly with nightly processing. This will also allow statistics to be gathered on this process.
• 22:05 MEH: ipp121 has crashed... looking at power cycle -- rebooted okay, running for a bit and then seems to be ext3 errors -- take out of processing and leave in neb-host repair (so MOPS can get stamps already possibly on the disks)

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD