PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.02.08

• 13:58 MEH: Haydn needs to rebuild drive in ippc10 so will be offline for bit
• 16:11 CZW: Starting raw data shuffle from the rest of the cluster to ippb03, which has some free disk space. This will be running in the replication pantasks as the ipp user. This should be a minimal load on the cluster, and will only run 10k jobs at a time (I'm using this to tune how quickly we fill hosts up). ippb03 will be set to up, but the xattr bit will prevent random file creation there.
  • 16:31 CZW: I've stopped this pantasks, as ippb03 seems to have issues with glockfile, and I'm not sure how to sort that out.
  • 17:20 CZW: Restarted again, as Gene helped debug the odd nfs/lockd/glockfile issue that was preventing ippb03 from responding correctly.
• 18:00 Haydn reporting ippdb03 recovery problematic -- will likely use ippc15 as a replacement unless someone using -- taking out of default nightly processing

Tuesday : YYYY.MM.DD

• 11:00 CZW Restarting rsync to ipp1XX. 10x nodes from ipp screen session on stare03. These don't always die cleanly, so if they need to actually die suddenly, email me.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD