PS1_IPP_Czarlog_20160208 – Pan-STARRS IPP

Context Navigation

Version 9 (modified by Mark Huber, 10 years ago) ( diff )
--

13:58 MEH: Haydn needs to rebuild drive in ippc10 so will be offline for bit
16:11 CZW: Starting raw data shuffle from the rest of the cluster to ippb03, which has some free disk space. This will be running in the replication pantasks as the ipp user. This should be a minimal load on the cluster, and will only run 10k jobs at a time (I'm using this to tune how quickly we fill hosts up). ippb03 will be set to up, but the xattr bit will prevent random file creation there.
- 16:31 CZW: I've stopped this pantasks, as ippb03 seems to have issues with glockfile, and I'm not sure how to sort that out.
- 17:20 CZW: Restarted again, as Gene helped debug the odd nfs/lockd/glockfile issue that was preventing ippb03 from responding correctly.
18:00 Haydn reporting ippdb03 recovery problematic -- will likely use ippc15 as a replacement unless someone using -- taking out of default nightly processing

11:00 CZW Restarting rsync to ipp1XX. 10x nodes from ipp screen session on stare03. These don't always die cleanly, so if they need to actually die suddenly, email me.
22:20 MEH: large number of faults unable to find detrends on ipp040 (ipp041,042,038 and others) of the group with rsync transfers lagging nightly processing -- wasn't it discussed to set down if running during nightly processing? jobs also lagging on ipp040 so turning the nightly jobs down at least there... others may need to be turned down as well (or the rsyncs..)