PS1_IPP_Czarlog_20150427 – Pan-STARRS IPP

Context Navigation

Version 12 (modified by watersc1, 11 years ago) ( diff )
--

8:12 Bill restarted postage stamp pantasks. Increased the poll limit for dependency checking to 160 to speed up the image updates.
11:00 EAM: stopped ~ipp pantasks for Bill's pstamp upgrade
13:10 EAM: restarted ~ipp pantasks post-Bill's pstamp upgrade
17:25 EAM: Chris has generated separate left and right queues for pv3 full-force (left is running from 11h west, right is continuing from about 1hr going east). I've moved the pv3 full-force processing to ~ippsky/pv3ffrt and ~ippsky/pv3fflt. I've also added a new check nightly queue script which lives in ~ipp/check.nightly.queue.sh. ~ipp/stdscience and both fforce pantasks execute this script. the fforce pantasks turn off storage nodes if the outstanding nightly jobs are > 200, and shut off compute nodes if jobs > 1000. Finally, I've updated the ippMonitor code to know about fullforce.

08:20 EAM: we are having stsci problems again. last night, stsci06 crashed (it took gavin's intervention to reboot as the console was mis-mapped). this morning, stsci15 crashed around 3am; after I booted it, stsci17 crashed within 30 min (around 6:00).
11:20 MEH: ipps08 non-responsive on console, power cycle --
- ipps00 also been behaving strangely and rebooting
13:00 MEH: ipp063 disk warnings, put neb-host repair
17:45 CZW: restarting ippsky pv3ffrt/pv3fflt.

10:46 MEH: ~ipp/raidstatus/ipp063 reports raid ok after disk fail yesterday. neb-host up again

CZW: 13:30 Restart ffUHcray pantasks to ensure the rsync flood on the Cray was completely stopped.
CZW: 16:00 I've added ippx016 and ippx081 to the ffUHcray pantasks to help with the ff summary jobs. The input script will add them 4x by default, and I've manually added them another 4x. This 8x may end up the default if no one else runs on those nodes. Looking at the number of ffRuns in state 'new' with an entry in ffResult, we seem to have a backlog of about 11k right now. This may be suppressing the observed rate, as we're not limited by the normal steps but by the summary step.