| Version 8 (modified by , 12 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2014.08.25 - 2014.08.31
(Up to PS1 IPP Czar Logs)
Monday : 2014.08.25
- 06:45 Bill: distribution's rcserver.make.fileset.load task has been timing out, so filesets were not getting posted to the data store. Ther are about 88500 of these short jobs pending. I debugged this a bit and found that if I have the MD06.nightlyscience label in the list the query takes > 10 minutes. If I remove it it is quite fast. Must be some strange interaction of the priority ordering because the md06 label has no filesets pending. Removed that label, verified that data started flowing again, then set distribution to stop so as not to interfere with nightlyscience processing.
- please set the distribution pantasks to run when nightly processing completes
- 08:45 : ipp035 crashed, nothing no console. rebooting now (back up @ 8:46)
- 16:48 Bill: set distribution to run
- 17:50 MEH: another quick MD run using the compute3 nodes before nightly starts
- 17:55 Bill: restarted distribution pantasks
Tuesday : 2014.08.26
- 08:15 MEH: with the half-night of nightly finished, starting more MD pv2 on compute3 nodes
- 09:45 MEH: shouldn't need to pause MD while MOPS is making stamps, but keeping an eye on
Wednesday : 2014.08.27
Thursday : 2014.08.28
Friday : 2014.08.29
Saturday : 2014.08.30
Sunday : 2014.08.31
Note:
See TracWiki
for help on using the wiki.
