PS1 IPP Czar Logs for the week 2014.08.25 - 2014.08.31

(Up to PS1 IPP Czar Logs)

Monday : 2014.08.25

• 06:45 Bill: distribution's rcserver.make.fileset.load task has been timing out, so filesets were not getting posted to the data store. Ther are about 88500 of these short jobs pending. I debugged this a bit and found that if I have the MD06.nightlyscience label in the list the query takes > 10 minutes. If I remove it it is quite fast. Must be some strange interaction of the priority ordering because the md06 label has no filesets pending. Removed that label, verified that data started flowing again, then set distribution to stop so as not to interfere with nightlyscience processing.
  • please set the distribution pantasks to run when nightly processing completes

• 08:45 : ipp035 crashed, nothing no console. rebooting now (back up @ 8:46)
• 16:48 Bill: set distribution to run
• 17:50 MEH: another quick MD run using the compute3 nodes before nightly starts
• 17:55 Bill: restarted distribution pantasks

Tuesday : 2014.08.26

• 08:15 MEH: with the half-night of nightly finished, starting more MD pv2 on compute3 nodes
• 09:45 MEH: shouldn't need to pause MD while MOPS is making stamps, but keeping an eye on
• 11:22 Bill: queued more skycal distribution runs for processing. ~10 %left to go.
• 16:43 Bill: restarted the distribution pantasks which was getting sluggish after >~200,000 jobs

Wednesday : 2014.08.27

• 05:22 Bill: restated postage stamp server with a bug fix to pstamp_job_run.pl
  • 07:34 turned pstamp parser off in order to let backed up cleanup jobs free up some space
  • 08:05 removed MPE label from pstamp to see of the jobs are having a significant effect on stdscience throughput.

Thursday : 2014.08.28

Friday : 2014.08.29

Saturday : 2014.08.30

Sunday : 2014.08.31