PS1 IPP Czar Logs for the week 2014.08.25 - 2014.08.31

(Up to PS1 IPP Czar Logs)

Monday : 2014.08.25

• 06:45 Bill: distribution's rcserver.make.fileset.load task has been timing out, so filesets were not getting posted to the data store. Ther are about 88500 of these short jobs pending. I debugged this a bit and found that if I have the MD06.nightlyscience label in the list the query takes > 10 minutes. If I remove it it is quite fast. Must be some strange interaction of the priority ordering because the md06 label has no filesets pending. Removed that label, verified that data started flowing again, then set distribution to stop so as not to interfere with nightlyscience processing.
  • please set the distribution pantasks to run when nightly processing completes

• 08:45 : ipp035 crashed, nothing no console. rebooting now (back up @ 8:46)
• 16:48 Bill: set distribution to run
• 17:50 MEH: another quick MD run using the compute3 nodes before nightly starts
• 17:55 Bill: restarted distribution pantasks

Tuesday : 2014.08.26

• 08:15 MEH: with the half-night of nightly finished, starting more MD pv2 on compute3 nodes
• 09:45 MEH: shouldn't need to pause MD while MOPS is making stamps, but keeping an eye on
• 11:22 Bill: queued more skycal distribution runs for processing. ~10 %left to go.
• 16:43 Bill: restarted the distribution pantasks which was getting sluggish after >~200,000 jobs

Wednesday : 2014.08.27

• 05:22 Bill: restated postage stamp server with a bug fix to pstamp_job_run.pl
  • 07:34 turned pstamp parser off in order to let backed up cleanup jobs free up some space
  • 08:05 removed MPE label from pstamp to see of the jobs are having a significant effect on stdscience throughput.
• 10:48 EAM : stopping all processing in prep of a UPS test @ MRTC-B
• 11:55 EAM : UPS test postponed : restarted processing
• 16:10 MEH: ipp043 very unhappy w/ cpu_wait and load -- putting into repair
• 16:15 MEH: s2+s3 being used by ippmd to distribute MD pv2 products -- finished before nightly starts

Thursday : 2014.08.28

• 05:00 Bill: removed MPIA and MPE labels from pstamp to try and get an idea if the stamp processing is having any effect on stdscience throughput.
  • 06:27 looks like the postage stamp processing does not noticeably affect the processing rate. labels back on
• 10:56 Bill: changed the labels for the final batch of lap skycal dist runs to run label and queued runs for the pole. Set distribution back to run. 39868 left to go
  • 18:33 set labels for skycal distRuns for LAP.ThreePi.20130717.pole3 to LAP.ThreePi.20130717 last 12,500 runs running. PV2 will then be finished. For IPP anyways

Friday : 2014.08.29

• 06:20 Bill: PV2 skycal distribution is done. Shut down distribution pantasks as it has nothing to do

Saturday : 2014.08.30

Sunday : 2014.08.31

• 01:15 MEH: looks like ipp034 is down since ~0100.. nothing on console, minor load --