PS1 IPP Czar Logs for the week 2014.01.20 - 2014.01.26

(Up to PS1 IPP Czar Logs)

Monday : 2014.01.20

MLK

• 07:30 MEH: nightly finishing up okay

Tuesday : 2014.01.21

mark is czar

• 07:00 MEH: nightly slowly finishing, stdsci needs a regular restart later today, raising poll to 500 to help keep fully loaded
• 09:00 MEH: regular restart of many pantasks for the week -- registration,summitcopy,pstamp,update,distribution,publishing,stack,stdsci
• 09:30 appears to be a hanging QUB stamp 330715
  • 11:15 Bill: modified pstamp_checkdependent.pl to detect that input warp had quality error on update and fault the job
  • 14:00 MEH: manually set warp 879292 skycell.0803.024 quality to 0 and tried to update again, sets quality 8006 again
  • 15:00 MEH: manually set diff_id 509887 skycell.0803.024 to quality 42 then since will never update either
• 17:00 Haydn fixed mobo for ipp046, neb-host repair but still out of processing
• 17:05: MEH: update neb-host status
  • ippb01 back to repair
  • stsci00,01,02 repair in prep for move tomorrow
  • stsci03-09 back up since checkout okay
  • ipp031,032 must remain down until all stsci machines moved

Wednesday : 2014.01.22

Bill is czar today

• 05:45 set staticsky and cleanup to stop in preparation for the ippc18 shutdown and move
• 06:15 shut down all pantasks except staticsky, pstamp, and update
• 09:00 all processes shut down.
• 10:00 machines shut down
• 11:35 ippc18 booted
• 12:28 started roboczar and czarpoll scripts on ippc11
• ippc17 and the data store is up c30 is not so the postage stamp server apace is unavailable
• 14:32 started up the pantasks including ~ippsky/staticsky fullforce jobs are falling over because of stsci00-02 being down for moving. pstamp set to stop waiting for the working directories to be available.

Thursday : 2014.01.23

Friday : 2014.01.24

Saturday : 2014.01.25

Sunday : 2014.01.26