PS1 IPP Czar Logs for the week 2014.01.20 - 2014.01.26

(Up to PS1 IPP Czar Logs)

Monday : 2014.01.20

MLK

• 07:30 MEH: nightly finishing up okay

Tuesday : 2014.01.21

mark is czar

• 07:00 MEH: nightly slowly finishing, stdsci needs a regular restart later today, raising poll to 500 to help keep fully loaded
• 09:00 MEH: regular restart of many pantasks for the week -- registration,summitcopy,pstamp,update,distribution,publishing,stack,stdsci
• 09:30 appears to be a hanging QUB stamp 330715
  • 11:15 Bill: modified pstamp_checkdependent.pl to detect that input warp had quality error on update and fault the job
  • 14:00 MEH: manually set warp 879292 skycell.0803.024 quality to 0 and tried to update again, sets quality 8006 again
  • 15:00 MEH: manually set diff_id 509887 skycell.0803.024 to quality 42 then since will never update either
• 17:00 Haydn fixed mobo for ipp046, neb-host repair but still out of processing
• 17:05: MEH: update neb-host status
  • ippb01 back to repair
  • stsci00,01,02 repair in prep for move tomorrow
  • stsci03-09 back up since checkout okay
  • ipp031,032 must remain down until all stsci machines moved

Wednesday : 2014.01.22

Bill is czar today

• 05:45 set staticsky and cleanup to stop in preparation for the ippc18 shutdown and move
• 06:15 shut down all pantasks except staticsky, pstamp, and update
• 09:00 all processes shut down.
• 10:00 machines shut down
• 11:35 ippc18 booted
• 12:28 started roboczar and czarpoll scripts on ippc11
• ippc17 and the data store is up c30 is not so the postage stamp server apace is unavailable
• 14:32 started up the pantasks including ~ippsky/staticsky fullforce jobs are falling over because of stsci00-02 being down for moving. pstamp set to stop waiting for the working directories to be available.

Thursday : 2014.01.23

Bill is czar today

• 08:30 prepared for cab7 move. Down are ippc10, c12-c16, ippc19, db03, and ipp049-053
• 10:19 restarted pantasks. Moved stdscience to ippc02, and the servers usually on ippc15 to ippc03
• 10:35 changed czarpoll config file to point to scidbm (db01) instead of scidbs(db03)
  • note ippmonitor is on db03 so is not available. There is old version on ippdb01 mostly works, but doesn't show staticsky
• 11:18 ippsky/staticsky pantasks restarted
• 14:20 turned off staticsky tasks in the staticsky pantasks. I want to allow my fullforce test to finish.
• 14:30 cab 7 machines are back up. Set ipp049-53 to nebulous repair state. (There were a large number of chips failing because both copies of a detrend image were in that cabinet.
• 14:42 restarted czarpoll with nominal configuration
• 16:00 fixed about 40 chips whose burtool tables have gone missing. All class_ids were XY60 and XY61
• 16:15 set pantasks to stop
• 16:25 restarted distribution, summitcopy, and publishing on their regular system ippc15
• 16:30 restarted stdsciene on it's usual system ippc16

Friday : 2014.01.24

Saturday : 2014.01.25

Sunday : 2014.01.26