PS1 IPP Czar Logs for the week 2014.12.01 - 2014.12.07

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.01

• 09:53 Bill: restarted distribution pantasks which was sluggish with diff stage enabled. SAS distribution is all finished except for diff stage.
• 13:00 Bill: ~ippsky is rerunning staticsky and skycal for sas.37
• 17:00 MEH: setting up ippx049+ for ippsky staticsky/cal to use -- manually adding when ready, ~3x should be fine for 50G ram
• 22:00 MEH: nightly processing not doing well w/ ippmd and ippsky running (or something else?) -- ipp083-087 overloaded/not responding and stalling registration

Tuesday : 2014.12.02

• 07:11 Bill: started up fullforce in ~ippsky/staticsky (staticsky and skycal are finished except for one skycell that faulted with a data error. It is running now
• 07:30 MEH: registration backed up quite a bit, too many nodes in use -- ippmd stop until nightly finishes
  • looks like something may have started ~0200, nightly rate dropped greatly then as well..

Wednesday : 2014.12.03

Thursday : 2014.12.04

Friday : 2014.12.05

Saturday : 2014.12.06

• planned downtime MRTC-B 590B Lipoa Automatic Transfer Switch (ATS) -- Saturday December 6th. They will be starting our shutdown processes at 9am and will hopefully start turning things back on at 3pm.
  • all processing stopped before ~8am
  • Haydn on site to start power down of systems @8am

Sunday : 2014.12.07