PS1 IPP Czar Logs for the week 2014-01-27 - 2014-02-02

(Up to PS1 IPP Czar Logs)

Monday : 2014-01-27

• 8:00 CZW: shutdown pantasks servers in advance of today's set of moves.
• 10:45 CZW: restarted nebulous apache servers, as they became cranky at some point. This was causing nebulous interactions to freeze. Post-restart, nebulous jobs appear to be completing as usual.
• 11:30 CZW: neb-host ipp024-ipp029
• 15:11 Bill: changed host status for ipp006-19 (except 17) from up to repair since they are going to be moved soon. changed neb-host status for ipp030 0 36 from down to repair since they have been moved.
  • 15:27 changed ipp017 from down to repair as well it was moved this morning
• 15:21 Bill restarted ~ippsky/staticsky, pstamp, and update pantasks
• 15:40 CZW: cab4 has been checked and appears fine, so I have updated the cabinet location in the nebulous database.
• 16:31 CZW: all cab3 hosts look fine with the exception of ipp023, which had problems last night anyway. I've set the rest of cab3 to repair, left ipp023 as "down", and am now bringing up the pantasks that haven't been restarted yet.
• 19:41 Bill set staticsky to run. It was stopped to investigate a potential problem.

Tuesday : 2014-01-28

• 17:30 CZW: Moves complete for the day; all hosts up. I've restarted pantasks for tonight, and set hosts to repair (I'm still finishing some of the disk scans).
• 21:55 Bill: set ippsky/staticsky to run

Wednesday : 2014-01-29

• 09:30 Bill restarted czarpoll which didn't seem to be updating, and roboczar just because (reminder: these are run in screen by ipp@ippc11 )
• 11:55 EAM : ipp028 died. various things were blocked on that partition, so stopped all processing, then laboriously checked each machine for hanging dfs, then killed jobs owned by ipp (but not ippsky), then did a force umount (tried until the df was cleared). restarted all services.
• 16:00 Bill after talking with Gene removed one set of compute3 nodes from stdscience and enabled one more in ippsky. More red on the board now.
• 16:43 Bill restarted stdscience including the ps_ud labels. The update pantasks is decomissioned for the time being.

Thursday : 2014-01-30

• 05:45 Bill : set staticsky to stop in preparation for server moves.
• 07:57 Bill : set stack to stop in " "

Friday :2014-01-31

Saturday : 2014-02-01

Sunday : 2014-02-02