PS1 IPP Czar Logs for the week 2012.03.05 - 2012.03.11

(Up to PS1 IPP Czar Logs)

Monday : 2012.03.05

• 20:07 Bill rebuilt psModules and psphot earlier to integrate some changes to fix psphotStack faults. Restarted stdscience.

Tuesday : 2012.03.06

Bill is czar today

• 10:15 psphotStack process on ipp060 has grown over 90GB in size. Since the node was doing other processing it got swapped out. Killed everything else off and removed the node from stack and stdscience pantasks and now it proceeding with the extended source analysis of the globular cluster M30. From the log

   total of 146208 sources for 6 images

• LAP seems to have run out of work to do. Perhaps it is waiting for the last few diffs to finish. They were bogged down because their files are on ipp060.
• 10:35 restarted stdscience. It was confused about the state of some of the jobs

Wednesday : 2012.03.07

Bill is czar today

• 11:34 stopped processing in preparation for picking up a couple of bug fixes in psModules
• 11:40 noticed that ipp014 went unresponsive a few hours ago. All sorts of jobs were stuck waiting for it. Gavin power cycled it.
• 12:15 stopped all pantasks, Issued pkill commmands on all nodes for ppImage psastro pswarp ppSub and ppStack
• 12:30 restarted processing
• 14:20 summitcopy is stopped while Craig fixes the fileset list for a broken fileset
• 14:48 summit copy back on last dark being copied

Thursday : 2012.03.08

• 07:00 Bill fixed a few XY26 instances. Killed a pswarp process that had been running for > 50000 seconds. After reverting the warp completed.
• 07:16 All pending LAP data through warp 803 stacks to process. Staticsky is proceeding but slowly. 2500s average time.

Friday : 2012.03.09

Saturday : 2012.03.10

Sunday : 2012.03.11