PS1 IPP Czar Logs for the week 2012.03.05 - 2012.03.11

(Up to PS1 IPP Czar Logs)

Monday : 2012.03.05

• 20:07 Bill rebuilt psModules and psphot earlier to integrate some changes to fix psphotStack faults. Restarted stdscience.

Tuesday : 2012.03.06

Bill is czar today

• 10:15 psphotStack process on ipp060 has grown over 90GB in size. Since the node was doing other processing it got swapped out. Killed everything else off and removed the node from stack and stdscience pantasks and now it proceeding with the extended source analysis of the globular cluster M30. From the log

   total of 146208 sources for 6 images

• LAP seems to have run out of work to do. Perhaps it is waiting for the last few diffs to finish. They were bogged down because their files are on ipp060.
• 10:35 restarted stdscience. It was confused about the state of some of the jobs

Wednesday : 2012.03.07

Bill is czar today

• 11:34 stopped processing in preparation for picking up a couple of bug fixes in psModules
• 11:40 noticed that ipp014 went unresponsive a few hours ago. All sorts of jobs were stuck waiting for it. Gavin power cycled it.
• 12:15 stopped all pantasks, Issued pkill commmands on all nodes for ppImage psastro pswarp ppSub and ppStack
• 12:30 restarted processing
• 14:20 summitcopy is stopped while Craig fixes the fileset list for a broken fileset
• 14:48 summit copy back on last dark being copied

Thursday : 2012.03.08

• 07:00 Bill fixed a few XY26 instances. Killed a pswarp process that had been running for > 50000 seconds. After reverting the warp completed.
• 07:16 All pending LAP data through warp 803 stacks to process. Staticsky is proceeding but slowly. 2500s average time.
• 07:29 Set one set of wave 4 hosts to off in stdscience. Added one more set of wave 4 hosts to deepstack.

Roy is czar, but Bill has already fixed everything...thanks!

• 11:20 Mark is grabbing ippc54-ippc63 compute3 from stack pantasks to do some deepstack tests over next +12hrs. swapping ippc62 to ippc53 as bill has psphotStack running there.
• 15:25 CZW Cycled power on ipp008 as it was unresponsive.
• 16:10 Bill stopped processing to pick up a couple of bug fixes for psphotStack.
• 16:19 Bill killed off some camera_exp processes that were hung due to ipp008 going down.
• 16:20 Bill Restarted stdscience

Friday : 2012.03.09

Roy is czar.

• 08:30 Roy: No data last night, czartool looks clean.

Saturday : 2012.03.10

Sunday : 2012.03.11