PS1 IPP Czar Logs for the week 2012.04.09 - 2012.04.15

(Up to PS1 IPP Czar Logs)

Monday : 2012.04.09

• 09:30 Bill set all data with label ps_ud% to be cleaned
• 09:55 stopped pantasks in order to pick up a bug fix for psphotStack
• 10:02 set pantasks to run. Restarted deepstack. Reverted faults with fault == 4
• 11:40 changed state of staticskyRuns in state new.wait and abs(glat > 10) to new
• 12:01 Started running camera destreak restore in the distribution pantasks. To stop use "destreak.revert.off"
• 23:10 Mark: looks like ipp056 has become unresponsive, power cycling and back up.

Tuesday : 2012.04.10

Wednesday : 2012.04.11

• 12:00 Bill stopped processing in order to rebuild psphot to pick up a workaround to the memory explosion issue. If the number of peaks found is > 50,000 we exit psphotStack with fault 1
• 17:42 distribution pantasks was acting funny so Bill restarted it

Thursday : 2012.04.12

• 11:27 ipp017 is backing up processing because it is in swap heck trying to do 6 STS warps at a time. Set two hosts to off.

Friday : 2012.04.13

• 06:05 EAM : a hung mount point for ipp008 on ipp042 was blocking jobs on ipp042. I tried to force.umount ipp008 (after killing off the relevant jobs), but failed. I rebooted ipp042.

• 06:10 EAM : pantasks have been running since Apr 5, and stdscience is running slow as a result. I've shutdown everything and restarted it all.
• 09:00 bill: distribution is backing up turning off camera destreak restore (destreak.revert)
• 09:30 Bill: restarted deepstack

Saturday : 2012.04.14

Sunday : 2012.04.15