PS1 IPP Czar Logs for the week 2013.10.28 - 2013.11.03

(Up to PS1 IPP Czar Logs)

Monday : 2013.10.28

• 12:00 heather: restarted stdsci - it had sine waves on ganglia
• 19:58 Bill: is running a pantasks_server on ipp064 out of ~bills/sas.30 running sas.30 staticsky. Once one set of compute3 nodes goes to off state in deepsky I will add one set of compute3 to the pantasks.
• 21:00 EAM : rebooted ipp041 as it has been having some NFS oddness
• 21:04 Bill: added compute3 to my sas.30 pantasks. Set ippc32, 33, 34, 35, 38, 40, 42, 45, 49, 58, and 62 to off since they are still working on stacks
• 23:30 MEH: doesn't look like will be much nightly, fixing LAP cam+warps that have languished a week to finish the stacks..

Tuesday : 2013.10.29

• 09:45 Bill: queued SAS.20131030 (EXTENDED_SOURCE_FITS_POISSON = FALSE)
• 11:20 MEH: regular restart of stdsci
• 12:29 Bill: set compute3 to off in stdscience to use for sas.30
  • 12:59 switched sas.30 to 2 x wave4 + 1 x compute 2 to avoid memory problems with deepstacks
  • put compute3 back into stdscience
  • stack is stopped as I am using the hosts
• 16:55 MEH: ipp061 nfs/mounts have been stuck for a while.. can't even access own export disk. restarting nfs and working again and backlogged jobs clearing
• 18:07 Bill: SAS.30 processing is done. Restarted stack pantasks
• 18:30 MEH: went though a large number of warps, doing a regular restart of stdsci before nightly

Wednesday : 2013.10.30

Bill is czar today

• 04:50 restarted pstamp and update pantasks
• 11:45 MEH: stdsci needs regular restart, also turning MD03,04 diffs with new refstacks -- tweak_ssdiff to get the MD03 marginal from last night
• 14:50 MEH: need to get through backlog of chip cleanup, other stages off

Thursday : 2013.10.31

Friday : 2013.11.01

Saturday : 2013.11.02

Sunday : 2013.11.03