== PS1 IPP Czar Logs for the week 2013.10.28 - 2013.11.03 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.10.28 ===
  
 * 12:00 heather: restarted stdsci - it had sine waves on ganglia
 * 19:58 Bill: is running a pantasks_server on ipp064 out of ~bills/sas.30 running sas.30 staticsky. Once one set of compute3 nodes goes to off state in deepsky I will add one set of compute3 to the pantasks.
 * 21:00 EAM : rebooted ipp041 as it has been having some NFS oddness
 * 21:04 Bill: added compute3 to my sas.30 pantasks. Set ippc32, 33, 34, 35, 38, 40, 42, 45, 49, 58, and 62 to off since they are still working on stacks
 * 23:30 MEH: doesn't look like will be much nightly, fixing LAP cam+warps that have languished a week to finish the stacks..
=== Tuesday : 2013.10.29 ===
 * 09:45 Bill: queued SAS.20131030 (EXTENDED_SOURCE_FITS_POISSON = FALSE)
 * 11:20 MEH: regular restart of stdsci
 * 12:29 Bill: set compute3 to off in stdscience to use for sas.30
   * 12:59 switched sas.30 to 2 x wave4 + 1 x compute 2 to avoid memory problems with deepstacks
   * put compute3 back into stdscience
   * stack is stopped as I am using the hosts
 * 16:55 MEH: ipp061 nfs/mounts have been stuck for a while.. can't even access own export disk. restarting nfs and working again and backlogged jobs clearing
 * 18:07 Bill: SAS.30 processing is done. Restarted stack pantasks
 * 18:30 MEH: went though a large number of warps, doing a regular restart of stdsci before nightly
=== Wednesday : 2013.10.30 ===
Bill is czar today
 * 04:50 restarted pstamp and update pantasks
 * 11:45 MEH: stdsci needs regular restart, also turning MD03,04 diffs with new refstacks -- tweak_ssdiff to get the MD03 marginal from last night
 * 14:50 MEH: need to get through backlog of chip cleanup, other stages off
=== Thursday : 2013.10.31 ===

=== Friday : 2013.11.01 ===

=== Saturday : 2013.11.02 ===

=== Sunday : 2013.11.03 ===