PS1 IPP Czar Logs for the week 2013.10.28 - 2013.11.03

(Up to PS1 IPP Czar Logs)

Monday : 2013.10.28

• 12:00 heather: restarted stdsci - it had sine waves on ganglia
• 19:58 Bill: is running a pantasks_server on ipp064 out of ~bills/sas.30 running sas.30 staticsky. Once one set of compute3 nodes goes to off state in deepsky I will add one set of compute3 to the pantasks.
• 21:00 EAM : rebooted ipp041 as it has been having some NFS oddness
• 21:04 Bill: added compute3 to my sas.30 pantasks. Set ippc32, 33, 34, 35, 38, 40, 42, 45, 49, 58, and 62 to off since they are still working on stacks
• 23:30 MEH: doesn't look like will be much nightly, fixing LAP cam+warps that have languished a week to finish the stacks..

Tuesday : 2013.10.29

• 09:45 Bill: queued SAS.20131030 (EXTENDED_SOURCE_FITS_POISSON = FALSE)
• 11:20 MEH: regular restart of stdsci
• 12:29 Bill: set compute3 to off in stdscience to use for sas.30
  • 12:59 switched sas.30 to 2 x wave4 + 1 x compute 2 to avoid memory problems with deepstacks
  • put compute3 back into stdscience
  • stack is stopped as I am using the hosts
• 16:55 MEH: ipp061 nfs/mounts have been stuck for a while.. can't even access own export disk. restarting nfs and working again and backlogged jobs clearing
• 18:07 Bill: SAS.30 processing is done. Restarted stack pantasks
• 18:30 MEH: went though a large number of warps, doing a regular restart of stdsci before nightly

Wednesday : 2013.10.30

Bill is czar today

• 04:50 restarted pstamp and update pantasks
• 11:45 MEH: stdsci needs regular restart, also turning MD03,04 diffs with new refstacks -- tweak_ssdiff to get the MD03 marginal from last night
• 14:50 MEH: need to get through backlog of chip cleanup, other stages off
• 18:30 MEH: again >100k in warps, restarting stdsci before nightly starts
• 18:36 Bill: restarted registration and summit copy pantasks (their pcontrols were spinning)
• reverted faulted chip after

  copied valid /data/ippb03.1/nebulous/42/5a/2169045416.gpc1:20120421:o6038g0140o:o6038g0140o.ota64.fits 
  over corrupt /data/ipp043.0/nebulous/42/5a/2021195867.gpc1:20120421:o6038g0140o:o6038g0140o.ota64.fits
  

• 20:20 MEH: looks like ipp061 is unhappy..
• 20:30 Bill killed hung summitcopy and registration processes on ipp061 took it out of the host lists. Sent email to heather about overload by 96GB ippdvo java process
• 20:51 java killed but ipp061 is still in a bad way. There are some stack processes there that are probably stuck forever. Tried restarting nfs but it still can't see some nodes. Machine probably needs a power cycle
• 01:00 MEH: cleared md09 stack fault 5, stack_id 2881582

Thursday : 2013.10.31

Bill is czar today

• 05:55 ipp038 has crashed. Nothing on the console. Attempted power cycle, but no response.
• 06:25 set ipp038 to down in nebulous to prevent processes from looking for detrend files with instances there
• 06:42 many processes are stuck and have been since 05:02 about the time ipp038 died. Setting stdscience to stop to make it clear which processes to kill
• 07:24 old stdscience processes killed. stdscience restarted with poll limit 42
• 07:30 Looks ok. poll limit 200
• 07:43 several processes may be stuck. will wait a while longer
• 08:17 All pantasks set to stop.
• 08:34 letting things settle cleared out all stdscience processes except those on ipp061. Taking that out of processing (forgot this morning). Set all pantasks to run except for the stackers. Going to wait for a while to make sure things are stable before setting those to run again.
• going out for a bit to get something to eat.
• 09:26 restarted stack pantasks. Deepstack processes are proceeding slowly. stdscience processes are taking cycles which is fine but using memory which is causing the ppStakcs (some approaching 60GB ram) to swap out. Turning compute 3 down in stdscience.
  • MEH: this has been fine in the past -- particularly since updates as the stdsci processing

Friday : 2013.11.01

Bill is czar today

• 08:52 restarted stdscience pantasks. Left compute3 on.
• mid morning stopped processing so that Haydn could reboot stdsci14.
• 1:45 ipp038 is back up. Set to repair in nebulous
• 1:50 setting deepstack to stop. I need compute3 to rerun the SAS.

Saturday : 2013.11.02

Sunday : 2013.11.03