PS1 IPP Czar Logs for the week 2015.02.09 - 2015.02.15

(Up to PS1 IPP Czar Logs)

Monday : 2015.02.09

• 08:35 EAM: stdscience and stdlocal both running slow, restarting both.
• 19:30 EAM: summit is reporting clouds, wind, and other reasons for no data tonight. i've turned on the storage nodes. I've also added a macro to activate / de-activate the auto-off:

  pantasks: storage.hosts.auto.off off <-- storage hosts will NOT automatically be turned off at night
  pantasks: storage.hosts.auto.off on  <-- storage hosts WILL automatically be turned off at night
  

Tuesday : 2015.02.10

• 08:00 EAM: restarted stdlocal
• 16:00 EAM: stopped staticsky, rebuilt psModules with the sign fix for lensing parameters, started sas pantasks (1x x0,x1,x2,x3), restarted staticsky with 3x x0,x1,x3
• 19:50 EAM: restarting stdlocal

Wednesday : 2015-02-11

• 11:30 CZW: LANL DST day. I've stopped stdlanl for a restart.
• 12:30 MEH: QUB has not been getting any of the OSS.WS in (g),r,i filters because not setup fully.. doing so now and distribution may be busy for a while after doing a regular restart of the pantasks
• 14:00 EAM: restarting staticsky (accidentally stopped when sas rerun finished). I'm also taking ippx001-ippx004 out of rotation to use for dvo tests of relastro
• 20:10 EAM: stdscience is getting close to the limit; stopping everything under ~ipp to restart everything
• 22:30 MEH: ipp077 down for >1hr? probably good to fix now rather than morning -- disk available but unable to log into, console shows kernel panic
  • turn off in processing.. set neb repair for night
  • no console prompt showing up.. needs a power cycle --
  • afternoon many errors? but wrong date so unknown -- is this the OS disk again?

    <Feb/12 03:28 pm>[631239.579016] journal commit I/O error
    

  • ipp077-20150211-crash.log

Thursday : 2015.02.12

• 14:45 : restarted stdlocal
• 18:33 : HAF : restart of stdsci/cleanup/pstamp/registration/summitcopy for start of night processing

Friday : 2015.02.13

• 10:50 MEH: restarting pstamp, temporarily adding s4+s5 as a boost for a large number of QUB updates
• 13:30 EAM: stopping & restarting stdlocal (>120k chip jobs)
• 15:00 MEH: ipp010 looks to have crashed ~20 min ago
  • neb-host down
  • console only has <Feb/13 02:37 pm>ipp010 login: [3544735.219248] Disabling IRQ #78 -- no prompt, so power cycle
  • had mount issues after power cycle, restarted nfs a couple times and back up

Saturday : 2015.02.14

• 07:30 EAM : restarted stdlocal
• 23:50 MEH: ps1 open and taking data, turn off storage nodes in stdlocal -- storage.hosts.auto.off on

Sunday : 2015.02.15

• 08:50 MEH: manually reverted distribution fault so doesn't get really stuck when diffs get cleaned tomorrow

  disttool -revertrun -dbname gpc1 -fault 2 -label OSS.WS.nightlyscience
  

• 20:20 MEH: looks like ipp044 has been down for >1hr and has blocked all nightly processing from needing detrends
  • neb-host down -- so we can do nightly processing
  • will try to power cycle, if not come back or if crashes again will turn power off