PS1 IPP Czar Logs for the week 2015.02.09 - 2015.02.15

(Up to PS1 IPP Czar Logs)

Monday : 2015.02.09

• 08:35 EAM: stdscience and stdlocal both running slow, restarting both.
• 19:30 EAM: summit is reporting clouds, wind, and other reasons for no data tonight. i've turned on the storage nodes. I've also added a macro to activate / de-activate the auto-off:

  pantasks: storage.hosts.auto.off off <-- storage hosts will NOT automatically be turned off at night
  pantasks: storage.hosts.auto.off on  <-- storage hosts WILL automatically be turned off at night
  

Tuesday : 2015.02.10

• 08:00 EAM: restarted stdlocal
• 16:00 EAM: stopped staticsky, rebuilt psModules with the sign fix for lensing parameters, started sas pantasks (1x x0,x1,x2,x3), restarted staticsky with 3x x0,x1,x3
• 19:50 EAM: restarting stdlocal

Wednesday : 2015-02-11

• 11:30 CZW: LANL DST day. I've stopped stdlanl for a restart.
• 12:30 MEH: QUB has not been getting any of the OSS.WS in (g),r,i filters because not setup fully.. doing so now and distribution may be busy for a while after doing a regular restart of the pantasks
• 14:00 EAM: restarting staticsky (accidentally stopped when sas rerun finished). I'm also taking ippx001-ippx004 out of rotation to use for dvo tests of relastro
• 20:10 EAM: stdscience is getting close to the limit; stopping everything under ~ipp to restart everything
• 22:30 MEH: ipp077 down for >1hr? probably good to fix now rather than morning -- disk available but unable to log into, console shows kernel panic
  • turn off in processing.. set neb repair for night
  • no console prompt showing up.. needs a power cycle --
  • afternoon many errors? but wrong date so unknown -- is this the OS disk again?

    <Feb/12 03:28 pm>[631239.579016] journal commit I/O error
    

  • ipp077-20150211-crash.log

Thursday : 2015.02.12

• 14:45 : restarted stdlocal
• 18:33 : HAF : restart of stdsci/cleanup/pstamp/registration/summitcopy for start of night processing

Friday : 2015.02.13

• 10:50 MEH: restarting pstamp, temporarily adding s4+s5 as a boost for a large number of QUB updates

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD