== PS1 IPP Czar Logs for the week 2015.02.09 - 2015.02.15 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.02.09 ===
  * 08:35 EAM: stdscience and stdlocal both running slow, restarting both.
  * 19:30 EAM: summit is reporting clouds, wind, and other reasons for no data tonight.  i've turned on the storage nodes.  I've also added a macro to activate / de-activate the auto-off:
{{{
pantasks: storage.hosts.auto.off off <-- storage hosts will NOT automatically be turned off at night
pantasks: storage.hosts.auto.off on  <-- storage hosts WILL automatically be turned off at night
}}}

=== Tuesday : 2015.02.10 ===
  * 08:00 EAM: restarted stdlocal
  * 16:00 EAM: stopped staticsky, rebuilt psModules with the sign fix for lensing parameters, started sas pantasks (1x x0,x1,x2,x3), restarted staticsky with 3x x0,x1,x3
  * 19:50 EAM: restarting stdlocal

=== Wednesday : 2015-02-11 ===
 * 11:30 CZW: LANL DST day.  I've stopped stdlanl for a restart.
 * 12:30 MEH: QUB has not been getting any of the OSS.WS in (g),r,i filters because not setup fully.. doing so now and distribution may be busy for a while after doing a regular restart of the pantasks
 * 14:00 EAM: restarting staticsky (accidentally stopped when sas rerun finished).  I'm also taking ippx001-ippx004 out of rotation to use for dvo tests of relastro
 * 20:10 EAM: stdscience is getting close to the limit; stopping everything under ~ipp to restart everything
 * 22:30 MEH: ipp077 down for >1hr? probably good to fix now rather than morning -- disk available but unable to log into, console shows kernel panic
  * turn off in processing.. set neb repair for night
  * no console prompt showing up.. needs a power cycle -- 
  * afternoon many errors? but wrong date so unknown -- is this the OS disk again?
{{{
<Feb/12 03:28 pm>[631239.579016] journal commit I/O error
}}}
  * [wiki:ipp077-20150211-crash.log]

=== Thursday : 2015.02.12 ===

 * 14:45 : restarted stdlocal
 * 18:33 : HAF : restart of stdsci/cleanup/pstamp/registration/summitcopy for start of night processing
=== Friday : 2015.02.13 ===
 * 10:50 MEH: restarting pstamp, temporarily adding s4+s5 as a boost for a large number of QUB updates
 * 13:30 EAM: stopping & restarting stdlocal (>120k chip jobs)
 * 15:00 MEH: ipp010 looks to have crashed ~20 min ago
  * neb-host down
  * console only has <Feb/13 02:37 pm>ipp010 login: [3544735.219248] Disabling IRQ #78 -- no prompt, so power cycle
  * had mount issues after power cycle, restarted nfs a couple times and back up

=== Saturday : 2015.02.14 ===

 * 07:30 EAM : restarted stdlocal
 * 23:50 MEH: ps1 open and taking data, turn off storage nodes in stdlocal -- storage.hosts.auto.off on
=== Sunday : 2015.02.15 ===
 * 08:50 MEH: manually reverted distribution fault so doesn't get really stuck when diffs get cleaned tomorrow
{{{
disttool -revertrun -dbname gpc1 -fault 2 -label OSS.WS.nightlyscience
}}}

 * 20:20 MEH: looks like ipp044 has been down for >1hr and has blocked all nightly processing from needing detrends
  * neb-host down
  * will try to power cycle, if not come back or if crashes again will turn power off