== PS1 IPP Czar Logs for the week 2015.02.09 - 2015.02.15 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.02.09 ===
  * 08:35 EAM: stdscience and stdlocal both running slow, restarting both.
  * 19:30 EAM: summit is reporting clouds, wind, and other reasons for no data tonight.  i've turned on the storage nodes.  I've also added a macro to activate / de-activate the auto-off:
{{{
pantasks: storage.hosts.auto.off off <-- storage hosts will NOT automatically be turned off at night
pantasks: storage.hosts.auto.off on  <-- storage hosts WILL automatically be turned off at night
}}}

=== Tuesday : 2015.02.10 ===
  * 08:00 EAM: restarted stdlocal
  * 16:00 EAM: stopped staticsky, rebuilt psModules with the sign fix for lensing parameters, started sas pantasks (1x x0,x1,x2,x3), restarted staticsky with 3x x0,x1,x3
  * 19:50 EAM: restarting stdlocal

=== Wednesday : 2015-02-11 ===
 * 11:30 CZW: LANL DST day.  I've stopped stdlanl for a restart.
 * 12:30 MEH: QUB has not been getting any of the OSS.WS in (g),r,i filters because not setup fully.. doing so now and distribution may be busy for a while after doing a regular restart of the pantasks
 * 14:00 EAM: restarting staticsky (accidentally stopped when sas rerun finished).  I'm also taking ippx001-ippx004 out of rotation to use for dvo tests of relastro
 * 20:10 EAM: stdscience is getting close to the limit; stopping everything under ~ipp to restart everything
 * 22:30 MEH: ipp077 down for >1hr? -- disk available but unable to log into, console shows kernel panic
  * turn off in processing.. set neb repair 
  * no console prompt showing up.. needs a power cycle 
  * afternoon many errors 
{{{
<Feb/12 03:28 pm>[631239.579016] journal commit I/O error
}}}
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===