== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017.12.11 ===
 * Haydn at ITC until Wednesday -- unclear what full plans are
  * ipp121 -- been up for a while but never notified w/ half ram -- put another stick in and to 3/4 (198GB) and the other stick must be bad -- since done early evening leave neb-host down until tomorrow and then put into repair
  * ipp124 OS backup drive failed, will need to shutdown to replace -- put in neb-host repair tonight so no nightly data goes on it --
  * stare00.1 -- drive failed so raid lost -- 

=== Tuesday : 2017.12.12 ===
 * MEH: ipp124 seems ok, put back neb-host up; ipp121 seems fine, put back neb-host repair
 * Hadyn replaced ipp068 mobo + raid card -- should test before leaves tomorrow -- mobo had older bios and wants update for cpu, didn't do since can be problems -- 
  * set neb-host repair 
  * running NCU 2016 chip+warp updates using group s4 (ipp067-071) and ipps to load ipp068 and compare to similar nodes -- target ipp123-126 only to not overload space

=== Wednesday : 2017.12.13 ===
 * Haydn working on ippc111 (new mobo),c118 (new power supply) in morning and back online now -- can be used in nightly processing again as needed
 * MEH: ipp124 high mem swap use -- restart gmond and fixed
 * MEH: ippc125 down, nothing on console -- Haydn powercycle @ITC
 * MEH: pstamp/ipp113.0 disk mostly full -- will block critical stamps for MOPS+QUB -- find old stamp bundles not properly cleaned to help free space -- 

=== Thursday : 2017.12.14 ===
 * MEH: ippc125 down, nothing on console again -- Haydn setting power off until can look at it (PSU fan problem?)

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===