== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2018.04.09 ===
 * MEH: running updates for QUB on older compute nodes to finally test the kernel and network improvements, includes putting nodes ipp100-104,118-122 into repair to force data onto ipp123-126 -- looks like ippc40, 056 showing same network stalling problem (ipps04,14 only to a minor level) so unless the network changed again, the change in early March didn't help it 
  
=== Tuesday : 2018.04.10 ===
 * MEH: Gavin updating kernel on ippc23-c63 to 3.7.6 that the majority of nodes are at in order to prevent regular crashing -- c30-c63 will likely show stalling if reading from ipp123-126, but will test in detail when all updated (simple updates of any data product chip->warp clearly show issue if force some chip data to be put to ipp123-126 by setting ipp100-104,118-122 data targeted nodes into repair) 
  * 
 * MEH: mops needing access to more of the broken LAP.PV3 files -- reprocessing exposures o7023g0258o, o7023g0278o similarly as in [wiki:PS1_IPP_Czarlog_20180319] -- this should preserve the broken LAP files so any code for file fixes can be tested in updating the original broken files (in chip)
  * another set also reprocessed -- o6390g0419o, o6392g0504o, o6364g0328o (in chip); o6938g0644o, o6364g0343o, o6938g0626o (in warp so error_cleaned state)
=== Wednesday : 2018.04.11 ===
 * MEH: nightly stdscience running diffs for mops during day
 * MEH: turning down some of the general update test processing running on ippc23-c63, seeing some faults (unable to replicate files) in nightly processing -- 
=== Thursday : 2018.04.12 ===
 * MEH: more updates, diffs and publishing for MOPS during day
 * MEH: pstamp updates
=== Friday : 2018.04.13 ===
 * MEH: running updates, diffs, stacks for QUB in nightly pantasks during day
=== Saturday : YYYY.MM.DD ===

=== Sunday : 2018.04.15 ===
 * MEH: fixing more broken exposures missing files from LAP.PV3 for mops with basic reprocessing -- o6993g0297o,o6993g0313o