== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2018.04.09 ===
 * MEH: running updates for QUB on older compute nodes to finally test the kernel and network improvements, includes putting nodes ipp100-104,118-122 into repair to force data onto ipp123-126 -- looks like ippc40, 056 showing same network stalling problem (ipps04,14 only to a minor level) so unless the network changed again, the change in early March didn't help it 
  
=== Tuesday : 2018.04.10 ===
 * MEH: Gavin updating kernel on ippc23-c63 to 3.7.6 that the majority of nodes are at in order to prevent regular crashing -- c30-c63 will likely show stalling if reading from ipp123-126, but will test in detail when all updated (simple updates of any data product chip->warp clearly show issue if force some chip data to be put to ipp123-126 by setting ipp100-104,118-122 data targeted nodes into repair) 
  * 
 * MEH: mops needing access to more of the broken LAP.PV3 files -- reprocessing exposures o7023g0258o, o7023g0278o similarly as in [wiki:PS1_IPP_Czarlog_20180319] -- this should preserve the broken LAP files so any code for file fixes can be tested in updating the original broken files (in chip)
  * another set also reprocessed -- o6390g0419o, o6392g0504o, o6364g0328o (in chip); o6938g0644o, o6364g0343o, o6938g0626o (in warp so error_cleaned state)

=== Wednesday : 2018.04.11 ===
 * MEH: nightly stdscience running diffs for mops during day
 * MEH: turning down some of the general update test processing running on ippc23-c63, seeing some faults (unable to replicate files) in nightly processing -- 

=== Thursday : 2018.04.12 ===
 * MEH: more updates, diffs and publishing for MOPS during day
 * MEH: continuing updates runs to test kernel update on ippc23-c63
 * MEH: pstamp update -- adding option for MOPS to provide all images having pixels within the ROI (no stitching) and only by a very specific request type -- REQ_TYPE byexp, COORD_MASK 0 (sky coords) and OPTION_MASK include 8192 (last unused value?) -- r40389 and r40395

=== Friday : 2018.04.13 ===
 * MEH: running updates, diffs, stacks for QUB in nightly pantasks during day
 * MEH: fixing more broken exposures from LAP.PV3 missing files with basic reprocessing (label/data_group mopsfix/mops.fixbrokenLAP.20180323) -- o7001g0316o,o7001g0295o,o6919g0700o,o6290g0138o,o6290g0119o
 * MEH: MOPS behind large QUB pstamp request wanting same warps -- manually update chip+warps setting label to ps_ud_MOPS and remove QUB label from pstamp so dependent in limit query can be found and stamps finished for MOPS

=== Saturday : YYYY.MM.DD ===

=== Sunday : 2018.04.15 ===
 * MEH: fixing more broken exposures missing files from LAP.PV3 for mops with basic reprocessing -- o6993g0297o,o6993g0313o