PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2018.04.09

• MEH: running updates for QUB on older compute nodes to finally test the kernel and network improvements, includes putting nodes ipp100-104,118-122 into repair to force data onto ipp123-126 -- looks like ippc40, 056 showing same network stalling problem (ipps04,14 only to a minor level) so unless the network changed again, the change in early March didn't help it

Tuesday : 2018.04.10

• MEH: Gavin updating kernel on ippc23-c63 to 3.7.6 that the majority of nodes are at in order to prevent regular crashing -- c30-c63 will likely show stalling if reading from ipp123-126, but will test in detail when all updated (simple updates of any data product chip->warp clearly show issue if force some chip data to be put to ipp123-126 by setting ipp100-104,118-122 data targeted nodes into repair)
• MEH: mops needing access to more of the broken LAP.PV3 files -- reprocessing exposures o7023g0258o, o7023g0278o similarly as in PS1_IPP_Czarlog_20180319 -- this should preserve the broken LAP files so any code for file fixes can be tested in updating the original broken files (in chip)
  • another set also reprocessed -- o6390g0419o, o6392g0504o, o6364g0328o (in chip); o6938g0644o, o6364g0343o, o6938g0626o (in warp so error_cleaned state)

Wednesday : 2018.04.11

• MEH: nightly stdscience running diffs for mops during day
• MEH: turning down some of the general update test processing running on ippc23-c63, seeing some faults (unable to replicate files) in nightly processing --

Thursday : 2018.04.12

• MEH: more updates, diffs and publishing for MOPS during day
• MEH: continuing updates runs to test kernel update on ippc23-c63
• MEH: pstamp update -- adding option for MOPS to provide all images having pixels within the ROI (no stitching) and only by a very specific request type -- REQ_TYPE byexp, COORD_MASK 0 (sky coords) and OPTION_MASK include 8192 (last unused value?) -- r40389 and r40395

Friday : 2018.04.13

• MEH: running updates, diffs, stacks for QUB in nightly pantasks during day
• MEH: fixing more broken exposures from LAP.PV3 missing files with basic reprocessing (label/data_group mopsfix/mops.fixbrokenLAP.20180323) -- o7001g0316o,o7001g0295o,o6919g0700o,o6290g0138o,o6290g0119o
• MEH: MOPS behind large QUB pstamp request wanting same warps -- manually update chip+warps setting label to ps_ud_MOPS and remove QUB label from pstamp so dependent in limit query can be found and stamps finished for MOPS

Saturday : YYYY.MM.DD

Sunday : 2018.04.15

• MEH: fixing more broken exposures missing files from LAP.PV3 for mops with basic reprocessing -- o6993g0297o,o6993g0313o