PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2018.09.17

• MEH: appears to be red/faults on nightly/MOPS related processing left on ippMonitor from previous czar, these should be cleared and not left to later czars to cleanup or sort though which ones have been fixed and which ones not

Tuesday : 2018.09.18

• MEH: PS2/GPC2 dsmaster access to conductor is down, need to access conductor directly and at a different port (details vague since this wiki is not private...) -- update uri in gpc2:pzDataStore to reflect the change and also update tools/croot so tools/checkexp can be used
  • gpc2 use_compress 0 -- should remain off until Craig has GPC2 compression turned on
• MEH: some test bias exposures faulting due to gone? -- leave since LED and darks okay
• MEH: gpc2 nightly data all stalled at chip due to XY36 fault 4 -- manually setting quality 42 for XY36 so nightly processing can proceed... then adding a crontab ippps2@c18 to regularly set
  • to catchup from the backlog, manually adding datanode groups s4-7 to ps2:stdscience since no ps1 data

Wednesday : 2018.09.19

• MEH: ippb22 down again -- same error with EDAC -- accidentally cleared the display hitting the key s next to d (what a poor choice of key assignments) -- power cycled and back up
• MEH: ippx005 crash -- Curt & Co to look at tomorrow when working on other ippx nodes/PDU

Thursday : 2018.09.20

• MEH: plan for ippx nodes 057-068 and 081-088 to be powered down to move 081-088 to new cabinet and replace the older PDUs in H.04
  • instead moved ipp077-ippx084 to H.13 -- uses console switch in H.12 for PDU and serial console control and Haydn will need to add another console switch for H.13 in the future

Friday : 2018.09.21

• MEH: ippx005 was physically power cycled at ITC yesterday and been up and ok for past day with just nightly processing ok
• MEH: ipp032 crashed some 12 days ago and not rebooted -- power cycled and got on "Scanning for uhci-hcd...", power cycled again and seems to be back up ok
• MEH: Haydn replaced PDU for ippb09-b15 and all seem back up ok -- leaving neb-host down for day or so
  • ippb21,b22 network issue on power cycle

Saturday : 2018.09.22

• MEH: nightly processing a bit slow it seems, the ippx node balance must have been short last week -- adding the datanodes to help push through
  • added the PDU down nodes back in and properly balanced across all the xnodes so the MOPS nice'd processing doesn't get stalled on any sub-group of nodes, overpowered some to allow for downed nodes or for catchup and accommodate MOPS using both WW and WS diffs for PS2
    • ippx057 was out for cpu io wait issues -- back in and will need to watch
    • ippx037 was out due to no space on OS disk and put back in
• MEH: PS1 diff cleanup still not turned off -- set -1 in ~ippitc/psconfig/ipp-20170121.lin64/share/ippconfig/recipes/nightly_science.config
  • WS diffs also not turned off even though label not in stdscience -- hard coded to do rather than config based in nightly_science.pl... should manually trigger WS diffs to cleanup week by week since not having a problem and help reduce excess space used until this issue fixed for WWdiff inverse CMF being corrupted on update

Sunday : 2018.09.23

• MEH: another single instance file (on ippb09 neb-host down) needing a proper 2nd copy... ippb09 repair to get the data for MOPS and then replicate

  neb-replicate gpc1/LAP.PV3.20140730/2015/01/22/RINGS.V3/skycell.2092.082/RINGS.V3.skycell.2092.082.stk.4225628.unconv.mask.fits --volume ipp077.0
  

• MEH: setting ippb07-b15 neb-host up now, b16,17,19,20,23 repair before Olivia