PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2018.06.04

Tuesday : 2018.06.05

• MEH: continued working on clearing broken file issues for MOPS getting warp stamps -- make no changes to system without discussing with me first

Wednesday : 2018.06.06

• MEH: continued working on clearing backlog of hundreds of broken file issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • set pstampDependent broken new state to stop in preparation for updated MOPS pstamp requests with some exposures removed and new requests still not done (will likely trigger more broken file faults...)

Thursday : 2018.06.07

• MEH: still dealing with mess of broken files blocking MOPS from getting time-sensitive stamps -- make no changes to system without discussing with me first
  • cleared ~90 broken file issues with reprocessing and ~20 missing burn.tbl and other issues
  • change ps_ud_MOPS.2 label to ps_ud_MOPS.2.hold180607 and set pstampDependent broken new state to stop in preparation for next MOPS pstamp request set
  • remove pstamp from daily auto-restart to avoid interruption of large queue requests that takes hours to parse --

Friday : 2018.06.08

• MEH: still dealing with mess of broken files blocking MOPS from getting time-sensitive stamps -- make no changes to system without discussing with me first

Saturday : 2018.06.09

• MEH: still dealing with mess of broken files blocking MOPS from getting time-sensitive stamps -- make no changes to system without discussing with me first

Sunday : 2018.06.10

• MEH: ippx037 is causing PS2 processing to fault 3 in warp stage, removing from processing manually until can look into -- OS disk has no space... -- log rotation really needs to be implemented for all the compute nodes at the very least -- cleared mysqld.slow.log also for ippx022,023,025,038,057 since <2GB space available and used for PS2 nightly processing

  -rw------- 1 root    13G Jun 10 07:16 messages
  -rw-rw---- 1 mysql 5.5G Feb 18 20:39 mysqld.slow.log
  

• MEH: still dealing with mess of broken files blocking MOPS from getting time-sensitive stamps -- make no changes to system without discussing with me first