PS1 IPP Czar Logs for the week 2016.10.24 - 2016.10.30

(Up to PS1 IPP Czar Logs)

Monday : 2016.10.24

  • 11:45 CZW: More HSC processing on the stare nodes.
  • 15:15 CZW: The ITC storage nodes (ipp105-ipp117) will be getting new Maui-visible IP addresses, and everyone is likely to need to edit their ssh keys to be able to connect. This is true even if those nodes haven't been logged into, as their IP addresses are ones recycled from the stsci nodes. I'll send an email when this happens.
  • 16:00 CZW: Restarting IPP pantasks.
  • 17:45 CZW: Restarting again, after making changes to the stdscience list that are no longer required as relastro is done.

Tuesday : 2016.10.25

  • MEH: possible NCU start night (run for 4 nights, window until 10/30+4 days) -- decision made at start-of-night, balancing of various processing outside of normal nightly will need to be manually done
    • x2+3 will be in use during nightly and will be needed during day for various processing, ipps nodes when available again after relphot finished
  • 18:30 CZW: Still running HSC data on stare00-04. Due to use elsewhere, I have not expanded beyond this.

Wednesday : 2016.10.26

Thursday : 2016.10.27

  • 09:xx MEH: restarted ~ipp/stdscience to adjust some NCU configs
  • MEH: NCU/MOPS processing to continue at least through Saturday -- upper ippx, ipps nodes still in use. normal nightly processing w/ OSS, SNIa continue to run independently in their default configurations. running in ~ippmops/stdscience, ~ippmops/stdscience_ws

Friday : 2016.10.28

  • MEH: apparently ippc47 (and c59) didn't have the /local/ipp/tmp link directory on /export/ippc47.1/ipp/tmp setup and has been regularly causing faults
  • 17:50 EAM : restarting pantaskses, replacing s3 & s4 with x0 and x1
    • MEH: will pause all extra NCU/MOPS processing prior to nightly ~1900 to avoid any issues w/ rebalance for nightly processing
  • 17:51 CZW: I didn't note it here, but I did in the neb-host note: ippb03 is back to "up" in nebulous, so there's a clear target for not-ITC, not-MRTCB replications if needed. It has the xattr=1 field set, so it will not be used for primary copies.

Saturday : 2016.10.29

  • MEH: additional NCU nights possible, NCU/MOPS processing to continue at least through Sunday -- upper ippx, ipps nodes still in use. normal nightly processing w/ OSS, SNIa continue to run independently in their default configurations. running in ~ippmops/stdscience, ~ippmops/stdscience_ws

Sunday : 2016.10.30

  • MEH: additional NCU night, NCU/MOPS processing to continue -- upper ippx, ipps nodes still in use. normal nightly processing w/ OSS, SNIa continue to run independently in their default configurations. running in ~ippmops/stdscience, ~ippmops/stdscience_ws