PS1 IPP Czar Logs for the week 2016.12.12 - 2016.12.18

(Up to PS1 IPP Czar Logs)

Monday : 2016.12.12

  • MEH: large set QUB/MOPS warp updates/reprocessing continues and targeting ipp100+, running ~ippmops/stdscience, ~ippqub/stack using ipps and c2 (normally used for QUB WS and stacks) and ippx nodes -- if pantasks is stopped, then you must notify me directly as it can block MOPS+QUB stamps and/or interrupt top priority QUB processing
  • 17:50 CZW: Restarting IPP pantasks. I have not stopped or changed ippmops/stdscience or ippqueb/stack (double checked after seeing the message and having sent the check_system.sh stop).
  • 20:30 CZW: Something was stuck in summitcopy, but a reset.advance in the pantasks cleared the book and let it try those database twiddles again.

Tuesday : 2016.12.13

  • 16:00 CZW: Earlier I added the new available ATRC ippb nodes to nebulous. They are in repair as a precaution for today, and I'll probably switch them to be active tomorrow.

Wednesday : 2016.12.14

Thursday : 2016.12.15

Friday : 2016.12.16

  • 13:43 CZW: Setting ipp105 to down in nebulous in anticipation of potential network tests involving that host.
  • 17:23 CZW: Setting ipp105 to up in nebulous as it should now be visible again. It is not running on the vxlan, so it cannot connect to Maui. I will be running shuffle over the weekend to see if it gets a higher data rate in that configuration.
  • MEH: ippdb01 and ippc18 weren't reporting stats to ganglia again, really need to keep monitor on critical db01 nebulous machine -- /etc/init.d/gmond restart
  • MEH: restarting nightly pantasks since weather is clearing -- may need to make QUB config modification

Saturday : 2016.12.17

  • MEH: @noon -- shutting down apache on ippc20-c26 sequentially to free up space removing /tmp/nebulous_server.log and restarting -- done
    • adding crontab on ippc18 like home disk to also check the apache nebulous machines and send email to ps-ipp-ops --

Sunday : 2016.12.18

  • MEH: sending old ps_ud% and broken updated to cleanup, recent QUB not auto-cleaned as well