PS1 IPP Czar Logs for the week 2016.12.12 - 2016.12.18

(Up to PS1 IPP Czar Logs)

Monday : 2016.12.12

• MEH: large set QUB/MOPS warp updates/reprocessing continues and targeting ipp100+, running ~ippmops/stdscience, ~ippqub/stack using ipps and c2 (normally used for QUB WS and stacks) and ippx nodes -- if pantasks is stopped, then you must notify me directly as it can block MOPS+QUB stamps and/or interrupt top priority QUB processing
• 17:50 CZW: Restarting IPP pantasks. I have not stopped or changed ippmops/stdscience or ippqueb/stack (double checked after seeing the message and having sent the check_system.sh stop).
• 20:30 CZW: Something was stuck in summitcopy, but a reset.advance in the pantasks cleared the book and let it try those database twiddles again.

Tuesday : 2016.12.13

• 16:00 CZW: Earlier I added the new available ATRC ippb nodes to nebulous. They are in repair as a precaution for today, and I'll probably switch them to be active tomorrow.

Wednesday : 2016.12.14

Thursday : 2016.12.15

Friday : 2016.12.16

• 13:43 CZW: Setting ipp105 to down in nebulous in anticipation of potential network tests involving that host.
• 17:23 CZW: Setting ipp105 to up in nebulous as it should now be visible again. It is not running on the vxlan, so it cannot connect to Maui. I will be running shuffle over the weekend to see if it gets a higher data rate in that configuration.
• MEH: ippdb01 and ippc18 weren't reporting stats to ganglia again, really need to keep monitor on critical db01 nebulous machine -- /etc/init.d/gmond restart
• MEH: restarting nightly pantasks since weather is clearing -- may need to make QUB config modification

Saturday : 2016.12.17

Sunday : 2016.12.18