== PS1 IPP Czar Logs for the week 2016.12.12 - 2016.12.18 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2016.12.12 === * MEH: large set QUB/MOPS warp updates/reprocessing continues and targeting ipp100+, running ~ippmops/stdscience, ~ippqub/stack using ipps and c2 (normally used for QUB WS and stacks) and ippx nodes -- if pantasks is stopped, then you must notify me directly as it can block MOPS+QUB stamps and/or interrupt top priority QUB processing * 17:50 CZW: Restarting IPP pantasks. I have not stopped or changed ippmops/stdscience or ippqueb/stack (double checked after seeing the message and having sent the check_system.sh stop). * 20:30 CZW: Something was stuck in summitcopy, but a reset.advance in the pantasks cleared the book and let it try those database twiddles again. === Tuesday : 2016.12.13 === * 16:00 CZW: Earlier I added the new available ATRC ippb nodes to nebulous. They are in repair as a precaution for today, and I'll probably switch them to be active tomorrow. === Wednesday : 2016.12.14 === === Thursday : 2016.12.15 === === Friday : 2016.12.16 === * 13:43 CZW: Setting ipp105 to down in nebulous in anticipation of potential network tests involving that host. * 17:23 CZW: Setting ipp105 to up in nebulous as it should now be visible again. It is not running on the vxlan, so it cannot connect to Maui. I will be running shuffle over the weekend to see if it gets a higher data rate in that configuration. * MEH: ippdb01 and ippc18 weren't reporting stats to ganglia again, really need to keep monitor on critical db01 nebulous machine -- /etc/init.d/gmond restart * MEH: restarting nightly pantasks since weather is clearing -- may need to make QUB config modification === Saturday : 2016.12.17 === * MEH: @noon -- shutting down apache on ippc20-c26 sequentially to free up space removing /tmp/nebulous_server.log and restarting * adding crontab on ippc18 like home disk to also check the apache nebulous machines and send email to ps-ipp-ops -- === Sunday : 2016.12.18 ===