PS1 IPP Czar Logs for the week 2019.02.18 - 2019.02.24

(Up to PS1 IPP Czar Logs)

Monday : 2019.02.18

• 11:30 EAM : stopped mysql on ipp121 & started rsync of mysql directory (/export/ipp121.0/mysql.nebulous.20180308 to /export/ipp121.1/mysql.nebulous.20190218/)
• TdB : There was 1 processing error on IPP monitor, which was due to missing detrend files. This was in turn due to both versions of the file being located on ippb06, which is without power at the ATRC. The exposure has been re-run using:

  pantasks_client
  add.label PSNSC.wfix.20180525
  exit
  
  chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o6980g0199o
  

  after running through, set the old runs to be cleaned to remove them from IPPmonitor

  chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXX
  warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
  

• TdB: ipp054-ipp066 should remain in repair until further notice.

Tuesday : 2019.02.19

• MEH: large warp update process continuing to run in ~ippmops/stdscience on the ippc31-c63, ipps00-14, ipp067-ipp123 nodes
  • ps_ud% chips and warps label changed to update.hold20190219 so not to be cleaned by accident

• TdB: John, Roger and myself went through the logfiles to figure out what went wrong yesterday with ipp113. We looked through the reports for ipp113 on Ganglia, noticing the spiking load after 18:00 until 20:00. We went into the serial console using:

  console --user xxxx -v ipp113
  

and check the rolling log information. We check the status (option 8) under power management (CTRL + P) and noticed it was back to normal now. Mark showed us how to powercycle the machine and check the boot messages in /var/log/messages (as sudo user)

When a machine has been power cycled, MYSQL is probably down. Check that by doing:

ps -ef | grep mysql

Check the config file under /etc/mysql/my.cnf for the location of the error log and see if MYSQL is what caused the issue. In this case, it did not.

To restart MYSQL do:

/etc/init.d/mysql start

After a reboot, the screen session also need to be checked and rebooted. As ippitc on ipp113 check them using:

screen -list

and restart screens using:

screen -S RoboCzar (or CzarPoll)

once in the screen session, launch the script using:

cd /data/ippc64.1/ippitc/src/ippMonitor/czartool
./roboczar.pl (or ./czarpoll.pl)

and get out using CTRL-a d

Finally, upon reboot, clear active processes from the pantasks by using:

./check_systems.sh stop

and set the pantasks back to run after jobs have cleared, using:

./check_systems.sh run

Remember, if there is any non-standard (i.e. not nightlyscience) processing happening, these are not affected by the check_systems script and need to be done by hand.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD