PS1 IPP Czar Logs for the week 2019.02.18 - 2019.02.24

(Up to PS1 IPP Czar Logs)

Monday : 2019.02.18

• 11:30 EAM : stopped mysql on ipp121 & started rsync of mysql directory (/export/ipp121.0/mysql.nebulous.20180308 to /export/ipp121.1/mysql.nebulous.20190218/)
• TdB : There was 1 processing error on IPP monitor, which was due to missing detrend files. This was in turn due to both versions of the file being located on ippb06, which is without power at the ATRC. The exposure has been re-run using:

  pantasks_client
  add.label PSNSC.wfix.20180525
  exit
  
  chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o6980g0199o
  

  after running through, set the old runs to be cleaned to remove them from IPPmonitor

  chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXX
  warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
  

• TdB: ipp054-ipp066 should remain in repair until further notice.

Tuesday : 2019.02.19

• MEH: large warp update process continuing to run in ~ippmops/stdscience on the ippc31-c63, ipps00-14, ipp067-ipp123 nodes
  • ps_ud% chips and warps label changed to update.hold20190219 so not to be cleaned by accident

Wednesday : 2019.02.20

• TdB: John, Roger and myself went through the logfiles to figure out what went wrong yesterday with ipp113. We looked through the reports for ipp113 on Ganglia, noticing the spiking load after 18:00 until 20:00. We went into the serial console using:

  console --user xxxx -v ipp113
  

and check the rolling log information. We check the status (option 8) under power management (CTRL + P) and noticed it was back to normal now. Mark showed us how to powercycle the machine and check the boot messages in /var/log/messages (as sudo user)

When a machine has been power cycled, MYSQL is probably down. Check that by doing:

ps -ef | grep mysql

Check the config file under /etc/mysql/my.cnf for the location of the error log and see if MYSQL is what caused the issue. In this case, it did not.

To restart MYSQL do:

/etc/init.d/mysql start

After a reboot, the screen session also need to be checked and rebooted. As ippitc on ipp113 check them using:

screen -list

and restart screens using:

screen -S RoboCzar (or CzarPoll)

once in the screen session, launch the script using:

cd /data/ippc64.1/ippitc/src/ippMonitor/czartool
./roboczar.pl (or ./czarpoll.pl)

and get out using CTRL-a d

Finally, upon reboot, clear active processes from the pantasks by using:

./check_systems.sh stop

and set the pantasks back to run after jobs have cleared, using:

./check_systems.sh run

Remember, if there is any non-standard (i.e. not nightlyscience) processing happening, these are not affected by the check_systems script and need to be done by hand.

• TdB : There were a few processing errors on IPP monitor, due to missing files.These exposures have been re-run using the standard wfix set-up:

  pantasks_client
  add.label PSNSC.wfix.20180525
  exit
  
  chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o6980g0199o
  

Thursday : 2019.02.21

• TdB: ipp125 experienced stalling after high load and high wait yesterday night. messages in /var/log/messages showed some issues related to the swapper and the raid. Conferring with Haydn, the raid looks fine and a reboot is warranted. We went into the pantasks_client in ippqub:stdscience_ws to stop all the current processing:

  pantasks_client
  
  stop
  

  To check that all current processing was properly synced, we did:

  sudo sync
  

  This showed nothing, indicating everything is synced correctly. Following this, we logged in to the machine using the serial console and did a regular reboot using:

  sudo reboot -f
  

  boot messages indicate the Superblock last mount time was in the future and the clock was not set. Haydn confirmed the system clock does not automatically set on this motherboard/kernel combination, and set the clock manually using:

   ntpdate -u galileo
  

  which fixes the clock issue. The machine has subsequently been set back to repair to monitor its activity and load levels, and finally put back to up.

• TdB: several machines need a kernel update, among which ippx052. We removed it for current processing by going into the ps2 stdscience pantasks_client at ~ippps2/stdscience and doing:

  controller status
  controller host off ippx052 (6 times)
  controller host delect ippx052 (6 times)
  

  After this, I let Gavin know the machine is ready for upgrade.

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD