PS1 IPP Czar Logs for the week 2019.02.25 - 2019.03.04

(Up to PS1 IPP Czar Logs)

Monday : 2019.02.25

• 10:00 EAM : I have modified things to prevent spurious information related to disks on the machines used as backup-backup nodes (in case ATRC is offline) : ipp130, ipp134, ipp135:
  • assigned these backup-backup volumes xattr = 4 (a new value with the meaning 'backup-backup')
  • updated nebdiskd to exclude xattr = 4 (as well as 5, already excluded)
  • installed the new version for user ipp@ipp117

JRF: There was an email warning about the load on ippdb08 getting high. EAM handled the problem and performed the following steps:

• check on ganglia that load has been getting too high i.e. that the email is correct.
• ssh to the machine, and log into the database using the nebulous login (the mysql database in this case is ippdb09). The reason for ssh to the machine is that you have to be local to see some of the processes.

Can view the processes with:

SHOW PROCESSLIST;

alternatively:

SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST ORDER BY TIME DESC LIMIT 10;

Note that you need to have all privileges to see all these processes. In this case can you use the nebulous user, but beware that it has write privileges too!

• the end column provides the process id for the job, go kill this to stop the request. It is best to kill this where it is running

Tuesday : 2019.02.26

Wednesday : 2019.02.27

Thursday : 2019.02.28

Friday : 2019.03.01

Saturday : 2019.03.02

Sunday : 2019.03.03