== PS1 IPP Czar Logs for the week 2019.02.25 - 2019.03.04 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2019.02.25 ===
 * 10:00 EAM : I have modified things to prevent spurious information related to disks on the machines used as backup-backup nodes (in case ATRC is offline) : ipp130, ipp134, ipp135:  
  * assigned these backup-backup volumes xattr = 4 (a new value with the meaning 'backup-backup')
  * updated nebdiskd to exclude xattr = 4 (as well as 5, already excluded)
  * installed the new version for user ipp@ipp117

JRF: There was an email warning about the load on ippdb08 getting high. EAM handled the problem and performed the following steps:
   - check on ganglia that load has been getting too high i.e. that the email is correct.
   - ssh to the machine, and log into the database using the nebulous login (the mysql database in this case is ippdb09). The reason for ssh to the machine is that you have to be local to see some of the processes.
Can view the processes with:
{{{ 
SHOW PROCESSLIST;
}}}
alternatively:
{{{
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST ORDER BY TIME DESC LIMIT 10;
}}}
Note that you need to have all privileges to see all these processes. In this case can you use the {{{nebulous}}} user, but beware that it has write privileges too!
   - the end column provides the process id for the job, go kill this to stop the request. It is best to kill this where it is running

  
=== Tuesday : 2019.02.26 ===

JRF: set ipp097 to repair due to battery warnings

=== Wednesday : 2019.02.27 ===

=== Thursday : 2019.02.28 ===

=== Friday : 2019.03.01 ===

JRF: Gavin has added the requested DNS changes of ops3-5, which will be the new machine names for ippc20-22 in time:
ippc20 -> ippops3
ippc21 -> ippops4
ippc22 -> ippops5
Either name will work right now for getting to the machine.
The addition of these DNS entries confused the ITC mail server (leavitt) and meant that the mailing lists were down from around 6am-8am. Adding the new name to the postfix settings solved the issues.

JRF: set ippb21 to down as it's producing a lot of hardware errors. Haydn will be replacing the disk.

=== Saturday : 2019.03.02 ===

=== Sunday : 2019.03.03 ===