IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 13 and Version 14 of PS1_IPP_Czarlog_20190204


Ignore:
Timestamp:
Feb 6, 2019, 4:56:24 PM (7 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20190204

    v13 v14  
    8585=== Wednesday : 2019.02.06 ===
    8686
     87MEH: recent [ps-ipp-ops] email
     88 * PROBLEM alert - ippc75/Root Partition is CRITICAL -- nagios on 20190123 -- / nearly full, getting dangerous -- need to plan on resetting the nebulous_server.log (log rotation already in place)
     89  * ippc75 apache nebulous disk warning -- cronjob check on 20190130 -- 98% left, now dangerous since rotated logs can reach this size before rotation, need to deal with ASAP
     90  * RECOVERY alert - ippc75/Root Partition is OK -- nagios on 20190206 -- good
     91 * Fatal | Event occured on: ipp097.ifa.hawaii.edu -- 20190202 -- BBU failed, could consider putting neb-host repair particularly if many of the ipp067-122 nodes start filling up (close)
     92 * Fatal | Event occured on: ipp087.ifa.hawaii.edu -- 20190206 -- BBU failed, already in neb-host repair for this reason
     93 * FATAL  Event occurred on: ipp054 -- 20190205 -- BBU failed, already in neb-host repair, nothing to do and just reminder (applies for ipp054-066)
     94 * PROBLEM alert - ippdb05/Root Partition is WARNING -- 20190112 -- / nearly full and very dangerous, log rotate on syslog will help
     95  * RECOVERY alert - ippdb05/Root Partition is OK -- 20190206 -- good
     96 * MySQL replication problem on ippc17 -- 20190206 -- reminder replication for ippRequestServer (pstamp+datastore) DB on ipp113 doesn't have replication setup still
     97 * Cron <ippps2@ippc23> /bin/tcsh /data/ippc18.0/home/ippps2/cron_fix_fault_c23/ipp_ps2_rev4.bat --
     98 * PROBLEM alert - ipp112//dev/sda1 is WARNING -- nagios on 20190203 --
     99
    87100JRF: MEH, TdB and myself are going to archive the nebulous logs as they are getting very large, hence nagios complaining about lack of disk space on root. Log sizes:
    88101 * ippc71 - 20GB