PS1_IPP_Czarlog_20160404 – Pan-STARRS IPP

Context Navigation

Version 13 (modified by Mark Huber, 10 years ago) ( diff )
--

MEH: gpc1 hasn't had a proper dump since ~2/4 when ippdb03 had OS disk raid issues and had to be swapped with ippc15 and was not checked...
- /var/spool/cron/crontabs group stb-admin not crontab.. fixed so now ipp has crontab access again on ippdb03...
- re-creating crontab best can to follow info on DatabaseBackups as best possible..

MEH: ippMonitor has been reporting problems (red) with ippdb03 (gpc1 replication) and ippdb06 (nebulous replication) for some time now with no one looking into -- this dates back to the swap of ippMonitor to another machine while ippdb03 being remade -- there are all kinds of inconsistencies in host names, passwords in ippMonitor AND mysql setups... tweaked to get ippMonitor to report on those so we can be alerted to problems now.. but cannot easily clean up inconsistencies as who knows what all rely on them as things are not documented..

15:45 MEH: Haydn replacing BBU in ipp085 -- all pantasks stop for now
17:40 MEH: restarting nightly pantasks with data targeting adjustment off the nodes with less free space
- adding ipp100,102,103,104.0+1 as targets and also adding manually to summitcopy+registration for testing for tonight with disk space imbalance

MEH: home disk <35GB again, ~ipp registration and stdscience already >3GB of logs and should've been bzip'd for the previous month...
- replication/logs is wasting 17GB of space and needs to be bzip'd
MEH: adding ipp100-104 into hosts_s6 group -- similar to s5 group machines but those are used for pv3 right now -- could use s6 for boosting summitcopy+registration while s5 removed for pv3 work
15:20 MEH: sending ps_ud% labels to cleanup now that STScI reports having download complete --
MEH: restarting nightly pantasks and manually swapping in ipphosts.mhpcc.config.20160407avoidfull for data targets to avoid full nodes w/ warps+2x diffs (replaced with ipp100-104 since pixel products cleaned up)