PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : YYYY.MM.DD

Wednesday : 2017.09.20

• MEH: neb-host up ipp100,102,103.0 since regularly not-red again; neb-host repair ipp101.0 since regularly red now
• MEH: uncommenting out ci1 nodes recently fixed by Haydn (ippc107,c111,c121,c127) in ippqub WSdiff processing for testing
  • ippc127 ganglia /etc/gmond.conf also needed changing to ippc19 (10.10.20.17) and restart

Thursday : 2017.09.21

• MEH: uncommenting out ippc107,c111,c121,c127 from ippitc/ippconfig/pantasks_hosts.input for nightly processing ci1 group (ippc125 could also be added in by Chris once his test processing finishes)
  • ippc107,c127 needed proper link /local/ipp setup

Friday : YYYY.MM.DD

Saturday : 2017.09.23

• 10:15 EAM : some problems with nebulous / neb-host and incorrect config. Mark sent the following:

  strangely it looked like ipp100.0,103.0,104.0 were not getting data last night even though they have been neb-host up for past few days (and ipp076 has been getting data while in repair) -- it appears the ~ippitc account was not updated for ippdb01->ippdb09 when switch was made
  
  /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/neb-host
  uses
  /data/ippc64.1/ippitc/ippconfig/nebulous.config
  has set
  NEB_DBHOST  STR    ippdb01.ifa.hawaii.edu
  
  don't know the full implications of this, so am not making any changes to anything, but suggest the config (as well as for ~ipp it seems) be returned to the dns one nebulous.ipp.ifa.hawaii.edu one
  

I explored and wrote back:

 I dug into things a bit further and I'm not completely certain of the correct solution, but I have at least a short term fix.

I believe ipp and ippitc only use that config value for neb-host.

the ipp user, which runs nebdiskd, uses the config values in /etc/nebdiskd.rc on the machine where nebdiskd is running (ipp117) -- that is pointing at ippdb09.

the apache nodes, which do all of the other interactions, are all pointing at ippdb09.

the fact that both ipp and ippitc were pointing at ippdb01 meant that the changes they made to the 'mountedvol' tables (using neb-host) were not being seen by the apache nodes.  Thus, the apache nodes continues to respect the previous settings.

Fortunately, the error did not affect the mysql replication process.

I tried to change the two config files to point at nebulous.ipp.ifa.hawaii.edu, but this did not work.  I'm not certain why this did not work.  I am able to get neb-host to work using ippdb09, but nebulous.ipp does not, even though it resolves to ippdb09 correctly.

(Also note that I can connect to mysql using the mysql shell with the machine name nebulous.ipp, so this problem is somewhere in the perl DBI layer or our use of it).

I have set both ~ipp and ~ippitc to point at ippdb09 for now.  

I have also confirmed that changes we make to ippdb09 with neb-host are propagated to ippdb01 via replication.

Sunday : YYYY.MM.DD