PS1 IPP Czar Logs for the week 2018-03-05 - 2018-03-11

(Up to PS1 IPP Czar Logs)

Monday : 2018-03-05

• 16:30 CZW: Reverted ippitc version of pswarp to r40343 to remove the PSREFCAT changes. Bug fix in place on the trunk (r40354), but without knowing if the test process correctly tests warps with non-zero quality, I'm not merging this change.

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : 2018.03.08

• 02:30 EAM : stopped slave for neb replication on ippdb06 in prep of setting up the rsync copies.
• 12:00 EAM : I started the rsync copy of ippdb06's mysql (nebulous). This will take a while (8-10h?), so I'll restart the mysql after it is complete.

Friday : 2018.03.09

• 15:10 EAM : update re: nebulous mysql backup:

  I have made an rsync copy of the nebulous database from ippdb06 (one of the current slaves) to ipp121.  I have restarted the mysql replication process on ippdb06 (it is already caught up). 
  
  I have also built mysql 5.6 on ipp121 and turned on replication on that machine. 
  
  I have also implemented a new strategy to have a backup copy to recover from catastrophic errors which are passed through the replication process (e.g., DROP TABLE...).  It turns out that mysql 5.6 includes a replication option to delay the replication process.  By issuing a command like
  
  CHANGE MASTER TO MASTER_DELAY = N;
  
  the replicant is made to lag the master by N seconds.  
  
  For the moment, I have set the ipp121 copy to have a delay of 50,000 seconds.  This seems to be working as this slave is currently at 50,000 seconds behind the master and holding.
  
  My plan, to be implemented next week, is to have two copies on some appropriate machines, one with a delay of 24 hours and a second with a delay of 1 week (or so, TBD).  That will give us two timescales for disaster recovery.  
  
  

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD