Back to the related czar log wikipage

Purpose

Preparation

Just in case there is any problem and I can't do things myself in case of a moped accident. Moreover searching for "moped" should be a good way to find this wiki page in a couple of years when we will have to do this again...

• DONE change apache configuration files on ippc01-ippc10 I simply replaced the value of nebulous.ipp.ifa.hawaii.edu in /etc/apache2/modules.d/apache2-mod_perl-startup.pl by ippdb02.ipp.ifa.hawaii.edu. Note that it could have been possible to change the value in the dns (but Gene didn't want it). A copy of the original files is named /etc/apache2/modules.d/apache2-mod_perl-startup.pl.20120529_ippdb00

• DONE (scripts are finished): check if jt scripts are still running on ippc19: make them asleep
• DONE make sure ippdb02 is synchronized with ippdb00, stop slave on ippb02
• DONE apache is connected to ippdb02 (check process table) restart apache servers on ippc01-ippc10
• N/A wake up jt scripts: if crashed, tell jt
• DONE kill nebdiskd on ippdb00
• DONE start nebulous mysqldump on ippdb00 (use pbzip2)

2012-05-29 19:00 The dump is running in screen session named nebulous_dump (i.e. reconnect on ippdb00 with screen -r nebulous_dump). The dump file is /export/ippdb00.0/nebulous_dumps/nebulous-20120529_fulldump.sql.bz2 and should have a size of 26G (according to the last successful dump of ippdb02 started this morning)

2012-05-29 19:50 The dump should be finished around 0:30 (6 GB in 54 minutes / 27 GB for the complete dump).

2012-05-29 21:50 The dumping rate has dropped down to about 10 GB in 3 hours... Should be finished around 3am?

2012-05-30 06:20 Dump finished for info:

# stat nebulous-20120529_fulldump.sql.bz2
  File: `nebulous-20120529_fulldump.sql.bz2'
  Size: 27041967453     Blocks: 52816600   IO Block: 4096   regular file
Device: 804h/2052d      Inode: 1257        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-05-29 18:54:50.019218105 -1000
Modify: 2012-05-30 04:27:42.479312043 -1000
Change: 2012-05-30 04:27:42.479312043 -1000

Once neb dump is finished

• DONE copy reference dump to ippdb02 (and from ippdb02 to ipp001)

   /export/ippdb02.0/mysql-dumps/backups/20120529/nebulous-20120529_fulldump.sql.bz2
     Took 08:02 at 53.5MB/s (scp output)
   /export/ipp001.0/ipp/mysql-dumps/Archives/2012/nebulous-20120529_fulldump.sql.bz2
     Took 12:05 at 35.7MB/s (scp output)

• DONE (2012-05-30 06:34) stop the mysql server

• DONE delete nebulous database on ippdb00

For info in /var/lib/mysql/nebulous:

 # ls -la
total 1143656632
drwx------ 2 mysql mysql         4096 Apr 25 13:42 .
drwxr-x--- 6 mysql root         28672 May 29 08:15 ..
-rw-rw---- 1 mysql mysql         8632 Sep 16  2011 cabinet.frm
-rw-rw---- 1 mysql mysql       131072 Nov 16  2011 cabinet.ibd
-rw-rw---- 1 mysql mysql           65 Nov 23  2009 db.opt
-rw-rw---- 1 mysql mysql         8632 Sep 16  2011 deleted.frm
-rw-rw---- 1 mysql mysql    348127232 May 24 14:43 deleted.ibd
-rw-rw---- 1 mysql mysql         8640 Sep 16  2011 directory.frm
-rw-rw---- 1 mysql mysql  22993174528 May 29 17:19 directory.ibd
-rw-rw---- 1 mysql mysql         8722 Mar  2  2011 instance.frm
-rw-rw---- 1 mysql mysql 558735818752 May 29 18:00 instance.ibd
-rw-rw---- 1 mysql mysql         8637 Sep 16  2011 lock_record.frm
-rw-rw---- 1 mysql mysql       114688 Sep 16  2011 lock_record.ibd
-rw-rw---- 1 mysql mysql         8704 Sep 16  2011 log.frm
-rw-rw---- 1 mysql mysql        98304 Sep 16  2011 log.ibd
-rw-rw---- 1 mysql mysql         8748 Apr 25 13:42 lost_instances.frm
-rw-rw---- 1 mysql mysql       688128 Apr 25 20:33 lost_instances.ibd
-rw-rw---- 1 mysql mysql         8898 Sep 16  2011 mountedvol.frm
-rw-rw---- 1 mysql mysql       229376 May 29 18:43 mountedvol.ibd
-rw-rw---- 1 mysql mysql         8723 Mar  3  2011 storage_object.frm
-rw-rw---- 1 mysql mysql 493929627648 May 29 18:00 storage_object.ibd
-rw-rw---- 1 mysql mysql         8716 Sep 16  2011 storage_object_attr.frm
-rw-rw---- 1 mysql mysql  50675580928 May 29 18:00 storage_object_attr.ibd
-rw-rw---- 1 mysql mysql         8624 Sep 16  2011 storage_object_xattr.frm
-rw-rw---- 1 mysql mysql  44417679360 May 29 17:30 storage_object_xattr.ibd
-rw-rw---- 1 mysql mysql         8870 Sep 16  2011 volume.frm
-rw-rw---- 1 mysql mysql       212992 May 24 14:42 volume.ibd
# du
1143656584      .

• PARTIALLY DONE clean mysql on ippdb00 (remove old binlogs, move and compress mysql logs)

TODO: Compress logs in /var/log/mysql

• DONE start the mysql server

• RUNNING ingest neb dump on ippdb00

    Running in screen session {{{nebulous_dump}}}
    Started: Wed May 30 06:45:36 HST 2012
    Finished: Crashed

SINCE ippdb00 is not finished

• We will use ippdb02 as a backup. I flushed the logs on ippdb02. The binlogs are now: mysqld-bin.000680 (and mysqld-relay-bin.003160)
• Ingestion crashed see ​czar log
• Ingestion restarted with slow log queries disabled

      Running in screen session 'ingest' (root account)
      Started: May 31 15:14:39
      Finished: 
  

Once ippdb00 ingestion is complete

• stop pantasks
• ask gavin to change dns so that nebulous.ipp.ifa.hawaii.edu points to ippdb00 again
• play binlogs taken from ippdb02 on ippdb00
• write down new master coordinates
• move and compress nebdiskd log
• restart nebdiskd

Once ingestion is finished

• restore apache configuration files on ippc01-ippc10 (backup files are called /etc/apache2/modules.d/apache2-mod_perl-startup.pl.20120529_ippdb00)
• restart apache servers (jt scripts should be finished?!)
• restart ipp
• delete neb dump on ippdb00

Once everything seems fine, * on ippb02 *

• comment neb dump in root crontab
• delete nebulous database
• stop the mysql server
• clean mysql (remove old binlogs, move and compress mysql logs)
• start the mysql server
• ingest neb dump
• set master coordinates
• start slave
• activate neb dump in root crontab
• delete neb dump