Just in case there is any problem and I can't do things myself in case of a moped accident. Moreover searching for "moped" should be a good way to find this wiki page in a couple of years...

 * '''DONE''' change apache configuration files on ippc01-ippc10

== At 8pm ==
 * '''DONE''' (scripts are finished): check if jt scripts are still running on ippc19: make them asleep
 * '''DONE''' make sure ippdb02 is synchronized with ippdb00, stop slave on ippb02
 * '''DONE''' ''apache is connected to ippdb02 (check process table)'' restart apache servers on ippc01-ippc10
 * '''N/A''' wake up jt scripts: if crashed, tell jt
 * '''DONE''' kill nebdiskd on ippdb00
 * '''DONE''' start nebulous mysqldump on ippdb00 (use pbzip2)

   2012-05-29 19:00 The dump is running in screen session named nebulous_dump (i.e. reconnect on ippdb00 with {{{screen -r nebulous_dump}}}). The dump file is /export/ippdb00.0/nebulous_dumps/nebulous-20120529_fulldump.sql.bz2 and should have a size of 26G (according to the last successful dump of ippdb02 started this morning)

   2012-05-29 19:50 The dump should be finished around 0:30 (6 GB in 54 minutes / 27 GB for the complete dump).

   2012-05-29 21:50 The dumping rate has dropped down to about 10 GB in 3 hours... Should be finished around 3am?

   2012-05-30 06:20 Dump finished for info:
{{{
# stat nebulous-20120529_fulldump.sql.bz2
  File: `nebulous-20120529_fulldump.sql.bz2'
  Size: 27041967453     Blocks: 52816600   IO Block: 4096   regular file
Device: 804h/2052d      Inode: 1257        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-05-29 18:54:50.019218105 -1000
Modify: 2012-05-30 04:27:42.479312043 -1000
Change: 2012-05-30 04:27:42.479312043 -1000
}}}
== Once neb dump is finished ==
 * '''RUNNING''' copy reference dump to ippdb02 (and from ippdb02 to ipp001)

{{{
   /export/ippdb02.0/mysql-dumps/backups/20120529/nebulous-20120529_fulldump.sql.bz2
   /export/ipp001.0/ipp/mysql-dumps/Archives/2012/nebulous-20120529_fulldump.sql.bz2
}}}

 * '''DONE''' ''(2012-05-30 06:34)'' stop the mysql server

 * '''DONE''' delete nebulous database on ippdb00

For info in {{{/var/lib/mysql/nebulous}}}:
{{{
 # ls -la
total 1143656632
drwx------ 2 mysql mysql         4096 Apr 25 13:42 .
drwxr-x--- 6 mysql root         28672 May 29 08:15 ..
-rw-rw---- 1 mysql mysql         8632 Sep 16  2011 cabinet.frm
-rw-rw---- 1 mysql mysql       131072 Nov 16  2011 cabinet.ibd
-rw-rw---- 1 mysql mysql           65 Nov 23  2009 db.opt
-rw-rw---- 1 mysql mysql         8632 Sep 16  2011 deleted.frm
-rw-rw---- 1 mysql mysql    348127232 May 24 14:43 deleted.ibd
-rw-rw---- 1 mysql mysql         8640 Sep 16  2011 directory.frm
-rw-rw---- 1 mysql mysql  22993174528 May 29 17:19 directory.ibd
-rw-rw---- 1 mysql mysql         8722 Mar  2  2011 instance.frm
-rw-rw---- 1 mysql mysql 558735818752 May 29 18:00 instance.ibd
-rw-rw---- 1 mysql mysql         8637 Sep 16  2011 lock_record.frm
-rw-rw---- 1 mysql mysql       114688 Sep 16  2011 lock_record.ibd
-rw-rw---- 1 mysql mysql         8704 Sep 16  2011 log.frm
-rw-rw---- 1 mysql mysql        98304 Sep 16  2011 log.ibd
-rw-rw---- 1 mysql mysql         8748 Apr 25 13:42 lost_instances.frm
-rw-rw---- 1 mysql mysql       688128 Apr 25 20:33 lost_instances.ibd
-rw-rw---- 1 mysql mysql         8898 Sep 16  2011 mountedvol.frm
-rw-rw---- 1 mysql mysql       229376 May 29 18:43 mountedvol.ibd
-rw-rw---- 1 mysql mysql         8723 Mar  3  2011 storage_object.frm
-rw-rw---- 1 mysql mysql 493929627648 May 29 18:00 storage_object.ibd
-rw-rw---- 1 mysql mysql         8716 Sep 16  2011 storage_object_attr.frm
-rw-rw---- 1 mysql mysql  50675580928 May 29 18:00 storage_object_attr.ibd
-rw-rw---- 1 mysql mysql         8624 Sep 16  2011 storage_object_xattr.frm
-rw-rw---- 1 mysql mysql  44417679360 May 29 17:30 storage_object_xattr.ibd
-rw-rw---- 1 mysql mysql         8870 Sep 16  2011 volume.frm
-rw-rw---- 1 mysql mysql       212992 May 24 14:42 volume.ibd
# du
1143656584      .
}}}
 * '''PARTIALLY DONE''' clean mysql on ippdb00 (remove old binlogs, move and compress mysql logs)

   TODO: Compress logs in /var/log/mysql

== BLAH ==

 * start the mysql server
 * ingest neb dump on ippdb00
 * write down new master coordinates
 * move and compress nebdiskd log
 * restart nebdiskd

== Once ingestion is finished ==
 * restore apache configuration files on ippc01-ippc10 (backup files are called /etc/apache2/modules.d/apache2-mod_perl-startup.pl.20120529_ippdb00)
 * restart apache servers (jt scripts should be finished?!)
 * restart ipp
 * delete neb dump on ippdb00

== Once everything seems fine, *** on ippb02 *** ==
 * comment neb dump in root crontab
 * delete nebulous database
 * stop the mysql server
 * clean mysql (remove old binlogs, move and compress mysql logs)
 * start the mysql server
 * ingest neb dump
 * set master coordinates
 * start slave
 * activate neb dump in root crontab
 * delete neb dump