[[PageOutline]]

[wiki:PS1_IPP_CzarLog_20120528#Wednesday:2012-05-30 Back to the related czar log wikipage (2012-05-30)]

[wiki:PS1_IPP_CzarLog_20120604 Back to the related czar log wikipage (2012-06-04)]

== Purpose ==

Just in case there is any problem and I can't do things myself in case of a moped accident. Moreover searching for "moped" should be a good way to find this wiki page in a couple of years when we will have to do this again...

== Preparation ==

 * '''DONE''' change apache configuration files on ippc01-ippc10
   I simply replaced the value of nebulous.ipp.ifa.hawaii.edu in /etc/apache2/modules.d/apache2-mod_perl-startup.pl by ippdb02.ipp.ifa.hawaii.edu. Note that it could have been possible to change the value in the dns (but Gene didn't want it). A copy of the original files is named /etc/apache2/modules.d/apache2-mod_perl-startup.pl.20120529_ippdb00

 * '''DONE''' (scripts are finished): check if jt scripts are still running on ippc19: make them asleep
 * '''DONE''' make sure ippdb02 is synchronized with ippdb00, stop slave on ippb02
 * '''DONE''' ''apache is connected to ippdb02 (check process table)'' restart apache servers on ippc01-ippc10
 * '''N/A''' wake up jt scripts: if crashed, tell jt
 * '''DONE''' kill nebdiskd on ippdb00
 * '''DONE''' start nebulous mysqldump on ippdb00 (use pbzip2)

   2012-05-29 19:00 The dump is running in screen session named nebulous_dump (i.e. reconnect on ippdb00 with {{{screen -r nebulous_dump}}}). The dump file is /export/ippdb00.0/nebulous_dumps/nebulous-20120529_fulldump.sql.bz2 and should have a size of 26G (according to the last successful dump of ippdb02 started this morning)

   2012-05-29 19:50 The dump should be finished around 0:30 (6 GB in 54 minutes / 27 GB for the complete dump).

   2012-05-29 21:50 The dumping rate has dropped down to about 10 GB in 3 hours... Should be finished around 3am?

   2012-05-30 06:20 Dump finished for info:
{{{
# stat nebulous-20120529_fulldump.sql.bz2
  File: `nebulous-20120529_fulldump.sql.bz2'
  Size: 27041967453     Blocks: 52816600   IO Block: 4096   regular file
Device: 804h/2052d      Inode: 1257        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-05-29 18:54:50.019218105 -1000
Modify: 2012-05-30 04:27:42.479312043 -1000
Change: 2012-05-30 04:27:42.479312043 -1000
}}}
== Once neb dump is finished ==
 * '''DONE''' copy reference dump to ippdb02 (and from ippdb02 to ipp001)

{{{
   /export/ippdb02.0/mysql-dumps/backups/20120529/nebulous-20120529_fulldump.sql.bz2
     Took 08:02 at 53.5MB/s (scp output)
   /export/ipp001.0/ipp/mysql-dumps/Archives/2012/nebulous-20120529_fulldump.sql.bz2
     Took 12:05 at 35.7MB/s (scp output)
}}}

 * '''DONE''' ''(2012-05-30 06:34)'' stop the mysql server

 * '''DONE''' delete nebulous database on ippdb00

For info in {{{/var/lib/mysql/nebulous}}}:
{{{
 # ls -la
total 1143656632
drwx------ 2 mysql mysql         4096 Apr 25 13:42 .
drwxr-x--- 6 mysql root         28672 May 29 08:15 ..
-rw-rw---- 1 mysql mysql         8632 Sep 16  2011 cabinet.frm
-rw-rw---- 1 mysql mysql       131072 Nov 16  2011 cabinet.ibd
-rw-rw---- 1 mysql mysql           65 Nov 23  2009 db.opt
-rw-rw---- 1 mysql mysql         8632 Sep 16  2011 deleted.frm
-rw-rw---- 1 mysql mysql    348127232 May 24 14:43 deleted.ibd
-rw-rw---- 1 mysql mysql         8640 Sep 16  2011 directory.frm
-rw-rw---- 1 mysql mysql  22993174528 May 29 17:19 directory.ibd
-rw-rw---- 1 mysql mysql         8722 Mar  2  2011 instance.frm
-rw-rw---- 1 mysql mysql 558735818752 May 29 18:00 instance.ibd
-rw-rw---- 1 mysql mysql         8637 Sep 16  2011 lock_record.frm
-rw-rw---- 1 mysql mysql       114688 Sep 16  2011 lock_record.ibd
-rw-rw---- 1 mysql mysql         8704 Sep 16  2011 log.frm
-rw-rw---- 1 mysql mysql        98304 Sep 16  2011 log.ibd
-rw-rw---- 1 mysql mysql         8748 Apr 25 13:42 lost_instances.frm
-rw-rw---- 1 mysql mysql       688128 Apr 25 20:33 lost_instances.ibd
-rw-rw---- 1 mysql mysql         8898 Sep 16  2011 mountedvol.frm
-rw-rw---- 1 mysql mysql       229376 May 29 18:43 mountedvol.ibd
-rw-rw---- 1 mysql mysql         8723 Mar  3  2011 storage_object.frm
-rw-rw---- 1 mysql mysql 493929627648 May 29 18:00 storage_object.ibd
-rw-rw---- 1 mysql mysql         8716 Sep 16  2011 storage_object_attr.frm
-rw-rw---- 1 mysql mysql  50675580928 May 29 18:00 storage_object_attr.ibd
-rw-rw---- 1 mysql mysql         8624 Sep 16  2011 storage_object_xattr.frm
-rw-rw---- 1 mysql mysql  44417679360 May 29 17:30 storage_object_xattr.ibd
-rw-rw---- 1 mysql mysql         8870 Sep 16  2011 volume.frm
-rw-rw---- 1 mysql mysql       212992 May 24 14:42 volume.ibd
# du
1143656584      .
}}}
 * '''DONE''' clean mysql on ippdb00 (remove old binlogs)

 * '''DONE''' start the mysql server

 * '''RUNNING''' ingest neb dump on ippdb00

{{{
    Running in screen session {{{nebulous_dump}}}
    Started: Wed May 30 06:45:36 HST 2012
    Finished: Crashed
}}}

== SINCE ippdb00 is not finished ==
 * '''DONE''' We will use ippdb02 as a backup. I flushed the logs on ippdb02. The binlogs are now: mysqld-bin.000680 (and mysqld-relay-bin.003160)
 * Ingestion crashed see [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_CzarLog_20120528#Thursday:2012-05-31 czar log]
 * '''DONE''' Ingestion restarted with slow log queries disabled
{{{
    Running in screen session 'ingest' (root account)
    Started: May 31 15:14:39
    Finished: Jun 05 00:02:42
}}}

 * Some ingestion statistics:
   * Directory contents
{{{
ippdb00 ~ # ls -l /var/lib/mysql/nebulous/
total 932014828
-rw-rw---- 1 mysql mysql         8632 May 31 15:14 cabinet.frm
-rw-rw---- 1 mysql mysql       131072 May 31 15:14 cabinet.ibd
-rw-rw---- 1 mysql mysql         8632 May 31 15:14 deleted.frm
-rw-rw---- 1 mysql mysql    289406976 May 31 15:15 deleted.ibd
-rw-rw---- 1 mysql mysql         8640 May 31 15:14 directory.frm
-rw-rw---- 1 mysql mysql  21583888384 May 31 15:32 directory.ibd
-rw-rw---- 1 mysql mysql         8722 May 31 15:30 instance.frm
-rw-rw---- 1 mysql mysql 452494098432 Jun  5 03:10 instance.ibd
-rw-rw---- 1 mysql mysql         8637 Jun  4 05:26 lock_record.frm
-rw-rw---- 1 mysql mysql       114688 Jun  4 05:32 lock_record.ibd
-rw-rw---- 1 mysql mysql         8704 Jun  4 05:26 log.frm
-rw-rw---- 1 mysql mysql        98304 Jun  4 05:32 log.ibd
-rw-rw---- 1 mysql mysql         8748 Jun  4 05:26 lost_instances.frm
-rw-rw---- 1 mysql mysql       688128 Jun  4 05:32 lost_instances.ibd
-rw-rw---- 1 mysql mysql         8898 Jun  4 05:26 mountedvol.frm
-rw-rw---- 1 mysql mysql       229376 Jun  4 05:32 mountedvol.ibd
-rw-rw---- 1 mysql mysql         8723 Jun  4 05:26 storage_object.frm
-rw-rw---- 1 mysql mysql 412044230656 Jun  5 03:08 storage_object.ibd
-rw-rw---- 1 mysql mysql         8716 Jun  4 19:18 storage_object_attr.frm
-rw-rw---- 1 mysql mysql  37694210048 Jun  4 22:33 storage_object_attr.ibd
-rw-rw---- 1 mysql mysql         8624 Jun  4 22:32 storage_object_xattr.frm
-rw-rw---- 1 mysql mysql  30274486272 Jun  5 00:02 storage_object_xattr.ibd
-rw-rw---- 1 mysql mysql         8870 Jun  4 23:59 volume.frm
-rw-rw---- 1 mysql mysql       212992 Jun  5 00:02 volume.ibd
}}}
   * Tables contents:
{{{
directory: 16895977 entries in 911 seconds (= "2012-05-31 15:14:56" - "2012-05-31 15:30:07")
instance: 1216215431 entries in 309374 seconds (= "2012-05-31 15:30:07" - "2012-06-04 05:26:21")
storage_object: 944739798 entries in 49940 seconds (= "2012-06-04 05:26:22" - "2012-06-04 19:18:42")
storage_object_attr: 944739798 entries in 11623 seconds (= "2012-06-04 19:18:42" - "2012-06-04 22:32:25")
storage_object_xattr: 274638705 entries in 5250 seconds (= "2012-06-04 22:32:25" - "2012-06-04 23:59:55")
}}}
   * The full ingestion created 450 binlogs (see /export/ippdb00.0/mysql_recovery/ippdb00_after_recovery_20120605 for the history).
   * {{{/var/lib/mysql}}} contents:
{{{
ippdb00 mysql # du
2210620	./log
1688	./test
784	./mysql
932014832	./nebulous
1416110816	.
}}}
   * TODO: Add statistics about the ingestion rate, size gain
== Once ippdb00 ingestion is complete ==

 * stop '''DONE''' pantasks, '''DONE''' pstamp apache server, '''DONE''' apache servers
 * '''DONE''' flush the logs on ippdb02 after making sure noone is connected to ippdb02 mysql server
 * '''DONE''' copy binlogs from ippdb02 to ippdb00 (copied to {{{/export/ippdb00.0/mysql_recovery/ippdb02_binlogs_20120605}}})
 * '''DONE''' enable mysql slow logs
 * '''DONE''' restart the mysql server on ippdb00.
 * play binlogs taken from ippdb02 on ippdb00
 * '''DONE''' compress all possible logs on ippdb00: I compressed the mysql slowlog. I didn't find the nebdiskd log though...
 * ask gavin to change dns so that nebulous.ipp.ifa.hawaii.edu points to ippdb00 again
 * write down new master coordinates
 * restart nebdiskd
 * '''DONE''' restore apache configuration files on ippc01-ippc10 (backup files are called /etc/apache2/modules.d/apache2-mod_perl-startup.pl.20120529_ippdb00)
 * restart apache servers (jt scripts should be finished?!)
 * restart ipp
 * delete neb dump on ippdb00

== Once everything seems fine, *** on ippb02 *** ==
 * comment neb dump in root crontab
 * delete nebulous database
 * stop the mysql server
 * clean mysql (remove old binlogs, move and compress mysql logs)
 * start the mysql server
 * ingest neb dump
 * set master coordinates
 * start slave
 * activate neb dump in root crontab
 * delete neb dump