IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 54 and Version 55 of Replication_Issues


Ignore:
Timestamp:
May 30, 2012, 10:25:30 AM (14 years ago)
Author:
Serge CHASTEL
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Replication_Issues

    v54 v55  
    44[[PageOutline]]
    55
    6 = 20120529-20120529 ippdb00/ippdb02 =
     6= 2012 =
     7== 20120529-20120529 ippdb00/ippdb02 ==
    78
    89Full nebulous ingestion. Details [wiki:Moped_accident here]
    910
    10 = 20120427/11:05 ipp001 =
     11== 20120427/11:05 ipp001 ==
    1112ipp001 replication broken because of {{{'Duplicate entry '32' for key 1' on query. Default database: 'ippadmin'. Query: 'insert into users (username, password) values ('chenx', 'chenx')'}}}. I ran the usual {{{STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;}}}. Then {{{'Duplicate entry '30' for key 1' on query. Default database: 'ippadmin'. Query: 'INSERT into cookies (username, cookie) values ('chenx', '32709')'}}}. Another {{{STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;}}} made it.
    1213
    13 = 20120313/12:05 ipp001 =
     14== 20120313/12:05 ipp001 ==
    1415ipp001 replication is broken again (broken foreign keys?! e.g. {{{Cannot add or update a child row: a foreign key constraint fails (`gpc1/chipProcessedImfile`, CONSTRAINT `chipProcessedImfile_ibfk_1` FOREIGN KEY (`chip_id`, `exp_id`) REFERENCES `chipRun`}}}). Made manual backup of ippdb03 to /export/ippdb03.0/mysql-dumps/backup_20120313T1208.sql.bz2
    1516{{{
    … …  
    1819}}}
    1920
    20 = 20120312/13:31 ippdb03/ipp001 =
     21== 20120312/13:31 ippdb03/ipp001 ==
    2122{{{Error 'Table 'skycells.skycell' doesn't exist' on query. Default database: 'gpc1'. Query: 'update staticskyRun join staticskyInput using(sky_id) join stackRun using(stack_id) join skycells.skycell using(skycell_id) set staticskyRun.state = 'new.wait' where staticskyRun.label ='lap.threepi.20110809' and staticskyRun.state ='new' and ra/15 > 12'}}}
    2223
    … …  
    2728It looks though that there are other problems (multiple insert attempts for the same pk) for ipp001.
    2829
    29 = 20120223/08:40: ippdb03 =
     30== 20120223/08:40: ippdb03 ==
    3031{{{Duplicate entry 'XXXXX-X' for key 1' on query. Default database: 'ipptopsps'}}}. I ran {{{~schastel/dev/many_failures/fix_replication.sh}}} about 200 times without success. I talked to Roy who will replicate ipptopsps somewhere else. I'm dropping it from replication for now.
    3132I stopped the mysql server on ippdb03, modified /etc/mysql/my.cnf (commented out ipptopsps replication), and restarted the server.
    … …  
    3839}}}
    3940Then "SHOW SLAVE STATUS\G" commands showed that replication worked again (Seconds_Behind_Master is decreasing).
    40 = 20120203/08:51: ippc19  =
     41
     42== 20120203/08:51: ippc19 ==
    4143
    4244ippc17 experienced crashes yesterday and tables were corrupted (see http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_CzarLog_20120130#Friday:2012.02.03). Repairing the tables crashed the replication server: the error messages were about "Duplicate entr[ies]"
    4345
    4446I ran the usual: {{{STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;}}} about 16 times to fix it.
    45 = 20120131/08:57: ipp001 =
     47
     48== 20120131/08:57: ipp001 ==
    4649Ken Smith told us about ipp001 unavailability. The mysql server on that host crashed for some unknown reason (it was 62000 seconds behind so I assume it happened around 15:00 HST on Jan 30? Maybe it's related to the global loss of connection to the development cluster that happened around that time). I restarted the mysql server and had to fix it with {{{STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;}}}
    4750
    48 = 20111121/08:47: ippdb03/ipp001  =
     51= 2011 =
     52
     53== 20111121/08:47: ippdb03/ipp001  ==
    4954Two instances of czarpoll were started at the same time, running two "OPTIMIZE TABLE burntool". Replication broke on both clients. The message on both client was: {{{"Query partially completed on the master (error on master: 1053) and was aborted. There is a chance that your master is inconsistent at this point. If you are sure that your master is ok, run this query manually on the slave and then restart the slave with SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE; . Query: 'OPTIMIZE TABLE burntool'"}}}
    5055
    5156Fixed with {{{STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;}}}
    5257
    53 = 20111117/16:23: ipp001 crashed =
     58== 20111117/16:23: ipp001 crashed ==
    5459mysqld crashed. Nothing interesting in mysqld.err. Once restarted, slave resumed normally.
    5560
    56 = 20111115/09:23: ippc19 errors =
     61== 20111115/09:23: ippc19 errors ==
    5762
    5863Errors like 'Duplicate entry...' after ippc17 crash last night. The usual "STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;" fixed the problem.
    5964
    60 
    61 = 20111109/12:11: ippdb03/ipp001 errors =
     65== 20111109/12:11: ippdb03/ipp001 errors ==
    6266
    6367Alterations to gpc1:
    … …  
    8892mysql> SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;
    8993}}}
    90 = 20110921/13:56: ippdb03 error =
     94
     95== 20110921/13:56: ippdb03 error ==
    9196From Anonymous: "screwed-up the replication to ippdb03 while adding the new ippMonitor user (i didn't think i could update a slave, so believed i was safe). i've tried to fix it"
    9297The usual {{{STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;}}} fixed it
    9398
    94 = 20110913/1720: ippdb03 "error" =
     99== 20110913/1720: ippdb03 "error" ==
    95100Not really an error actually.
    96101isp was being updating with the statement ({{{SHOW FULL PROCESSLIST}}} or look at mysqld.err):
    … …  
    104109Note: I had to do it twice.
    105110
    106 = 20110913/0930: ipp001 error =
     111== 20110913/0930: ipp001 error ==
    107112{{{
    108113Error 'The table 'incoming' is full' on query. Default database: 'isp'. Query:
    … …  
    113118I don't understand why this _temporary_ table can't be created. I dropped isp from replication.
    114119
    115 = 20110801/1505: ippdb03 error =
     120== 20110801/1505: ippdb03 error ==
    116121
    117122ippdb03 crashed often during the last weekend. SHOW SLAVE STATUS\G showed this error message:
    … …  
    169174The problem seems to have been fixed
    170175
    171 = 20110727/1410: ippdb03 Replication Error =
     176== 20110727/1410: ippdb03 Replication Error ==
    172177ippdb03 crashed. SHOW SLAVE STATUS messages was like: 'Duplicate entry ...'
    173178
    … …  
    175180(Messages were like: 'Duplicate entry ...')
    176181
    177 = 20110726/1810: ippdb03 Replication Error =
     182== 20110726/1810: ippdb03 Replication Error ==
    178183ippdb03 crashed. SHOW SLAVE STATUS messages was like: 'Duplicate entry ...'
    179184
    … …  
    185190
    186191I checked with Chris. I manually dropped the table in gpc1 and did a 'STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;'. Hopefully we will not have to ingest a new dump once again...
    187 = 20110413/1450: ippadmin Replication Error =
     192
     193== 20110413/1450: ippadmin Replication Error ==
    188194 *  Logged onto ipp001
    189195 * Started mysql client as root
    … …  
    207213  * I'm setting up a new replicated server.
    208214
    209 = 20101116/1659: gpc1 Replication Error =
     215= 2010 =
     216
     217== 20101116/1659: gpc1 Replication Error ==
    210218  * Logged onto ipp001
    211219  * Started mysql client as root
    … …  
    213221  * Same solution was applied (Statement skipped)
    214222
    215 = 20101104/1429: gpc1 Replication Error =
     223== 20101104/1429: gpc1 Replication Error ==
    216224  Problem detected through czartool
    217225  * Logged onto ipp001