| 16 | | |
| | 16 | The issue was with so_id = 6604319651 which didn't (and still doesn't) exist in ippdb01. |
| | 17 | When attempting to execute the statement: |
| | 18 | # at 1040640772 |
| | 19 | #190328 17:56:35 server id 2017081601 end_log_pos 1040641174 CRC32 0xb21142d6 Update_rows: table id 73 flags: STMT_END_F |
| | 20 | ... UPDATE nebulous`.`storage_object` WHERE so_id=6604319651 |
| | 21 | [...] |
| | 22 | # at 1040641174 |
| | 23 | replication stopped since row with so_id=6604319651 didn't exist. |
| | 24 | |
| | 25 | In db09 a few hours after the crash: |
| | 26 | mysql> SELECT * FROM storage_object WHERE so_id = 6604319651; |
| | 27 | Empty set (0.00 sec) |
| | 28 | So I assumed that so_id = 6604319651 would be deleted at some point in the replicant future. It is: |
| | 29 | # at 1040669677 |
| | 30 | #190328 17:56:35 server id 2017081601 end_log_pos 1040669911 CRC32 0x131ec7a0 Delete_rows: table id 73 flags: STMT_END_F |
| | 31 | ### DELETE FROM `nebulous`.`storage_object` WHERE so_id=6604319651 |
| | 32 | [...] |
| | 33 | # at 1040669911 |
| | 34 | |
| | 35 | So... |
| | 36 | * I stopped replication: STOP SLAVE |
| | 37 | * I skipped one stmt: SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1 |
| | 38 | * I manually played the INSERT stmt following #at 1040641174. Even if I thought the replicant would restart with it, I was not exactly sure if it would start there (better safe than sorry!). I checked that - in db01 -- the last so_id was 7876537519 and 7876537520 was not present. |
| | 39 | * I restarted replication (START SLAVE) which immediately crashed since it tried to insert 7876537520. It crashed at the expected position: 1040641548 |
| | 40 | * I skipped the (now offending) INSERT: STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE |
| | 41 | * And waited the crash at the DELETE stmt for so_id = 6604319651 (since there is nothing to delete) ... Which happened a few 100ths seconds later. |
| | 42 | * I skipped the offending DELETE statement: STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE |
| | 43 | * And the replication seems fine so far |