Forum Stats

  • 3,733,344 Users
  • 2,246,744 Discussions
  • 7,856,663 Comments

Discussions

MySQL InnoDB Cluster 5.7 Group Replication Failure

MySQL_Checkpoint
MySQL_Checkpoint Member Posts: 7
edited March 2019 in MySQL Community Space

Hello,

We have configured MySQL InnoDB cluster on version 5.7 and one of the nodes(3 node cluster) were down. This was happened when all three cluster nodes were taken down for security patching.

We have tried to recover the cluster by issuing the command "dba.rebootClusterFromCompleteOutage()" but looks like the binlogs are purged and the node is unable to recover.

What are the steps to recover this node? Since it is community version I cannot use MySQL Enterprise Backup to restore. Do we have any steps to recover or restore a node in a InnoDB group replication using mysql dump utility?

Part of error log from the failed node:

2019-03-04T23:49:36.970839Z 3624 [Note] Slave SQL thread for channel 'group_replication_recovery' initialized, starting replication in log 'FIRST' at position 0, relay log './E2LXQA1ALFDB01-relay-bin-group_replication_recovery.000001' position: 4

2019-03-04T23:49:36.985336Z 3623 [Note] Slave I/O thread for channel 'group_replication_recovery': connected to master '[email protected]:3306',replication started in log 'FIRST' at position 4

2019-03-04T23:49:36.988164Z 3623 [ERROR] Error reading packet from server for channel 'group_replication_recovery': The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires. (server_errno=1236)

2019-03-04T23:49:36.988213Z 3623 [ERROR] Slave I/O for channel 'group_replication_recovery': Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.', Error_code: 1236

2019-03-04T23:49:36.988226Z 3623 [Note] Slave I/O thread exiting for channel 'group_replication_recovery', read up to log 'FIRST', position 4

2019-03-04T23:49:36.988286Z 41 [Note] Plugin group_replication reported: 'Terminating existing group replication donor connection and purging the corresponding logs.'

2019-03-04T23:49:36.988358Z 3624 [Note] Error reading relay log event for channel 'group_replication_recovery': slave SQL thread was killed

2019-03-04T23:49:36.988435Z 3624 [Note] Slave SQL thread for channel 'group_replication_recovery' exiting, replication stopped in log 'FIRST' at position 0

2019-03-04T23:49:37.016864Z 41 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='E2LXQA1ALFDB02', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.

2019-03-04T23:49:37.030769Z 41 [ERROR] Plugin group_replication reported: 'Maximum number of retries when trying to connect to a donor reached. Aborting group replication recovery.'

2019-03-04T23:49:37.030798Z 41 [Note] Plugin group_replication reported: 'Terminating existing group replication donor connection and purging the corresponding logs.'

2019-03-04T23:49:37.051169Z 41 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.

2019-03-04T23:49:37.069184Z 41 [ERROR] Plugin group_replication reported: 'Fatal error during the Recovery process of Group Replication. The server will leave the group.'

2019-03-04T23:49:37.069304Z 41 [Note] Plugin group_replication reported: 'Going to wait for view modification'

2019-03-04T23:49:40.336938Z 0 [Note] Plugin group_replication reported: 'Group membership changed: This member has left the group.'

Answers

  • 3770300
    3770300 Member Posts: 7
    edited March 2019

    1) How big is the database size?

    2) if it is small gb size you can use mysqldump.

    3) if it is large database size I can suggest to use percona xtrabackup tool(because this mysql community version)

  • Cobert
    Cobert Member Posts: 564 Silver Badge
    edited March 2019

    Hello,

    What are your backup position(s) at the moment? Do you have a hot backup, xtrabackup or mysqldump? I'm guessing only mysqldump if this is the CE?


    If you have lost the binary logs, have transactions been propagated to the slave nodes? You will need to verify the log sequence position in slave status / read master log pos and log file.

    You do not mention if this is master / slave / slave?

    If you are using GTID replication, there is a good post here, have you tried this, given error code 1234 so most likely GTID based replication?

    https://stackoverflow.com/questions/38390765/mysql-error-1236-when-using-gtid

Sign In or Register to comment.