bootstrapping a galera cluster
I am going through the article
http://severalnines.com/blog/how-bootstrap-mysqlmariadb-galera-cluster
I have question around this assuming all nodes are down and we need to bootstrap.
Let's say node-3 goes down first, then node-2, and finally node-1 while it was still processing data. Which would mean that node-1 has the latest changes. Since we want to have the lastest data sync'd across all nodes, is it best to start the cluster with the highest value in grastate.dat (which I am guessing will be on node-1) or does it not matter and I can start with any node and it would still sync the latest changes?
My question is assuming that myql service has not yet been started so cannot check wsrep_last_committed variable yet.
Thank you!
-
Hi John,
To get the latest changes, you have to bootstrap from node-1 (the last which was down). Bootstrapping process will create a reference node, using data on that particular node to initialize the cluster. The other members will have the exact same data as this reference node. If you bootstrapped from node-3, you might miss some data which was written to node-1.
ClusterControl will perform 'mysqld_safe --wsrep-recover' to recover the last sequence number by checking it's transactional state. This is safer compare to checking the state inside grastate.dat as the sequence number would not be written there if the node got shutdown improperly (mysqld killed,power failure,hard reset).
Regards,
Ashraf
-
Thank you Ashraf. Is there another way apart from ClusterControl to find out which node has the latest data considering if we don't know which order or time wise the nodes went down?
I am looking into CC but it is not currently implemented. I want to cover my bases in case something happens before we start using it.
-
Hi John,
As I mentioned in the earlier reply, the best way to determine the most updated node is to run the following command on each of the Galera node before starting them up:
mysqld_safe --wsrep-recover
Then compare the sequence number generated by the process. The one with the highest number shall get bootstrapped. I recommend you to read this blog post by Percona's Przemysław Malkowski: https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/Regards,
Ashraf
-
Hi Ashraf,
This is the output of running mysqld_safe --wsrep-recover
I am not seeing a seq no here.
[root@server2 ~]# mysqld_safe --wsrep-recover
151208 11:10:11 mysqld_safe Logging to '/var/lib/mysql/server2.domain7.local.err'.
151208 11:10:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151208 11:10:12 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.4gFepa' --pid-file='/var/lib/mysql/server2.domain7.local-recover.pid'
151208 11:11:17 mysqld_safe WSREP: Recovered position 0ff9d7ec-782c-11e5-926c-cbce19f2eb60:2110253
151208 11:11:22 mysqld_safe mysqld from pid file /var/lib/mysql/server2.domain7.local.pid ended
Thanks for your help -
Hi,
What you want to look at is:
151208 11:11:17 mysqld_safe WSREP: Recovered position 0ff9d7ec-782c-11e5-926c-cbce19f2eb60:2110253Sequence number is at the end of the UUID. In your case it is: 2110253. InnoDB sequence number is another thing, not related to the Galera internals.
Thanks,
Krzysztof
Please sign in to leave a comment.
Comments
8 comments