Restore a galera cluster on a new host

Follow

Simon Tardif

• August 16, 2018 18:42

Hi we have two 2 galera cluster for MySQL. 
Each cluster has 3 nodes.
We are on centos.

We take backup with Percona xtrabackup. I have succesfully tested a restore on the entire galera cluster from a full backup of the same cluster.
Here are the step i use :
1-Stop all 3 nodes with systemctl stop mysqld
2-Prepare the full backup
3-On the first node remove everything in /var/lib/mysql
4-use xtrabackup --copy-back command
5-change permissions in /var/lib/mysql
6-Start first node with mysqld_bootstrap
7-On the second node : remove everything in /var/lib/mysql
8-use xtrabackup --copy-back command
9-change permissions in /var/lib/mysql
10-Start second node with systemctl start mysqld
11-Repeat step 7 to 10 on the third node

This procedure works.

What I want to do now is use the backup form my production environement and restore it on my test environment.
I follow the same steps, first node starts without problem. But at step 10 when I want to start the second node with systemctl start mysqld, i see in the the log that a full sst is started from node 1 but for i reason i don't understand a second mysqld proccess is started and th start failed because on the two process.

If i then restore my test from his own backup it works.

Any idea what step is missing. I'm new to galera cluster.

Any help would be appreciated
Thank you.

0

• 

  Ashraf Sharif

  • August 17, 2018 02:30

  Hi Simon,

  The best way to understand what was going on is to examine the MySQL error log of both donor and joiner nodes. However, by making some educated based on your restoration procedures, I can tell that the reason SST happens is because the second Galera node doesn't have information on the "Galera state" it should have been held, since you didn't configure the grastate.dat under MySQL datadir of the joiner node (second node).

  I'd recommend you to look at this blog post, https://severalnines.com/blog/how-avoid-sst-when-adding-new-node-galera-cluster as we described steps to restore a joiner node from a backup, configure proper values inside grastate.dat file to avoid SST when the joiner node joins the cluster for the first time. Take note on step 6, where the backup was taken with "--galera-info" flag and step 8 where we configure the Galera state (UUID and seqno) in grastate.dat.

  Regards,
  Ashraf

  0

  Comment actions Permalink
• 

  Simon Tardif

  • August 17, 2018 11:27

  Thank for your reply, i will look at the post.  I also see the parameter wsrep_sst_auth is not the same user/password for both cluster, so I think it may be a part of the problem too?

  0

  Comment actions Permalink
• 

  Simon Tardif

  • August 17, 2018 12:33

  Finally it worked by changing wsrep_sst_auth.  I red the post about avoiding full sst and it will be very usefull when our database will be bigger and this post made me understand how sst and ist works. Thank you! 

  0

  Comment actions Permalink

Please sign in to leave a comment.