galera nodes wont start after force shutdown

Follow

john test

• October 21, 2015 19:10

hello,

I accidentally ran optimize on a large table. Since it was taking really long I forced kill the mysql procs on the galera nodes.

I have 4 nodes plus 1 garb node.

Now only one node starts with (sometimes)

service mysql start --wsrep-new-cluster

The rest of the nodes don't come up when I do "service mysql start"

 

Please help

Thanks.

0

• 

  Alex

  • October 21, 2015 22:50

  Hi

  Please check the mysql error log and the innobackup*.log on the donor and joiner node that fails to start up. Any hints there? 

  If you don't care and just want to try to get the cluster up again then try this.

  Select a DB node to initialize the new cluster from. This DB node should have the most recent commits or you could end up with data loss. Compare the commit sequence number on all the DB nodes by running (all nodes should be shutdown)

  $ mysqld_safe --wsrep-recover
  
  

  Start up the DB node with the highest sequence number w/ "service mysql start --wsrep-new-cluster"

   

  Then try on one of the other DB nodes

  $ rm <mysql datadir>/grastate.dat

  $ service mysql start

  and wait for it to complete the full dataset sync (SST).

  
  
  

   

  0

  Comment actions Permalink
• 

  john test

  • October 22, 2015 03:43

  Thanks Alex. I was in a hurry so I ended up rebuilding but this is good info as it has happened to me a few times already so I will try next time.

  Thank you!

  0

  Comment actions Permalink

Please sign in to leave a comment.