Issue with a node rejoing

Steven

September 30, 2015 13:52

I had a node get shutdown in AWS and now it will not rejoin

i keep getting this

Job failed Exit Code: 1	10896	30 Sep 2015 09:44:40
10.10.2.25:3306: Aborting recovery. Exit Code: 1	10896	30 Sep 2015 09:44:40
10.10.2.25:3306: Refusing to recover node, incorrect wsrep_cluster_address. Exit Code: 1	10896	30 Sep 2015 09:44:40
10.10.2.25:3306: Detected wsrep_cluster_address='' on 10.10.2.25:3306. This is unsafe and data loss may happen if node restarts. Correct problem immediately and set e.g: wsrep_cluster_address='gcomm://ip1,ip2,ip3...' Exit Code: 1	10896	30 Sep 2015 09:44:40
Starting node recovery. i have this in my my.cnf file is there something i am missing? wsrep_node_address=10.10.2.25 # Provider specific configuration options wsrep_provider_options="gcache.size=128M; gmcast.segment=0" # Logical cluster name. Should be the same for all nodes. wsrep_cluster_name="my_wsrep_cluster" # Group communication system handle #wsrep_cluster_address=gcomm:// wsrep_cluster_address=gcomm://10.10.2.25:4567,10.10.3.75:4567,10.10.4.48:4567

0

Comments

9 comments

Krzysztof Ksiazek

October 01, 2015 09:01
Hi Steven. Please allow us some time to try to reproduce your issue. It doesn't look correct as you have wsrep_cluster_address configured as it should be.
Can you also let us know which version of ClusterControl do you use? You can find it in Settings -> General Settings -> Version
0

Comment actions Permalink
Krzysztof Ksiazek

October 01, 2015 12:20
We were able to reproduce the problem and we are working on a fix. In the meantime if you could send us your full my.cnf (you can open a ticket with us if you want to do it in more private fashion), that'd be great - we'll see what's causing the problem and get back to you.
0

Comment actions Permalink

Steven

October 01, 2015 12:29

Here is my information. ill post it here in hopes that it helps others.

Database Version	10.0.12-MariaDB-wsrep-log(MariaDB Server, wsrep_25.10.r4002)
Vendor:	mariadb
ClusterControl UI version:	1.2.10.d096ba8
ClusterControl CMON Version:	1.2.10.836
CMON API Version:	1.2.10.d9153ce
OS:	redhat

i have attached the my.cnf file i commented out wsrep_sst_auth=root:

Please let me know if i can help in anyway on this.

0

Krzysztof Ksiazek

October 01, 2015 13:14
Actually your my.cnf works ok with my cluster so it's hard to tell what was going on. Can you please attach your cmon*.log files (cmon.log and cmon_1(2, 3, depending on the cluster id).log) here? We've seen that errors related to parsing my.cnf are being logged.
0

Comment actions Permalink
Steven

October 01, 2015 13:24
Here is the last 500 lines of the cmon.log if you need more i can post that in as well.
0

Comment actions Permalink
Krzysztof Ksiazek

October 01, 2015 14:35
Do you use, by chance, '!' in the root password? This is not supported character and can lead to parse errors. We allow every character except $ ' " \ !

In the log I'm seeing:

Oct 01 13:09:23 : (ERROR) 10.10.2.25: error: syntax error in line 169 near token '!'

and line 169 in your my.cnf is:

wsrep_sst_auth=root:***Removed****
0

Comment actions Permalink
Steven

October 01, 2015 14:38
Yes i do,

How can i go about changing this?
0

Comment actions Permalink
Krzysztof Ksiazek

October 01, 2015 15:08
Please take a look at this procedure: http://support.severalnines.com/entries/22432317-Changing-cmon-and-root-passwords
It should guide you through the process of changing the password.
0

Comment actions Permalink
Steven

October 01, 2015 15:21
Great, Thank you!

i will update the password and post if that resolves my issue.
0

Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk