node 2 unable to join galera cluster
I'm facing exactly similar error to this post (https://support.severalnines.com/hc/en-us/community/posts/360032318011-Cannot-join-the-New-Cluster?page=1#community_comment_360003753152)
except 2 error lines are missing from my log plus my local firewall is already disabled. (also i've checked mysql connection remotely)
FIREWALL STATUS on both nodes :
# firewall-cmd --list-all
FirewallD is not running
Node1 is starting with wsrep-new-cluster as expected
[root@prod-jmn-push-cass-test-3 centos]# ps aux | grep mysqld
mysql 206670 0.1 1.2 1788224 199064 ? Sl 09:21 0:01 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid --wsrep-new-cluster --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
But in node2, not able to start mysqld service with following logs :
NODE2 error log :
2019-01-11T12:51:19.429223Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.24) starting as process 29460 ...
2019-01-11T12:51:19.431879Z 0 [Note] WSREP: Setting wsrep_ready to 0
2019-01-11T12:51:19.431918Z 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2019-01-11T12:51:19.431927Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-3/libgalera_smm.so'
2019-01-11T12:51:19.437760Z 0 [Note] WSREP: wsrep_load(): Galera 3.25(rddf9876) by Codership Oy <info@codership.com> loaded successfully.
2019-01-11T12:51:19.437829Z 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2019-01-11T12:51:19.438231Z 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
2019-01-11T12:51:19.438290Z 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2019-01-11T12:51:19.446302Z 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.172.0.22; base_port = 4567; cert.log_conflicts = ON; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S;
2019-01-11T12:51:19.469268Z 0 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1
2019-01-11T12:51:19.471111Z 0 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2019-01-11T12:51:19.471148Z 0 [Note] WSREP: wsrep_sst_grab()
2019-01-11T12:51:19.471156Z 0 [Note] WSREP: Start replication
2019-01-11T12:51:19.471172Z 0 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2019-01-11T12:51:19.471271Z 0 [Note] WSREP: protonet asio version 0
2019-01-11T12:51:19.471405Z 0 [Note] WSREP: Using CRC-32C for message checksums.
2019-01-11T12:51:19.471458Z 0 [Note] WSREP: backend: asio
2019-01-11T12:51:19.471540Z 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2019-01-11T12:51:19.471659Z 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2019-01-11T12:51:19.471679Z 0 [Note] WSREP: restore pc from disk failed
2019-01-11T12:51:19.471901Z 0 [Note] WSREP: GMCast version 0
2019-01-11T12:51:19.472111Z 0 [Note] WSREP: (975977c6, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2019-01-11T12:51:19.472140Z 0 [Note] WSREP: (975977c6, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2019-01-11T12:51:19.472506Z 0 [Note] WSREP: EVS version 0
2019-01-11T12:51:19.472645Z 0 [Note] WSREP: gcomm: connecting to group 'g1', peer '192.172.0.21:3306,192.172.0.22:3306'
2019-01-11T12:51:22.473549Z 0 [Note] WSREP: (975977c6, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.172.0.21:3306 timed out, no messages seen in PT3S
2019-01-11T12:51:22.474056Z 0 [Warning] WSREP: no nodes coming from prim view, prim not possible
2019-01-11T12:51:22.474114Z 0 [Note] WSREP: view(view_id(NON_PRIM,975977c6,1) memb {
975977c6,0
} joined {
} left {
} partitioned {
})
2019-01-11T12:51:22.974257Z 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50175S), skipping check
2019-01-11T12:51:26.974048Z 0 [Note] WSREP: (975977c6, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://192.172.0.21:3306 timed out, no messages seen in PT3S
2019-01-11T12:51:52.482535Z 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():158
2019-01-11T12:51:52.482563Z 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out)
2019-01-11T12:51:52.482639Z 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel 'g1' at 'gcomm://192.172.0.21:3306,192.172.0.22:3306': -110 (Connection timed out)
2019-01-11T12:51:52.482653Z 0 [ERROR] WSREP: gcs connect failed: Connection timed out
2019-01-11T12:51:52.482662Z 0 [ERROR] WSREP: wsrep::connect(gcomm://192.172.0.21:3306,192.172.0.22:3306) failed: 7
2019-01-11T12:51:52.482671Z 0 [ERROR] Aborting
2019-01-11T12:51:52.482676Z 0 [Note] WSREP: unireg_abort
2019-01-11T12:51:52.482682Z 0 [Note] Giving 0 client threads a chance to die gracefully
2019-01-11T12:51:52.482696Z 0 [Note] WSREP: Service disconnected.
2019-01-11T12:51:53.482853Z 0 [Note] WSREP: Some threads may fail to exit.
2019-01-11T12:51:53.482970Z 0 [Note] Binlog end
2019-01-11T12:51:53.483126Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
node1 conf :
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
binlog_format=ROW
bind-address=0.0.0.0
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=122M
wsrep_provider=/usr/lib64/galera-3/libgalera_smm.so
wsrep_provider_options="gcache.size=300M; gcache.page_size=300M;socket.checksum=1"
wsrep_cluster_name="g1"
wsrep_cluster_address="gcomm://192.172.0.21:3306,192.172.0.22:3306"
wsrep_sst_method=rsync
server_id=21
wsrep_node_address="192.172.0.22"
wsrep_node_name="node2"
wsrep_log_conflicts=ON
wsrep_provider_options="cert.log_conflicts=ON"
wsrep_debug=ON
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
connect_timeout=60
wait_timeout=60
interactive_timeout=60
net_read_timeout=60
net_write_timeout=60
max_allowed_packet=128M
log_error_verbosity = 3
node2 conf :
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
binlog_format=ROW
bind-address=0.0.0.0
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=122M
wsrep_provider=/usr/lib64/galera-3/libgalera_smm.so
wsrep_provider_options="gcache.size=300M; gcache.page_size=300M;socket.checksum=1"
wsrep_cluster_name="g1"
wsrep_cluster_address="gcomm://192.172.0.21:3306,192.172.0.22:3306"
wsrep_sst_method=rsync
server_id=22
wsrep_node_address="192.172.0.22"
wsrep_node_name="atish-vm-3"
wsrep_log_conflicts=ON
wsrep_provider_options="cert.log_conflicts=ON"
wsrep_debug=ON
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
connect_timeout=60
wait_timeout=60
interactive_timeout=60
net_read_timeout=60
net_write_timeout=60
max_allowed_packet=128M
log_error_verbosity = 3
NODE1 status :
mysql> show status like '%wsrep%';
+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_local_state_uuid | d8d90b26-15a4-11e9-9c1e-8aaa1305091b |
| wsrep_protocol_version | 9 |
| wsrep_last_committed | 3 |
| wsrep_replicated | 3 |
| wsrep_replicated_bytes | 704 |
| wsrep_repl_keys | 3 |
| wsrep_repl_keys_bytes | 96 |
| wsrep_repl_data_bytes | 408 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 2 |
| wsrep_received_bytes | 146 |
| wsrep_local_commits | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_cached_downto | 1 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 1.000000 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 1 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_open_transactions | 0 |
| wsrep_open_connections | 0 |
| wsrep_incoming_addresses | 192.172.0.21:3306 |
| wsrep_cluster_weight | 1 |
| wsrep_desync_count | 0 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | d8d7c6e3-15a4-11e9-bb96-a372f2ab24c5 |
| wsrep_cluster_conf_id | 1 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | d8d90b26-15a4-11e9-9c1e-8aaa1305091b |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.25(rddf9876) |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+
60 rows in set (0.00 sec)
I'm getting same error with latest galera for mysql 5.7, galera-3 on centos 7.4 & 7.6
Can anyone pls help me identify issue with node2 not getting into cluster?
Please sign in to leave a comment.
Comments
0 comments