Galera Node Won't Rejoin Cluster
2015-05-29 19:59:52 5835 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.105.46' --auth 'root:XXXXX' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '5835'
'' '
WSREP_SST: [INFO] Streaming with xbstream (20150529 19:59:52.981)
WSREP_SST: [INFO] Using socat as streamer (20150529 19:59:52.982)
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150529 19:59:53.002)
2015-05-29 19:59:53 5835 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.1.105.46:4444/xtrabackup_sst//1
2015-05-29 19:59:53 5835 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-05-29 19:59:53 5835 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-05-29 19:59:53 5835 [Note] WSREP: Service thread queue flushed.
2015-05-29 19:59:53 5835 [Note] WSREP: Assign initial position for certification: 1712412, protocol version: 3
2015-05-29 19:59:53 5835 [Note] WSREP: Service thread queue flushed.
2015-05-29 19:59:53 5835 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (3b08de3b-dfe0-11e4-bfbf-129c4b153705): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
2015-05-29 19:59:53 5835 [Note] WSREP: Member 2.0 (10.1.105.46) requested state transfer from '*any*'. Selected 0.0 (10.1.105.48)(SYNCED) as donor.
2015-05-29 19:59:53 5835 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1712412)
2015-05-29 19:59:53 5835 [Note] WSREP: Requesting state transfer: success, donor: 0
WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /var/lib/mysql//.sst from previous state transfer (20150529 19:59:53.591)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150529 19:59:53.596)
WSREP_SST: [INFO] Proceeding with SST (20150529 19:59:53.605)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20150529 19:59:53.607)
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20150529 19:59:53.616)
2015-05-29 19:59:55 5835 [Note] WSREP: (447109e6, 'tcp://0.0.0.0:4567') turning message relay requesting off
xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 17 - File exists)
xbstream: failed to create file.
2015/05/29 20:00:04 socat[6119] E write(1, 0x12293a0, 8192): Broken pipe
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 1 (20150529 20:00:04.109)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150529 20:00:04.111)
2015-05-29 20:00:04 5835 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.105.46' --auth 'root:XXXXXX03' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '5835' '' : 32 (Broken pipe)
2015-05-29 20:00:04 5835 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-05-29 20:00:04 5835 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-05-29 20:00:04 5835 [ERROR] Aborting
2015-05-29 20:00:04 5835 [Warning] WSREP: 0.0 (10.1.105.48): State transfer to 2.0 (10.1.105.46) failed: -22 (Invalid argument)
2015-05-29 20:00:04 5835 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
2015-05-29 20:00:04 5835 [Note] WSREP: gcomm: terminating thread
2015-05-29 20:00:04 5835 [Note] WSREP: gcomm: joining thread
2015-05-29 20:00:04 5835 [Note] WSREP: gcomm: closing backend
2015-05-29 20:00:04 5835 [Note] WSREP: view(view_id(NON_PRIM,2822aa74,29) memb {
447109e6,0
} joined {
} left {
} partitioned {
2822aa74,0
3d591ad3,0
faf4ce91,0
})
2015-05-29 20:00:04 5835 [Note] WSREP: view((empty))
2015-05-29 20:00:04 5835 [Note] WSREP: gcomm: closed
2015-05-29 20:00:04 5835 [Note] WSREP: /usr/sbin/mysqld: Terminated.
150529 20:00:04 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
Any ideas? What else should I look for?
Please sign in to leave a comment.
Comments
2 comments