Node Failures Again
Can't get node 3 working. I have deleted .sst directory and yet still get errors:
2015-08-31 22:36:19 13777 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-08-31 22:36:19 13777 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-08-31 22:36:19 13777 [Note] WSREP: Service thread queue flushed.
2015-08-31 22:36:19 13777 [Note] WSREP: Assign initial position for certification: 72011027, protocol version: 3
2015-08-31 22:36:19 13777 [Note] WSREP: Service thread queue flushed.
2015-08-31 22:36:19 13777 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (8d45b507-be17-11e4-b5a6-5b88de84599e): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
2015-08-31 22:36:19 13777 [Note] WSREP: Member 3.0 (10.213.174.79) requested state transfer from '*any*'. Selected 1.0 (10.213.174.78)(SYNCED) as donor.
2015-08-31 22:36:19 13777 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 72011027)
2015-08-31 22:36:19 13777 [Note] WSREP: Requesting state transfer: success, donor: 1
WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /var/lib/mysql//.sst from previous state transfer (20150831 22:36:19.586)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150831 22:36:19.590)
WSREP_SST: [INFO] Proceeding with SST (20150831 22:36:19.597)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20150831 22:36:19.598)
removed `/var/lib/mysql/ibdata1'
removed `/var/lib/mysql/auto.cnf'
removed `/var/lib/mysql/ib_logfile1'
removed `/var/lib/mysql/ib_logfile0'
removed `/var/lib/mysql/mysql.sock'
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20150831 22:36:19.932)
2015-08-31 22:36:20 13777 [Note] WSREP: (b1896e2d, 'tcp://0.0.0.0:4567') turning message relay requesting off
xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 17 - File exists)
xbstream: failed to create file.
2015/08/31 22:36:30 socat[14127] E write(1, 0x110d380, 8192): Broken pipe
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 1 (20150831 22:36:30.097)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150831 22:36:30.099)
2015-08-31 22:36:30 13777 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.213.174.79' --auth 'root:PASSWORD' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '13777' '' : 32 (Broken pipe)
2015-08-31 22:36:30 13777 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-08-31 22:36:30 13777 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-08-31 22:36:30 13777 [ERROR] Aborting
2015-08-31 22:36:30 13777 [Warning] WSREP: 1.0 (10.213.174.78): State transfer to 3.0 (10.213.174.79) failed: -22 (Invalid argument)
2015-08-31 22:36:30 13777 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
2015-08-31 22:36:30 13777 [Note] WSREP: gcomm: terminating thread
2015-08-31 22:36:30 13777 [Note] WSREP: gcomm: joining thread
2015-08-31 22:36:30 13777 [Note] WSREP: gcomm: closing backend
2015-08-31 22:36:30 13777 [Note] WSREP: view(view_id(NON_PRIM,10c5aa33,2943) memb {
b1896e2d,0
} joined {
} left {
} partitioned {
10c5aa33,0
23e4a372,0
95cb583e,0
f170c0c3,0
})
2015-08-31 22:36:30 13777 [Note] WSREP: view((empty))
2015-08-31 22:36:30 13777 [Note] WSREP: gcomm: closed
2015-08-31 22:36:30 13777 [Note] WSREP: /usr/sbin/mysqld: Terminated.
150831 22:36:30 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
-
Hi
Are you sure you completely removed the .sst directory properly before restarting the node?
...
WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /var/lib/mysql//.sst from previous state transfer (20150831 22:36:19.586)
...
xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 17 - File exists)
xbstream: failed to create file.
...
Do you see any errors/issues in the innodb* logs on .78 (donor)?
Best RegardsAlex
Please sign in to leave a comment.
Comments
3 comments