MySQL Galera cluster keeps failing
I have setup a Galera cluster with the Severalnine configurator. Everything was working great for a few weeks. Now, once or twice a day the whole cluster fails and I have to reboot all the db nodes in the cluster.
In the Alarms for the Dashboard I see this message for all db nodes:
MYSQLD DISCONNECTED
In the /var/log/cmon.log, I see this for all the nodes:
Feb 07 07:34:31 : (INFO) Connected to MySQL Server @ 127.0.0.1
Feb 07 07:34:32 : (INFO) Stopping Query Sampling
Feb 07 07:34:39 : (INFO) Starting Query Sampling, Long_query_time: SET GLOBAL LONG_QUERY_TIME=0.5
Feb 07 07:34:58 : (INFO) Connected to MySQL Server @ 127.0.0.1
Feb 07 07:34:59 : (INFO) Stopping Query Sampling
Feb 07 07:35:06 : (INFO) Starting Query Sampling, Long_query_time: SET GLOBAL LONG_QUERY_TIME=0.5
Feb 07 07:35:10 : (INFO) pid did not exist
Feb 07 07:35:11 : (WARNING) Could not open the /proc/diskstats file
Feb 07 07:35:23 : (INFO) pid did not exist
Feb 07 07:36:11 : (WARNING) Could not open the /proc/diskstats file
Feb 07 07:36:15 : (INFO) pid did not exist
Feb 07 07:37:11 : (WARNING) Could not open the /proc/diskstats file
Feb 07 07:37:20 : (INFO) pid did not exist
Feb 07 07:38:11 : (WARNING) Could not open the /proc/diskstats file
Feb 07 07:38:12 : (INFO) pid did not exist
Feb 07 07:39:11 : (WARNING) Could not open the /proc/diskstats file
Feb 07 07:39:17 : (INFO) pid did not exist
Feb 07 07:40:11 : (WARNING) Could not open the /proc/diskstats file
Any idea what is going on and why the cluster is failing so much lately? Thankfully this cluster is not currently in a production environment, but I'm not too confident about using it now when I have to reboot all db nodes 2 times a day to get it working.
One other note... After I reboot all the db nodes, they connect back up and start working again...
-
Here is all the log items from this date:
130207 7:35:07 [ERROR] Slave SQL: Could not execute Delete_rows event on table prudhub.weather_forcast; Can$
130207 7:35:07 [Warning] WSREP: RBR event 2 Delete_rows apply warning: 137, 143022
130207 7:35:07 [ERROR] WSREP: Failed to apply trx: source: e2cbebe0-6fc4-11e2-0800-c32652869327 version: 2 $
130207 7:35:07 [ERROR] WSREP: Failed to apply app buffer: ê ^SQ^S, seqno: 143022, status: WSREP_FATAL
at galera/src/replicator_smm.cpp:apply_wscoll():49
at galera/src/replicator_smm.cpp:apply_trx_ws():120
130207 7:35:07 [ERROR] WSREP: Node consistency compromized, aborting...
130207 7:35:07 [Note] WSREP: Closing send monitor...
130207 7:35:07 [Note] WSREP: Closed send monitor.
130207 7:35:07 [Note] WSREP: gcomm: terminating thread
130207 7:35:07 [Note] WSREP: gcomm: joining thread
130207 7:35:07 [Note] WSREP: gcomm: closing backend
130207 7:35:09 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') turning message r$
130207 7:35:09 [Note] WSREP: view(view_id(NON_PRIM,046b4407-6fc5-11e2-0800-06e13b310150,5) memb {
178c246c-6fc5-11e2-0800-278eb5fd5509,
} joined {
} left {
} partitioned {
046b4407-6fc5-11e2-0800-06e13b310150,
0a96b9a1-6fc5-11e2-0800-aabaabcfbbed,
113af95f-6fc5-11e2-0800-fdef2e8eb4ef,
e2cbebe0-6fc4-11e2-0800-c32652869327,
})
130207 7:35:09 [Note] WSREP: view((empty))
130207 7:35:09 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
130207 7:35:09 [Note] WSREP: gcomm: closed
130207 7:35:09 [Note] WSREP: Flow-control interval: [16, 16]
130207 7:35:09 [Note] WSREP: Received NON-PRIMARY.
130207 7:35:09 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 143023)
130207 7:35:09 [Note] WSREP: Received self-leave message.
130207 7:35:09 [Note] WSREP: Flow-control interval: [0, 0]
130207 7:35:09 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130207 7:35:09 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 143023)
130207 7:35:09 [Note] WSREP: RECV thread exiting 0: Success
130207 7:35:09 [Note] WSREP: recv_thread() joined.
130207 7:35:09 [Note] WSREP: Closing slave action queue.
130207 7:35:09 [Note] WSREP: /usr/local/mysql/bin/mysqld: Terminated.
Aborted
130207 07:35:09 mysqld_safe Number of processes running now: 0
130207 07:35:09 mysqld_safe WSREP: not restarting wsrep node automatically
130207 07:35:09 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended -
My Galera version is 2.2.0. Also, I don't believe I'm using any foreign keys for any tables either.
Here is the full error log:
595cb892-6f11-11e2-0800-b1ef5eab7031,
725bd002-6b46-11e2-0800-c98523a682fa,
} joined {
} left {
} partitioned {
})
130205 1:25:08 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 5
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: sent state msg: 59a9d644-6f11-11e2-0800-cdee46053726
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: got state msg: 59a9d644-6f11-11e2-0800-cdee46053726 from 2 (db004)
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: got state msg: 59a9d644-6f11-11e2-0800-cdee46053726 from 4 (db001)
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: got state msg: 59a9d644-6f11-11e2-0800-cdee46053726 from 1 (db003)
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: got state msg: 59a9d644-6f11-11e2-0800-cdee46053726 from 0 (db005)
130205 1:25:08 [Note] WSREP: STATE EXCHANGE: got state msg: 59a9d644-6f11-11e2-0800-cdee46053726 from 3 (db002)
130205 1:25:08 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 9,
members = 3/5 (joined/total),
act_id = 379082,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 725c8b6a-6b46-11e2-0800-ea0610703e3e
130205 1:25:08 [Note] WSREP: Flow-control interval: [36, 36]
130205 1:25:08 [Note] WSREP: New cluster view: global state: 725c8b6a-6b46-11e2-0800-ea0610703e3e:379082, view# 10: Primary, number of nodes: 5, my index: 4, protocol version 2
130205 1:25:08 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 1:25:08 [Note] WSREP: Assign initial position for certification: 379082, protocol version: 2
130205 1:25:10 [Note] WSREP: 2 (db004): State transfer from 0 (db005) complete.
130205 1:25:10 [Note] WSREP: Member 2 (db004) synced with group.
130205 1:25:10 [Note] WSREP: Node 3 (db002) requested state transfer from '*any*'. Selected 0 (db005)(SYNCED) as donor.
130205 1:25:19 [Note] WSREP: 0 (db005): State transfer to 3 (db002) complete.
130205 1:25:19 [Note] WSREP: Member 0 (db005) synced with group.
130205 1:25:21 [Note] WSREP: 3 (db002): State transfer from 0 (db005) complete.
130205 1:25:21 [Note] WSREP: Member 3 (db002) synced with group.
130205 3:07:30 [ERROR] Slave SQL: Could not execute Delete_rows event on table prudhub.weather_forcast; Can't find record in 'weather_forcast', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log FIRST, end_log_pos 132, Error_code: 1032
130205 3:07:30 [Warning] WSREP: RBR event 2 Delete_rows apply warning: 137, 396123
130205 3:07:30 [Note] WSREP: (725bd002-6b46-11e2-0800-c98523a682fa, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.5:4567
130205 3:07:30 [Warning] WSREP: failed to replay trx: source: 725bd002-6b46-11e2-0800-c98523a682fa version: 2 local: 1 state: REPLAYING flags: 1 conn_id: 267789 trx_id: 11290255 seqnos (l: 396694, g: 396123, s: 396121, d: 396122, ts: 1360019250802573109)
130205 3:07:30 [Warning] WSREP: Failed to apply app buffer: 2?Q, seqno: 396123, status: WSREP_FATAL
at galera/src/replicator_smm.cpp:apply_wscoll():49
at galera/src/replicator_smm.cpp:apply_trx_ws():120
130205 3:07:30 [ERROR] WSREP: trx_replay failed for: 5, query: DELETE FROM weather_forcast WHERE location = '10.0.0.209'
130205 3:07:30 [ERROR] Aborting
130205 3:07:32 [Note] WSREP: (725bd002-6b46-11e2-0800-c98523a682fa, 'tcp://0.0.0.0:4567') reconnecting to 53010c39-6f11-11e2-0800-34592a77324e (tcp://10.10.20.5:4567), attempt 0
130205 3:07:34 [Note] WSREP: (725bd002-6b46-11e2-0800-c98523a682fa, 'tcp://0.0.0.0:4567') reconnecting to 595cb892-6f11-11e2-0800-b1ef5eab7031 (tcp://10.10.20.3:4567), attempt 0
130205 3:07:34 [Note] WSREP: (725bd002-6b46-11e2-0800-c98523a682fa, 'tcp://0.0.0.0:4567') reconnecting to 4c926c13-6f11-11e2-0800-6ec06cbeecd6 (tcp://10.10.20.4:4567), attempt 0
130205 3:07:34 [Note] WSREP: (725bd002-6b46-11e2-0800-c98523a682fa, 'tcp://0.0.0.0:4567') reconnecting to 462dbf21-6f11-11e2-0800-4eb3731d23dc (tcp://10.10.20.6:4567), attempt 0
130205 3:07:35 [Note] WSREP: evs::proto(725bd002-6b46-11e2-0800-c98523a682fa, GATHER, view_id(REG,462dbf21-6f11-11e2-0800-4eb3731d23dc,10)) suspecting node: 462dbf21-6f11-11e2-0800-4eb3731d23dc
130205 3:07:35 [Note] WSREP: evs::proto(725bd002-6b46-11e2-0800-c98523a682fa, GATHER, view_id(REG,462dbf21-6f11-11e2-0800-4eb3731d23dc,10)) suspecting node: 4c926c13-6f11-11e2-0800-6ec06cbeecd6
130205 3:07:35 [Note] WSREP: evs::proto(725bd002-6b46-11e2-0800-c98523a682fa, GATHER, view_id(REG,462dbf21-6f11-11e2-0800-4eb3731d23dc,10)) suspecting node: 53010c39-6f11-11e2-0800-34592a77324e
130205 3:07:35 [Note] WSREP: evs::proto(725bd002-6b46-11e2-0800-c98523a682fa, GATHER, view_id(REG,462dbf21-6f11-11e2-0800-4eb3731d23dc,10)) suspecting node: 595cb892-6f11-11e2-0800-b1ef5eab7031
130205 3:07:36 [Note] WSREP: view(view_id(PRIM,725bd002-6b46-11e2-0800-c98523a682fa,11) memb {
725bd002-6b46-11e2-0800-c98523a682fa,
} joined {
} left {
} partitioned {
462dbf21-6f11-11e2-0800-4eb3731d23dc,
4c926c13-6f11-11e2-0800-6ec06cbeecd6,
53010c39-6f11-11e2-0800-34592a77324e,
595cb892-6f11-11e2-0800-b1ef5eab7031,
})
130205 3:07:36 [Note] WSREP: forgetting 462dbf21-6f11-11e2-0800-4eb3731d23dc (tcp://10.10.20.6:4567)
130205 3:07:36 [Note] WSREP: forgetting 4c926c13-6f11-11e2-0800-6ec06cbeecd6 (tcp://10.10.20.4:4567)
130205 3:07:36 [Note] WSREP: forgetting 53010c39-6f11-11e2-0800-34592a77324e (tcp://10.10.20.5:4567)
130205 3:07:36 [Note] WSREP: forgetting 595cb892-6f11-11e2-0800-b1ef5eab7031 (tcp://10.10.20.3:4567)
130205 3:07:36 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
130205 3:07:36 [Note] WSREP: (725bd002-6b46-11e2-0800-c98523a682fa, 'tcp://0.0.0.0:4567') turning message relay requesting off
130205 3:07:36 [Note] WSREP: STATE_EXCHANGE: sent state UUID: aa67538e-6f1f-11e2-0800-54c41ffafffc
130205 3:07:36 [Note] WSREP: STATE EXCHANGE: sent state msg: aa67538e-6f1f-11e2-0800-54c41ffafffc
130205 3:07:36 [Note] WSREP: STATE EXCHANGE: got state msg: aa67538e-6f1f-11e2-0800-54c41ffafffc from 0 (db001)
130205 3:07:36 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 10,
members = 1/1 (joined/total),
act_id = 396124,
last_appl. = 395660,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 725c8b6a-6b46-11e2-0800-ea0610703e3e
130205 3:07:36 [Note] WSREP: Flow-control interval: [16, 16]
130205 3:07:42 [Note] WSREP: cleaning up 595cb892-6f11-11e2-0800-b1ef5eab7031 (tcp://10.10.20.3:4567)
130205 3:07:42 [Note] WSREP: cleaning up 4c926c13-6f11-11e2-0800-6ec06cbeecd6 (tcp://10.10.20.4:4567)
130205 3:07:42 [Note] WSREP: cleaning up 53010c39-6f11-11e2-0800-34592a77324e (tcp://10.10.20.5:4567)
130205 3:07:42 [Note] WSREP: cleaning up 462dbf21-6f11-11e2-0800-4eb3731d23dc (tcp://10.10.20.6:4567)
130205 03:32:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130205 03:32:00 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.ATlcr6aOOd
130205 03:32:02 mysqld_safe WSREP: Failed to recover position:
130205 3:32:02 [Note] WSREP: Read nil XID from storage engines, skipping position init
130205 3:32:02 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130205 3:32:02 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130205 3:32:02 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130205 3:32:02 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130205 3:32:02 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130205 3:32:02 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130205 3:32:02 InnoDB: The InnoDB memory heap is disabled
130205 3:32:02 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130205 3:32:02 InnoDB: Compressed tables use zlib 1.2.3
130205 3:32:02 InnoDB: Using Linux native AIO
130205 3:32:02 InnoDB: Initializing buffer pool, size = 523.0M
130205 3:32:02 InnoDB: Completed initialization of buffer pool
130205 3:32:02 InnoDB: highest supported file format is Barracuda.
130205 3:32:02 InnoDB: Waiting for the background threads to start
130205 3:32:03 InnoDB: 1.1.8 started; log sequence number 4258164834
130205 3:32:03 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130205 3:32:03 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130205 3:32:03 [Note] Server socket created on IP: '0.0.0.0'.
130205 3:32:03 [Warning] 'user' entry 'root@db001' ignored in --skip-name-resolve mode.
130205 3:32:03 [Warning] 'user' entry '@db001' ignored in --skip-name-resolve mode.
130205 3:32:03 [Warning] 'proxies_priv' entry '@ root@db001' ignored in --skip-name-resolve mode.
130205 3:32:03 [Note] Event Scheduler: Loaded 0 events
130205 3:32:03 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.28' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution, wsrep_23.7.r3829
130205 3:33:23 [Note] WSREP: Stop replication
130205 3:33:25 [Note] WSREP: killing local connection: 2
130205 3:33:25 [Note] WSREP: dtor state: CLOSED
130205 3:33:25 [Note] WSREP: apply mon: entered 0
130205 3:33:25 [Note] WSREP: apply mon: entered 0
130205 3:33:25 [Note] WSREP: apply mon: entered 0
130205 3:33:25 [Note] WSREP: cert index usage at exit 0
130205 3:33:25 [Note] WSREP: cert trx map usage at exit 0
130205 3:33:25 [Note] WSREP: deps set usage at exit 0
130205 3:33:25 [Note] WSREP: avg deps dist 0
130205 3:33:25 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130205 3:33:25 [Note] WSREP: Flushing memory map to disk...
130205 3:33:25 [Note] WSREP: Initial position: 725c8b6a-6b46-11e2-0800-ea0610703e3e:396122
130205 3:33:25 [Note] WSREP: wsrep_load(): loading provider library 'none'
130205 3:33:25 [ERROR] WSREP: Failed to get provider options
130205 3:33:25 [Note] WSREP: Stop replication
130205 3:33:27 [Note] WSREP: Initial position: 725c8b6a-6b46-11e2-0800-ea0610703e3e:396122
130205 3:33:27 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130205 3:33:27 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130205 3:33:27 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130205 3:33:27 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130205 3:33:27 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130205 3:33:27 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130205 03:33:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130205 03:33:44 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.J3QBPMz9Wh
130205 03:33:46 mysqld_safe WSREP: Failed to recover position:
130205 3:33:46 [Note] WSREP: Read nil XID from storage engines, skipping position init
130205 3:33:46 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130205 3:33:46 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130205 3:33:46 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130205 3:33:46 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130205 3:33:46 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130205 3:33:46 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130205 3:33:46 [Note] WSREP: wsrep_sst_grab()
130205 3:33:46 [Note] WSREP: Start replication
130205 3:33:46 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
130205 3:33:46 [Note] WSREP: protonet asio version 0
130205 3:33:46 [Note] WSREP: backend: asio
130205 3:33:46 [Note] WSREP: GMCast version 0
130205 3:33:46 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
130205 3:33:46 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
130205 3:33:46 [Note] WSREP: EVS version 0
130205 3:33:46 [Note] WSREP: PC version 0
130205 3:33:46 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer ''
130205 3:33:46 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,1) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
} joined {
} left {
} partitioned {
})
130205 3:33:46 [Note] WSREP: gcomm: connected
130205 3:33:46 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130205 3:33:46 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130205 3:33:46 [Note] WSREP: Opened channel 'my_wsrep_cluster'
130205 3:33:46 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
130205 3:33:46 [Note] WSREP: Waiting for SST to complete.
130205 3:33:46 [Note] WSREP: Starting new group from scratch: 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 3:33:46 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 524bef6b-6f23-11e2-0800-61ef89799881
130205 3:33:46 [Note] WSREP: STATE EXCHANGE: sent state msg: 524bef6b-6f23-11e2-0800-61ef89799881
130205 3:33:46 [Note] WSREP: STATE EXCHANGE: got state msg: 524bef6b-6f23-11e2-0800-61ef89799881 from 0 (db001)
130205 3:33:46 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 0,
members = 1/1 (joined/total),
act_id = 0,
last_appl. = -1,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 3:33:46 [Note] WSREP: Flow-control interval: [16, 16]
130205 3:33:46 [Note] WSREP: Restored state OPEN -> JOINED (0)
130205 3:33:46 [Note] WSREP: Member 0 (db001) synced with group.
130205 3:33:46 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
130205 3:33:46 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2
130205 3:33:46 [Note] WSREP: SST complete, seqno: 0
130205 3:33:46 InnoDB: The InnoDB memory heap is disabled
130205 3:33:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130205 3:33:46 InnoDB: Compressed tables use zlib 1.2.3
130205 3:33:46 InnoDB: Using Linux native AIO
130205 3:33:47 InnoDB: Initializing buffer pool, size = 523.0M
130205 3:33:47 InnoDB: Completed initialization of buffer pool
130205 3:33:47 InnoDB: highest supported file format is Barracuda.
130205 3:33:47 InnoDB: Waiting for the background threads to start
130205 3:33:48 InnoDB: 1.1.8 started; log sequence number 4258164834
130205 3:33:48 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130205 3:33:48 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130205 3:33:48 [Note] Server socket created on IP: '0.0.0.0'.
130205 3:33:48 [Warning] 'user' entry 'root@db001' ignored in --skip-name-resolve mode.
130205 3:33:48 [Warning] 'user' entry '@db001' ignored in --skip-name-resolve mode.
130205 3:33:48 [Warning] 'proxies_priv' entry '@ root@db001' ignored in --skip-name-resolve mode.
130205 3:33:48 [Note] Event Scheduler: Loaded 0 events
130205 3:33:48 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.28' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution, wsrep_23.7.r3829
130205 3:33:48 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:33:48 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2
130205 3:33:48 [Note] WSREP: Synchronized with group, ready for connections
130205 3:33:48 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:34:43 [Note] WSREP: declaring 73c37263-6f23-11e2-0800-63de6fa27d76 stable
130205 3:34:43 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,2) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
73c37263-6f23-11e2-0800-63de6fa27d76,
} joined {
} left {
} partitioned {
})
130205 3:34:43 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
130205 3:34:43 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 74109570-6f23-11e2-0800-e487b11b67e2
130205 3:34:43 [Note] WSREP: STATE EXCHANGE: sent state msg: 74109570-6f23-11e2-0800-e487b11b67e2
130205 3:34:43 [Note] WSREP: STATE EXCHANGE: got state msg: 74109570-6f23-11e2-0800-e487b11b67e2 from 0 (db001)
130205 3:34:44 [Note] WSREP: STATE EXCHANGE: got state msg: 74109570-6f23-11e2-0800-e487b11b67e2 from 1 (db004)
130205 3:34:44 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 1,
members = 1/2 (joined/total),
act_id = 25,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 3:34:44 [Note] WSREP: Flow-control interval: [23, 23]
130205 3:34:44 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:25, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 2
130205 3:34:44 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:34:44 [Note] WSREP: Assign initial position for certification: 25, protocol version: 2
130205 3:34:46 [Note] WSREP: Node 1 (db004) requested state transfer from '*any*'. Selected 0 (db001)(SYNCED) as donor.
130205 3:34:46 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 28)
130205 3:34:46 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:34:46 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.10.20.5:4444/rsync_sst' --auth 'root:galera' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --gtid '524bcc14-6f23-11e2-0800-d3c4b7c44389:28''
130205 3:34:46 [Note] WSREP: sst_donor_thread signaled with 0
130205 3:34:46 [Note] WSREP: Flushing tables for SST...
130205 3:34:46 [Note] WSREP: Provider paused at 524bcc14-6f23-11e2-0800-d3c4b7c44389:28
130205 3:34:46 [Note] WSREP: Tables flushed.
130205 3:34:53 [Note] WSREP: Provider resumed.
130205 3:34:53 [Note] WSREP: 0 (db001): State transfer to 1 (db004) complete.
130205 3:34:53 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 32)
130205 3:34:53 [Note] WSREP: Member 0 (db001) synced with group.
130205 3:34:53 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 32)
130205 3:34:53 [Note] WSREP: Synchronized with group, ready for connections
130205 3:34:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:34:54 [Note] WSREP: declaring 73c37263-6f23-11e2-0800-63de6fa27d76 stable
130205 3:34:54 [Note] WSREP: declaring 7a3461b1-6f23-11e2-0800-6d070e4c5cc6 stable
130205 3:34:54 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,3) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
73c37263-6f23-11e2-0800-63de6fa27d76,
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
} joined {
} left {
} partitioned {
})
130205 3:34:54 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
130205 3:34:54 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7a819635-6f23-11e2-0800-080aa56e90d8
130205 3:34:54 [Note] WSREP: STATE EXCHANGE: sent state msg: 7a819635-6f23-11e2-0800-080aa56e90d8
130205 3:34:54 [Note] WSREP: STATE EXCHANGE: got state msg: 7a819635-6f23-11e2-0800-080aa56e90d8 from 0 (db001)
130205 3:34:54 [Note] WSREP: STATE EXCHANGE: got state msg: 7a819635-6f23-11e2-0800-080aa56e90d8 from 1 (db004)
130205 3:34:54 [Note] WSREP: STATE EXCHANGE: got state msg: 7a819635-6f23-11e2-0800-080aa56e90d8 from 2 (db002)
130205 3:34:54 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 2,
members = 1/3 (joined/total),
act_id = 32,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 3:34:54 [Note] WSREP: Flow-control interval: [28, 28]
130205 3:34:54 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:32, view# 3: Primary, number of nodes: 3, my index: 0, protocol version 2
130205 3:34:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:34:54 [Note] WSREP: Assign initial position for certification: 32, protocol version: 2
130205 3:34:56 [Note] WSREP: 1 (db004): State transfer from 0 (db001) complete.
130205 3:34:56 [Note] WSREP: Member 1 (db004) synced with group.
130205 3:34:56 [Note] WSREP: Node 2 (db002) requested state transfer from '*any*'. Selected 0 (db001)(SYNCED) as donor.
130205 3:34:56 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 32)
130205 3:34:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:34:56 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.10.20.3:4444/rsync_sst' --auth 'root:galera' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --gtid '524bcc14-6f23-11e2-0800-d3c4b7c44389:32''
130205 3:34:56 [Note] WSREP: sst_donor_thread signaled with 0
130205 3:34:56 [Note] WSREP: Flushing tables for SST...
130205 3:34:56 [Note] WSREP: Provider paused at 524bcc14-6f23-11e2-0800-d3c4b7c44389:32
130205 3:34:56 [Note] WSREP: Tables flushed.
130205 3:35:04 [Note] WSREP: Provider resumed.
130205 3:35:04 [Note] WSREP: 0 (db001): State transfer to 2 (db002) complete.
130205 3:35:04 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 32)
130205 3:35:04 [Note] WSREP: Member 0 (db001) synced with group.
130205 3:35:04 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 32)
130205 3:35:04 [Note] WSREP: Synchronized with group, ready for connections
130205 3:35:04 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:05 [Note] WSREP: declaring 73c37263-6f23-11e2-0800-63de6fa27d76 stable
130205 3:35:05 [Note] WSREP: declaring 7a3461b1-6f23-11e2-0800-6d070e4c5cc6 stable
130205 3:35:05 [Note] WSREP: declaring 80d9486f-6f23-11e2-0800-89086bcae906 stable
130205 3:35:05 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,4) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
73c37263-6f23-11e2-0800-63de6fa27d76,
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
80d9486f-6f23-11e2-0800-89086bcae906,
} joined {
} left {
} partitioned {
})
130205 3:35:05 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4
130205 3:35:05 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 811210ca-6f23-11e2-0800-b70bc3aa3767
130205 3:35:05 [Note] WSREP: STATE EXCHANGE: sent state msg: 811210ca-6f23-11e2-0800-b70bc3aa3767
130205 3:35:05 [Note] WSREP: STATE EXCHANGE: got state msg: 811210ca-6f23-11e2-0800-b70bc3aa3767 from 0 (db001)
130205 3:35:05 [Note] WSREP: STATE EXCHANGE: got state msg: 811210ca-6f23-11e2-0800-b70bc3aa3767 from 1 (db004)
130205 3:35:05 [Note] WSREP: STATE EXCHANGE: got state msg: 811210ca-6f23-11e2-0800-b70bc3aa3767 from 2 (db002)
130205 3:35:05 [Note] WSREP: STATE EXCHANGE: got state msg: 811210ca-6f23-11e2-0800-b70bc3aa3767 from 3 (db005)
130205 3:35:05 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 3,
members = 2/4 (joined/total),
act_id = 32,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 3:35:05 [Note] WSREP: Flow-control interval: [32, 32]
130205 3:35:05 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:32, view# 4: Primary, number of nodes: 4, my index: 0, protocol version 2
130205 3:35:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:05 [Note] WSREP: Assign initial position for certification: 32, protocol version: 2
130205 3:35:07 [Note] WSREP: 2 (db002): State transfer from 0 (db001) complete.
130205 3:35:07 [Note] WSREP: Member 2 (db002) synced with group.
130205 3:35:07 [Note] WSREP: Node 3 (db005) requested state transfer from '*any*'. Selected 0 (db001)(SYNCED) as donor.
130205 3:35:07 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 32)
130205 3:35:07 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:07 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.10.20.6:4444/rsync_sst' --auth 'root:galera' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --gtid '524bcc14-6f23-11e2-0800-d3c4b7c44389:32''
130205 3:35:07 [Note] WSREP: sst_donor_thread signaled with 0
130205 3:35:07 [Note] WSREP: Flushing tables for SST...
130205 3:35:07 [Note] WSREP: Provider paused at 524bcc14-6f23-11e2-0800-d3c4b7c44389:32
130205 3:35:07 [Note] WSREP: Tables flushed.
130205 3:35:14 [Note] WSREP: Provider resumed.
130205 3:35:14 [Note] WSREP: 0 (db001): State transfer to 3 (db005) complete.
130205 3:35:14 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 32)
130205 3:35:14 [Note] WSREP: Member 0 (db001) synced with group.
130205 3:35:14 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 32)
130205 3:35:14 [Note] WSREP: Synchronized with group, ready for connections
130205 3:35:14 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:16 [Note] WSREP: declaring 73c37263-6f23-11e2-0800-63de6fa27d76 stable
130205 3:35:16 [Note] WSREP: declaring 7a3461b1-6f23-11e2-0800-6d070e4c5cc6 stable
130205 3:35:16 [Note] WSREP: declaring 80d9486f-6f23-11e2-0800-89086bcae906 stable
130205 3:35:16 [Note] WSREP: declaring 87443555-6f23-11e2-0800-e752e5bdf5fa stable
130205 3:35:16 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,5) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
73c37263-6f23-11e2-0800-63de6fa27d76,
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
80d9486f-6f23-11e2-0800-89086bcae906,
87443555-6f23-11e2-0800-e752e5bdf5fa,
} joined {
} left {
} partitioned {
})
130205 3:35:16 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 5
130205 3:35:16 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 8791a3ac-6f23-11e2-0800-9c47eec92862
130205 3:35:16 [Note] WSREP: STATE EXCHANGE: sent state msg: 8791a3ac-6f23-11e2-0800-9c47eec92862
130205 3:35:16 [Note] WSREP: STATE EXCHANGE: got state msg: 8791a3ac-6f23-11e2-0800-9c47eec92862 from 0 (db001)
130205 3:35:16 [Note] WSREP: STATE EXCHANGE: got state msg: 8791a3ac-6f23-11e2-0800-9c47eec92862 from 1 (db004)
130205 3:35:16 [Note] WSREP: STATE EXCHANGE: got state msg: 8791a3ac-6f23-11e2-0800-9c47eec92862 from 3 (db005)
130205 3:35:16 [Note] WSREP: STATE EXCHANGE: got state msg: 8791a3ac-6f23-11e2-0800-9c47eec92862 from 2 (db002)
130205 3:35:16 [Note] WSREP: STATE EXCHANGE: got state msg: 8791a3ac-6f23-11e2-0800-9c47eec92862 from 4 (db003)
130205 3:35:16 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 4,
members = 3/5 (joined/total),
act_id = 32,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 3:35:16 [Note] WSREP: Flow-control interval: [36, 36]
130205 3:35:16 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:32, view# 5: Primary, number of nodes: 5, my index: 0, protocol version 2
130205 3:35:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:16 [Note] WSREP: Assign initial position for certification: 32, protocol version: 2
130205 3:35:16 [Note] WSREP: 3 (db005): State transfer from 0 (db001) complete.
130205 3:35:16 [Note] WSREP: Member 3 (db005) synced with group.
130205 3:35:16 [Warning] WSREP: Protocol violation. JOIN message sender 3 (db005) is not in state transfer (SYNCED). Message ignored.
130205 3:35:18 [Note] WSREP: Node 4 (db003) requested state transfer from '*any*'. Selected 0 (db001)(SYNCED) as donor.
130205 3:35:18 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 35)
130205 3:35:18 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:18 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.10.20.4:4444/rsync_sst' --auth 'root:galera' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --gtid '524bcc14-6f23-11e2-0800-d3c4b7c44389:35''
130205 3:35:18 [Note] WSREP: sst_donor_thread signaled with 0
130205 3:35:18 [Note] WSREP: Flushing tables for SST...
130205 3:35:18 [Note] WSREP: Provider paused at 524bcc14-6f23-11e2-0800-d3c4b7c44389:35
130205 3:35:18 [Note] WSREP: Tables flushed.
130205 3:35:25 [Note] WSREP: Provider resumed.
130205 3:35:26 [Note] WSREP: 0 (db001): State transfer to 4 (db003) complete.
130205 3:35:26 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 35)
130205 3:35:26 [Note] WSREP: Member 0 (db001) synced with group.
130205 3:35:26 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 35)
130205 3:35:26 [Note] WSREP: Synchronized with group, ready for connections
130205 3:35:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 3:35:29 [Note] WSREP: 4 (db003): State transfer from 0 (db001) complete.
130205 3:35:29 [Note] WSREP: Member 4 (db003) synced with group.
130205 22:42:56 [Note] WSREP: declaring 73c37263-6f23-11e2-0800-63de6fa27d76 stable
130205 22:42:56 [Note] WSREP: declaring 7a3461b1-6f23-11e2-0800-6d070e4c5cc6 stable
130205 22:42:56 [Note] WSREP: declaring 87443555-6f23-11e2-0800-e752e5bdf5fa stable
130205 22:42:56 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.6:4567
130205 22:42:56 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,6) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
73c37263-6f23-11e2-0800-63de6fa27d76,
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
87443555-6f23-11e2-0800-e752e5bdf5fa,
} joined {
} left {
} partitioned {
80d9486f-6f23-11e2-0800-89086bcae906,
})
130205 22:42:56 [Note] WSREP: forgetting 80d9486f-6f23-11e2-0800-89086bcae906 (tcp://10.10.20.6:4567)
130205 22:42:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4
130205 22:42:56 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') turning message relay requesting off
130205 22:42:56 [Note] WSREP: STATE_EXCHANGE: sent state UUID: db8bce56-6fc3-11e2-0800-8a9bcbfb8f67
130205 22:42:56 [Note] WSREP: declaring 73c37263-6f23-11e2-0800-63de6fa27d76 stable
130205 22:42:56 [Note] WSREP: declaring 7a3461b1-6f23-11e2-0800-6d070e4c5cc6 stable
130205 22:42:56 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.4:4567
130205 22:42:56 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,7) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
73c37263-6f23-11e2-0800-63de6fa27d76,
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
} joined {
} left {
} partitioned {
87443555-6f23-11e2-0800-e752e5bdf5fa,
})
130205 22:42:56 [Note] WSREP: forgetting 80d9486f-6f23-11e2-0800-89086bcae906 (tcp://10.10.20.6:4567)
130205 22:42:56 [Note] WSREP: forgetting 87443555-6f23-11e2-0800-e752e5bdf5fa (tcp://10.10.20.4:4567)
130205 22:42:56 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') turning message relay requesting off
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: sent state msg: db8bce56-6fc3-11e2-0800-8a9bcbfb8f67
130205 22:42:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
130205 22:42:56 [Note] WSREP: STATE_EXCHANGE: sent state UUID: db8d904b-6fc3-11e2-0800-be75cc8e637f
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: sent state msg: db8d904b-6fc3-11e2-0800-be75cc8e637f
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: got state msg: db8d904b-6fc3-11e2-0800-be75cc8e637f from 1 (db004)
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: got state msg: db8d904b-6fc3-11e2-0800-be75cc8e637f from 2 (db002)
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: got state msg: db8d904b-6fc3-11e2-0800-be75cc8e637f from 0 (db001)
130205 22:42:56 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 5,
members = 3/3 (joined/total),
act_id = 108490,
last_appl. = 107848,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 22:42:56 [Note] WSREP: Flow-control interval: [28, 28]
130205 22:42:56 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490, view# 6: Primary, number of nodes: 3, my index: 0, protocol version 2
130205 22:42:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 22:42:56 [Note] WSREP: Assign initial position for certification: 108490, protocol version: 2
130205 22:42:56 [Note] WSREP: declaring 7a3461b1-6f23-11e2-0800-6d070e4c5cc6 stable
130205 22:42:56 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.5:4567
130205 22:42:56 [Note] WSREP: view(view_id(PRIM,524b1158-6f23-11e2-0800-2b22295f4492,8) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
} joined {
} left {
} partitioned {
73c37263-6f23-11e2-0800-63de6fa27d76,
})
130205 22:42:56 [Note] WSREP: forgetting 73c37263-6f23-11e2-0800-63de6fa27d76 (tcp://10.10.20.5:4567)
130205 22:42:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
130205 22:42:56 [Note] WSREP: forgetting 80d9486f-6f23-11e2-0800-89086bcae906 (tcp://10.10.20.6:4567)
130205 22:42:56 [Note] WSREP: forgetting 87443555-6f23-11e2-0800-e752e5bdf5fa (tcp://10.10.20.4:4567)
130205 22:42:56 [Note] WSREP: (524b1158-6f23-11e2-0800-2b22295f4492, 'tcp://0.0.0.0:4567') turning message relay requesting off
130205 22:42:56 [Note] WSREP: STATE_EXCHANGE: sent state UUID: dbab540d-6fc3-11e2-0800-aae3cffbace5
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: sent state msg: dbab540d-6fc3-11e2-0800-aae3cffbace5
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: got state msg: dbab540d-6fc3-11e2-0800-aae3cffbace5 from 0 (db001)
130205 22:42:56 [Note] WSREP: STATE EXCHANGE: got state msg: dbab540d-6fc3-11e2-0800-aae3cffbace5 from 1 (db002)
130205 22:42:56 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 6,
members = 2/2 (joined/total),
act_id = 108490,
last_appl. = 107848,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 524bcc14-6f23-11e2-0800-d3c4b7c44389
130205 22:42:56 [Note] WSREP: Flow-control interval: [23, 23]
130205 22:42:56 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490, view# 7: Primary, number of nodes: 2, my index: 0, protocol version 2
130205 22:42:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 22:42:56 [Note] WSREP: Assign initial position for certification: 108490, protocol version: 2
130205 22:42:56 [Note] /usr/local/mysql/bin/mysqld: Normal shutdown
130205 22:42:56 [Note] WSREP: Stop replication
130205 22:42:56 [Note] WSREP: Closing send monitor...
130205 22:42:56 [Note] WSREP: Closed send monitor.
130205 22:42:57 [Note] WSREP: gcomm: terminating thread
130205 22:42:57 [Note] WSREP: gcomm: joining thread
130205 22:42:57 [Note] WSREP: gcomm: closing backend
130205 22:42:59 [Note] WSREP: view(view_id(NON_PRIM,524b1158-6f23-11e2-0800-2b22295f4492,8) memb {
524b1158-6f23-11e2-0800-2b22295f4492,
} joined {
} left {
} partitioned {
7a3461b1-6f23-11e2-0800-6d070e4c5cc6,
})
130205 22:42:59 [Note] WSREP: view((empty))
130205 22:42:59 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
130205 22:42:59 [Note] WSREP: gcomm: closed
130205 22:42:59 [Note] WSREP: Flow-control interval: [16, 16]
130205 22:42:59 [Note] WSREP: Received NON-PRIMARY.
130205 22:42:59 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 108490)
130205 22:42:59 [Note] WSREP: Received self-leave message.
130205 22:42:59 [Note] WSREP: Flow-control interval: [0, 0]
130205 22:42:59 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130205 22:42:59 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 108490)
130205 22:42:59 [Note] WSREP: RECV thread exiting 0: Success
130205 22:42:59 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130205 22:42:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 22:42:59 [Note] WSREP: recv_thread() joined.
130205 22:42:59 [Note] WSREP: Closing slave action queue.
130205 22:42:59 [Note] WSREP: New cluster view: global state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130205 22:42:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 22:42:59 [Note] WSREP: applier thread exiting (code:0)
130205 22:42:59 [Note] WSREP: applier thread exiting (code:5)
130205 22:42:59 [Note] WSREP: applier thread exiting (code:5)
130205 22:42:59 [Note] WSREP: applier thread exiting (code:5)
130205 22:43:01 [Note] WSREP: rollbacker thread exiting
130205 22:43:01 [Note] Event Scheduler: Purging the queue. 0 events
130205 22:43:01 [Note] WSREP: dtor state: CLOSED
130205 22:43:01 [Note] WSREP: apply mon: entered 0
130205 22:43:01 [Note] WSREP: apply mon: entered 0
130205 22:43:01 [Note] WSREP: mon: entered 108807 oooe fraction 0 oool fraction 0
130205 22:43:01 [Note] WSREP: cert index usage at exit 0
130205 22:43:01 [Note] WSREP: cert trx map usage at exit 0
130205 22:43:01 [Note] WSREP: deps set usage at exit 0
130205 22:43:01 [Note] WSREP: avg deps dist 0
130205 22:43:01 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130205 22:43:01 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 108490)
130205 22:43:01 [Note] WSREP: Flushing memory map to disk...
130205 22:43:01 InnoDB: Starting shutdown...
130205 22:43:02 InnoDB: Shutdown completed; log sequence number 4495319473
130205 22:43:02 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete
130205 22:43:03 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
130205 22:49:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130205 22:49:11 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.hDXBbksFA9
130205 22:49:14 mysqld_safe WSREP: Failed to recover position:
130205 22:49:14 [Note] WSREP: Read nil XID from storage engines, skipping position init
130205 22:49:14 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130205 22:49:14 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130205 22:49:14 [Note] WSREP: Found saved state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490
130205 22:49:14 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130205 22:49:14 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130205 22:49:14 [Note] WSREP: Assign initial position for certification: 108490, protocol version: -1
130205 22:49:14 InnoDB: The InnoDB memory heap is disabled
130205 22:49:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130205 22:49:14 InnoDB: Compressed tables use zlib 1.2.3
130205 22:49:14 InnoDB: Using Linux native AIO
130205 22:49:14 InnoDB: Initializing buffer pool, size = 523.0M
130205 22:49:14 InnoDB: Completed initialization of buffer pool
130205 22:49:14 InnoDB: highest supported file format is Barracuda.
130205 22:49:14 InnoDB: Waiting for the background threads to start
130205 22:49:15 InnoDB: 1.1.8 started; log sequence number 4495319473
130205 22:49:15 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130205 22:49:15 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130205 22:49:15 [Note] Server socket created on IP: '0.0.0.0'.
130205 22:49:15 [Warning] 'user' entry 'root@db001' ignored in --skip-name-resolve mode.
130205 22:49:15 [Warning] 'user' entry '@db001' ignored in --skip-name-resolve mode.
130205 22:49:15 [Warning] 'proxies_priv' entry '@ root@db001' ignored in --skip-name-resolve mode.
130205 22:49:15 [Note] Event Scheduler: Loaded 0 events
130205 22:49:15 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.28' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution, wsrep_23.7.r3829
130205 22:50:02 [Note] WSREP: Stop replication
130205 22:50:04 [Note] WSREP: killing local connection: 3
130205 22:50:04 [Note] WSREP: dtor state: CLOSED
130205 22:50:04 [Note] WSREP: apply mon: entered 0
130205 22:50:04 [Note] WSREP: apply mon: entered 0
130205 22:50:04 [Note] WSREP: apply mon: entered 0
130205 22:50:04 [Note] WSREP: cert index usage at exit 0
130205 22:50:04 [Note] WSREP: cert trx map usage at exit 0
130205 22:50:04 [Note] WSREP: deps set usage at exit 0
130205 22:50:04 [Note] WSREP: avg deps dist 0
130205 22:50:04 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130205 22:50:04 [Note] WSREP: Flushing memory map to disk...
130205 22:50:04 [Note] WSREP: Initial position: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490
130205 22:50:04 [Note] WSREP: wsrep_load(): loading provider library 'none'
130205 22:50:04 [ERROR] WSREP: Failed to get provider options
130205 22:50:04 [Note] WSREP: Stop replication
130205 22:50:06 [Note] WSREP: Initial position: 524bcc14-6f23-11e2-0800-d3c4b7c44389:108490
130205 22:50:06 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130205 22:50:06 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130205 22:50:06 [Note] WSREP: Found saved state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:-1
130205 22:50:06 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130205 22:50:06 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130205 22:50:06 [Note] WSREP: Assign initial position for certification: 108490, protocol version: -1
130205 22:51:44 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130205 22:51:44 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.T9AxPDm3Ss
130205 22:51:46 mysqld_safe WSREP: Failed to recover position:
130205 22:51:46 [Note] WSREP: Read nil XID from storage engines, skipping position init
130205 22:51:46 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130205 22:51:46 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130205 22:51:46 [Note] WSREP: Found saved state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:-1
130205 22:51:46 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130205 22:51:46 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130205 22:51:46 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130205 22:51:46 [Note] WSREP: wsrep_sst_grab()
130205 22:51:46 [Note] WSREP: Start replication
130205 22:51:46 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
130205 22:51:46 [Note] WSREP: protonet asio version 0
130205 22:51:46 [Note] WSREP: backend: asio
130205 22:51:46 [Note] WSREP: GMCast version 0
130205 22:51:46 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
130205 22:51:46 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
130205 22:51:46 [Note] WSREP: EVS version 0
130205 22:51:46 [Note] WSREP: PC version 0
130205 22:51:46 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.10.20.6:4567'
130205 22:51:46 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.3:4567 tcp://10.10.20.4:4567 tcp://10.10.20.5:4567
130205 22:51:47 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x28b6b50 after established 0x28ad410
130205 22:51:47 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x28bf6e0 after established 0x28c7f50
130205 22:51:47 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') turning message relay requesting off
130205 22:51:47 [Note] WSREP: declaring 046b4407-6fc5-11e2-0800-06e13b310150 stable
130205 22:51:47 [Note] WSREP: declaring 0a96b9a1-6fc5-11e2-0800-aabaabcfbbed stable
130205 22:51:47 [Note] WSREP: declaring 113af95f-6fc5-11e2-0800-fdef2e8eb4ef stable
130205 22:51:47 [Note] WSREP: declaring e2cbebe0-6fc4-11e2-0800-c32652869327 stable
130205 22:51:47 [Note] WSREP: view(view_id(PRIM,046b4407-6fc5-11e2-0800-06e13b310150,5) memb {
046b4407-6fc5-11e2-0800-06e13b310150,
0a96b9a1-6fc5-11e2-0800-aabaabcfbbed,
113af95f-6fc5-11e2-0800-fdef2e8eb4ef,
178c246c-6fc5-11e2-0800-278eb5fd5509,
e2cbebe0-6fc4-11e2-0800-c32652869327,
} joined {
} left {
} partitioned {
})
130205 22:51:47 [Note] WSREP: gcomm: connected
130205 22:51:47 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130205 22:51:47 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130205 22:51:47 [Note] WSREP: Opened channel 'my_wsrep_cluster'
130205 22:51:47 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 5
130205 22:51:47 [Note] WSREP: Waiting for SST to complete.
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130205 22:51:47 [Warning] WSREP: SYNC message from member 0 in non-primary configuration. Ignored.
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: sent state msg: 17c0f04e-6fc5-11e2-0800-a0bcc449a238
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: got state msg: 17c0f04e-6fc5-11e2-0800-a0bcc449a238 from 0 (db004)
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: got state msg: 17c0f04e-6fc5-11e2-0800-a0bcc449a238 from 1 (db002)
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: got state msg: 17c0f04e-6fc5-11e2-0800-a0bcc449a238 from 4 (db005)
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: got state msg: 17c0f04e-6fc5-11e2-0800-a0bcc449a238 from 2 (db003)
130205 22:51:47 [Note] WSREP: STATE EXCHANGE: got state msg: 17c0f04e-6fc5-11e2-0800-a0bcc449a238 from 3 (db001)
130205 22:51:47 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 4,
members = 3/5 (joined/total),
act_id = 0,
last_appl. = -1,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = e2cf7900-6fc4-11e2-0800-750d80c3391c
130205 22:51:47 [Note] WSREP: Flow-control interval: [36, 36]
130205 22:51:47 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
130205 22:51:47 [Note] WSREP: State transfer required:
Group state: e2cf7900-6fc4-11e2-0800-750d80c3391c:0
Local state: 524bcc14-6f23-11e2-0800-d3c4b7c44389:-1
130205 22:51:47 [Note] WSREP: New cluster view: global state: e2cf7900-6fc4-11e2-0800-750d80c3391c:0, view# 5: Primary, number of nodes: 5, my index: 3, protocol version 2
130205 22:51:47 [Warning] WSREP: Gap in state sequence. Need state transfer.
130205 22:51:47 [Note] WSREP: Member 0 (db004) synced with group.
130205 22:51:49 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.10.20.2' --auth 'root:galera' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '3132''
130205 22:51:49 [Note] WSREP: Prepared SST request: rsync|10.10.20.2:4444/rsync_sst
130205 22:51:49 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 22:51:49 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2
130205 22:51:49 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (524bcc14-6f23-11e2-0800-d3c4b7c44389) does not match group state UUID (e2cf7900-6fc4-11e2-0800-750d80c3391c): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.
130205 22:51:49 [Note] WSREP: Node 3 (db001) requested state transfer from '*any*'. Selected 0 (db004)(SYNCED) as donor.
130205 22:51:49 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
130205 22:51:49 [Note] WSREP: Requesting state transfer: success, donor: 0
130205 22:51:49 [Warning] WSREP: Could not find peer:
130205 22:51:49 [Note] WSREP: 2 (db003): State transfer from -1 (left the group) complete.
130205 22:51:49 [Note] WSREP: Member 2 (db003) synced with group.
130205 22:51:57 [Note] WSREP: 0 (db004): State transfer to 3 (db001) complete.
130205 22:51:57 [Note] WSREP: Member 0 (db004) synced with group.
WSREP_SST: [INFO] Joiner cleanup. (20130205 22:51:57.607)
done.
130205 22:51:58 [Note] WSREP: SST complete, seqno: 0
130205 22:51:58 InnoDB: The InnoDB memory heap is disabled
130205 22:51:58 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130205 22:51:58 InnoDB: Compressed tables use zlib 1.2.3
130205 22:51:58 InnoDB: Using Linux native AIO
130205 22:51:58 InnoDB: Initializing buffer pool, size = 523.0M
130205 22:51:58 InnoDB: Completed initialization of buffer pool
130205 22:51:58 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130205 22:51:58 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130205 22:51:58 InnoDB: Waiting for the background threads to start
130205 22:51:59 InnoDB: 1.1.8 started; log sequence number 4495321344
130205 22:51:59 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130205 22:51:59 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130205 22:51:59 [Note] Server socket created on IP: '0.0.0.0'.
130205 22:51:59 [Warning] 'user' entry 'root@db001' ignored in --skip-name-resolve mode.
130205 22:51:59 [Warning] 'user' entry '@db001' ignored in --skip-name-resolve mode.
130205 22:51:59 [Warning] 'proxies_priv' entry '@ root@db001' ignored in --skip-name-resolve mode.
130205 22:51:59 [Note] Event Scheduler: Loaded 0 events
130205 22:51:59 [Note] WSREP: Signalling provider to continue.
130205 22:51:59 [Note] WSREP: SST received: e2cf7900-6fc4-11e2-0800-750d80c3391c:0
130205 22:51:59 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.28' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution, wsrep_23.7.r3829
130205 22:51:59 [Note] WSREP: 3 (db001): State transfer from 0 (db004) complete.
130205 22:51:59 [Note] WSREP: Shifting JOINER -> JOINED (TO: 0)
130205 22:51:59 [Note] WSREP: Member 3 (db001) synced with group.
130205 22:51:59 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
130205 22:51:59 [Note] WSREP: Synchronized with group, ready for connections
130205 22:51:59 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130207 7:35:07 [ERROR] Slave SQL: Could not execute Delete_rows event on table prudhub.weather_forcast; Can't find record in 'weather_forcast', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log FIRST, end_log_pos 132, Error_code: 1032
130207 7:35:07 [Warning] WSREP: RBR event 2 Delete_rows apply warning: 137, 143022
130207 7:35:07 [ERROR] WSREP: Failed to apply trx: source: e2cbebe0-6fc4-11e2-0800-c32652869327 version: 2 local: 0 state: APPLYING flags: 1 conn_id: 76715 trx_id: 12606794 seqnos (l: 143166, g: 143022, s: 143020, d: 142843, ts: 1360208106991527091)
130207 7:35:07 [ERROR] WSREP: Failed to apply app buffer: ê Q, seqno: 143022, status: WSREP_FATAL
at galera/src/replicator_smm.cpp:apply_wscoll():49
at galera/src/replicator_smm.cpp:apply_trx_ws():120
130207 7:35:07 [ERROR] WSREP: Node consistency compromized, aborting...
130207 7:35:07 [Note] WSREP: Closing send monitor...
130207 7:35:07 [Note] WSREP: Closed send monitor.
130207 7:35:07 [Note] WSREP: gcomm: terminating thread
130207 7:35:07 [Note] WSREP: gcomm: joining thread
130207 7:35:07 [Note] WSREP: gcomm: closing backend
130207 7:35:09 [Note] WSREP: (178c246c-6fc5-11e2-0800-278eb5fd5509, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.4:4567
130207 7:35:09 [Note] WSREP: view(view_id(NON_PRIM,046b4407-6fc5-11e2-0800-06e13b310150,5) memb {
178c246c-6fc5-11e2-0800-278eb5fd5509,
} joined {
} left {
} partitioned {
046b4407-6fc5-11e2-0800-06e13b310150,
0a96b9a1-6fc5-11e2-0800-aabaabcfbbed,
113af95f-6fc5-11e2-0800-fdef2e8eb4ef,
e2cbebe0-6fc4-11e2-0800-c32652869327,
})
130207 7:35:09 [Note] WSREP: view((empty))
130207 7:35:09 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
130207 7:35:09 [Note] WSREP: gcomm: closed
130207 7:35:09 [Note] WSREP: Flow-control interval: [16, 16]
130207 7:35:09 [Note] WSREP: Received NON-PRIMARY.
130207 7:35:09 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 143023)
130207 7:35:09 [Note] WSREP: Received self-leave message.
130207 7:35:09 [Note] WSREP: Flow-control interval: [0, 0]
130207 7:35:09 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130207 7:35:09 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 143023)
130207 7:35:09 [Note] WSREP: RECV thread exiting 0: Success
130207 7:35:09 [Note] WSREP: recv_thread() joined.
130207 7:35:09 [Note] WSREP: Closing slave action queue.
130207 7:35:09 [Note] WSREP: /usr/local/mysql/bin/mysqld: Terminated.
Aborted
130207 07:35:09 mysqld_safe Number of processes running now: 0
130207 07:35:09 mysqld_safe WSREP: not restarting wsrep node automatically
130207 07:35:09 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
130208 06:39:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130208 06:39:35 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.NBLQpOIPPt
130208 06:39:37 mysqld_safe WSREP: Failed to recover position:
130208 6:39:37 [Note] WSREP: Read nil XID from storage engines, skipping position init
130208 6:39:37 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130208 6:39:37 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130208 6:39:37 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130208 6:39:37 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130208 6:39:37 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130208 6:39:37 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130208 6:39:37 InnoDB: The InnoDB memory heap is disabled
130208 6:39:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130208 6:39:37 InnoDB: Compressed tables use zlib 1.2.3
130208 6:39:37 InnoDB: Using Linux native AIO
130208 6:39:37 InnoDB: Initializing buffer pool, size = 523.0M
130208 6:39:37 InnoDB: Completed initialization of buffer pool
130208 6:39:37 InnoDB: highest supported file format is Barracuda.
130208 6:39:37 InnoDB: Waiting for the background threads to start
130208 6:39:38 InnoDB: 1.1.8 started; log sequence number 4783198709
130208 6:39:38 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130208 6:39:38 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130208 6:39:38 [Note] Server socket created on IP: '0.0.0.0'.
130208 6:39:38 [Warning] 'user' entry 'root@db001' ignored in --skip-name-resolve mode.
130208 6:39:38 [Warning] 'user' entry '@db001' ignored in --skip-name-resolve mode.
130208 6:39:38 [Warning] 'proxies_priv' entry '@ root@db001' ignored in --skip-name-resolve mode.
130208 6:39:38 [Note] Event Scheduler: Loaded 0 events
130208 6:39:38 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.28' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution, wsrep_23.7.r3829
130208 6:40:48 [Note] WSREP: Stop replication
130208 6:40:50 [Note] WSREP: killing local connection: 4
130208 6:40:50 [Note] WSREP: dtor state: CLOSED
130208 6:40:50 [Note] WSREP: apply mon: entered 0
130208 6:40:50 [Note] WSREP: apply mon: entered 0
130208 6:40:50 [Note] WSREP: apply mon: entered 0
130208 6:40:50 [Note] WSREP: cert index usage at exit 0
130208 6:40:50 [Note] WSREP: cert trx map usage at exit 0
130208 6:40:50 [Note] WSREP: deps set usage at exit 0
130208 6:40:50 [Note] WSREP: avg deps dist 0
130208 6:40:50 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130208 6:40:50 [Note] WSREP: Flushing memory map to disk...
130208 6:40:50 [Note] WSREP: Initial position: e2cf7900-6fc4-11e2-0800-750d80c3391c:143021
130208 6:40:50 [Note] WSREP: wsrep_load(): loading provider library 'none'
130208 6:40:50 [ERROR] WSREP: Failed to get provider options
130208 6:40:50 [Note] WSREP: Stop replication
130208 6:40:52 [Note] WSREP: Initial position: e2cf7900-6fc4-11e2-0800-750d80c3391c:143021
130208 6:40:52 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130208 6:40:52 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130208 6:40:52 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130208 6:40:52 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130208 6:40:52 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130208 6:40:53 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130208 06:42:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130208 06:42:04 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.IUQKqW7SqU
130208 06:42:06 mysqld_safe WSREP: Failed to recover position:
130208 6:42:06 [Note] WSREP: Read nil XID from storage engines, skipping position init
130208 6:42:06 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130208 6:42:06 [Note] WSREP: wsrep_load(): Galera 23.2.2(r137) by Codership Oy <info@codership.com> loaded succesfully.
130208 6:42:06 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130208 6:42:06 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130208 6:42:06 [Note] WSREP: Passing config to GCS: base_host = 10.10.20.2; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130208 6:42:07 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130208 6:42:07 [Note] WSREP: wsrep_sst_grab()
130208 6:42:07 [Note] WSREP: Start replication
130208 6:42:07 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
130208 6:42:07 [Note] WSREP: protonet asio version 0
130208 6:42:07 [Note] WSREP: backend: asio
130208 6:42:07 [Note] WSREP: GMCast version 0
130208 6:42:07 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
130208 6:42:07 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
130208 6:42:07 [Note] WSREP: EVS version 0
130208 6:42:07 [Note] WSREP: PC version 0
130208 6:42:07 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.10.20.5:4567'
130208 6:42:07 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.4:4567
130208 6:42:07 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') turning message relay requesting off
130208 6:42:07 [Note] WSREP: declaring 1aaaec28-7199-11e2-0800-262efcf60d52 stable
130208 6:42:07 [Note] WSREP: declaring f94a9ef8-7198-11e2-0800-e28dc995a543 stable
130208 6:42:07 [Note] WSREP: view(view_id(PRIM,1aaaec28-7199-11e2-0800-262efcf60d52,3) memb {
1aaaec28-7199-11e2-0800-262efcf60d52,
20e20a59-7199-11e2-0800-5e53db9ec0dd,
f94a9ef8-7198-11e2-0800-e28dc995a543,
} joined {
} left {
} partitioned {
})
130208 6:42:08 [Note] WSREP: gcomm: connected
130208 6:42:08 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130208 6:42:08 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130208 6:42:08 [Note] WSREP: Opened channel 'my_wsrep_cluster'
130208 6:42:08 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
130208 6:42:08 [Note] WSREP: Waiting for SST to complete.
130208 6:42:08 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130208 6:42:08 [Note] WSREP: STATE EXCHANGE: sent state msg: 212f5345-7199-11e2-0800-edcf921bd724
130208 6:42:08 [Note] WSREP: STATE EXCHANGE: got state msg: 212f5345-7199-11e2-0800-edcf921bd724 from 0 (db003)
130208 6:42:08 [Note] WSREP: STATE EXCHANGE: got state msg: 212f5345-7199-11e2-0800-edcf921bd724 from 2 (db004)
130208 6:42:08 [Note] WSREP: STATE EXCHANGE: got state msg: 212f5345-7199-11e2-0800-edcf921bd724 from 1 (db001)
130208 6:42:08 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 2,
members = 1/3 (joined/total),
act_id = 19,
last_appl. = -1,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = f94b58b5-7198-11e2-0800-8bb1ce3646de
130208 6:42:08 [Note] WSREP: Flow-control interval: [28, 28]
130208 6:42:08 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 19)
130208 6:42:08 [Note] WSREP: State transfer required:
Group state: f94b58b5-7198-11e2-0800-8bb1ce3646de:19
Local state: 00000000-0000-0000-0000-000000000000:-1
130208 6:42:08 [Note] WSREP: New cluster view: global state: f94b58b5-7198-11e2-0800-8bb1ce3646de:19, view# 3: Primary, number of nodes: 3, my index: 1, protocol version 2
130208 6:42:08 [Warning] WSREP: Gap in state sequence. Need state transfer.
130208 6:42:09 [Warning] WSREP: Could not find peer:
130208 6:42:09 [Note] WSREP: 0 (db003): State transfer from -1 (left the group) complete.
130208 6:42:09 [Note] WSREP: Member 0 (db003) synced with group.
130208 6:42:10 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.10.20.2' --auth 'root:galera' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '24506''
130208 6:42:10 [Note] WSREP: Prepared SST request: rsync|10.10.20.2:4444/rsync_sst
130208 6:42:10 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130208 6:42:10 [Note] WSREP: Assign initial position for certification: 19, protocol version: 2
130208 6:42:10 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (f94b58b5-7198-11e2-0800-8bb1ce3646de): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.
130208 6:42:10 [Note] WSREP: Node 1 (db001) requested state transfer from '*any*'. Selected 0 (db003)(SYNCED) as donor.
130208 6:42:10 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 19)
130208 6:42:10 [Note] WSREP: Requesting state transfer: success, donor: 0
130208 6:42:17 [Note] WSREP: 0 (db003): State transfer to 1 (db001) complete.
130208 6:42:17 [Note] WSREP: Member 0 (db003) synced with group.
130208 6:42:18 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.3:4567
130208 6:42:18 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7ffc98001ab0 after established 0x16ddb00
130208 6:42:18 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] Joiner cleanup. (20130208 06:42:18.354)
130208 6:42:18 [Note] WSREP: declaring 1aaaec28-7199-11e2-0800-262efcf60d52 stable
130208 6:42:18 [Note] WSREP: declaring 277260fd-7199-11e2-0800-543840209148 stable
130208 6:42:18 [Note] WSREP: declaring f94a9ef8-7198-11e2-0800-e28dc995a543 stable
130208 6:42:18 [Note] WSREP: view(view_id(PRIM,1aaaec28-7199-11e2-0800-262efcf60d52,4) memb {
1aaaec28-7199-11e2-0800-262efcf60d52,
20e20a59-7199-11e2-0800-5e53db9ec0dd,
277260fd-7199-11e2-0800-543840209148,
f94a9ef8-7198-11e2-0800-e28dc995a543,
} joined {
} left {
} partitioned {
})
130208 6:42:18 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 4
130208 6:42:18 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130208 6:42:18 [Note] WSREP: STATE EXCHANGE: sent state msg: 27bfb477-7199-11e2-0800-2bf5ed3377e1
130208 6:42:18 [Note] WSREP: STATE EXCHANGE: got state msg: 27bfb477-7199-11e2-0800-2bf5ed3377e1 from 0 (db003)
130208 6:42:18 [Note] WSREP: STATE EXCHANGE: got state msg: 27bfb477-7199-11e2-0800-2bf5ed3377e1 from 1 (db001)
130208 6:42:18 [Note] WSREP: STATE EXCHANGE: got state msg: 27bfb477-7199-11e2-0800-2bf5ed3377e1 from 3 (db004)
done.
130208 6:42:18 [Note] WSREP: SST complete, seqno: 19
130208 6:42:18 InnoDB: The InnoDB memory heap is disabled
130208 6:42:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130208 6:42:18 InnoDB: Compressed tables use zlib 1.2.3
130208 6:42:18 InnoDB: Using Linux native AIO
130208 6:42:18 InnoDB: Initializing buffer pool, size = 523.0M
130208 6:42:18 InnoDB: Completed initialization of buffer pool
130208 6:42:18 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130208 6:42:18 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130208 6:42:19 [Note] WSREP: STATE EXCHANGE: got state msg: 27bfb477-7199-11e2-0800-2bf5ed3377e1 from 2 (db002)
130208 6:42:19 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 3,
members = 2/4 (joined/total),
act_id = 19,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = f94b58b5-7198-11e2-0800-8bb1ce3646de
130208 6:42:19 [Note] WSREP: Flow-control interval: [32, 32]
130208 6:42:19 InnoDB: Waiting for the background threads to start
130208 6:42:20 InnoDB: 1.1.8 started; log sequence number 4783202737
130208 6:42:20 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130208 6:42:20 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130208 6:42:20 [Note] Server socket created on IP: '0.0.0.0'.
130208 6:42:20 [Warning] 'user' entry 'root@db001' ignored in --skip-name-resolve mode.
130208 6:42:20 [Warning] 'user' entry '@db001' ignored in --skip-name-resolve mode.
130208 6:42:20 [Warning] 'proxies_priv' entry '@ root@db001' ignored in --skip-name-resolve mode.
130208 6:42:20 [Note] Event Scheduler: Loaded 0 events
130208 6:42:20 [Note] WSREP: Signalling provider to continue.
130208 6:42:20 [Note] WSREP: SST received: f94b58b5-7198-11e2-0800-8bb1ce3646de:19
130208 6:42:20 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.5.28' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution, wsrep_23.7.r3829
130208 6:42:20 [Note] WSREP: New cluster view: global state: f94b58b5-7198-11e2-0800-8bb1ce3646de:19, view# 4: Primary, number of nodes: 4, my index: 1, protocol version 2
130208 6:42:20 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130208 6:42:20 [Note] WSREP: Assign initial position for certification: 19, protocol version: 2
130208 6:42:20 [Note] WSREP: 1 (db001): State transfer from 0 (db003) complete.
130208 6:42:20 [Note] WSREP: Shifting JOINER -> JOINED (TO: 19)
130208 6:42:20 [Note] WSREP: Member 1 (db001) synced with group.
130208 6:42:20 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 19)
130208 6:42:20 [Note] WSREP: Synchronized with group, ready for connections
130208 6:42:20 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130208 6:42:21 [Note] WSREP: Node 2 (db002) requested state transfer from '*any*'. Selected 0 (db003)(SYNCED) as donor.
130208 6:42:28 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.20.6:4567
130208 6:42:28 [Note] WSREP: 0 (db003): State transfer to 2 (db002) complete.
130208 6:42:28 [Note] WSREP: Member 0 (db003) synced with group.
130208 6:42:28 [Warning] WSREP: evs::proto(20e20a59-7199-11e2-0800-5e53db9ec0dd, GATHER, view_id(REG,1aaaec28-7199-11e2-0800-262efcf60d52,4))lu (31) <= safe_seq(31), can't recover message
130208 6:42:28 [Warning] WSREP: evs::proto(20e20a59-7199-11e2-0800-5e53db9ec0dd, GATHER, view_id(REG,1aaaec28-7199-11e2-0800-262efcf60d52,4))lu (31) <= safe_seq(31), can't recover message
130208 6:42:28 [Warning] WSREP: evs::proto(20e20a59-7199-11e2-0800-5e53db9ec0dd, GATHER, view_id(REG,1aaaec28-7199-11e2-0800-262efcf60d52,4))lu (31) <= safe_seq(31), can't recover message
130208 6:42:28 [Warning] WSREP: evs::proto(20e20a59-7199-11e2-0800-5e53db9ec0dd, GATHER, view_id(REG,1aaaec28-7199-11e2-0800-262efcf60d52,4))lu (31) <= safe_seq(31), can't recover message
130208 6:42:28 [Warning] WSREP: evs::proto(20e20a59-7199-11e2-0800-5e53db9ec0dd, GATHER, view_id(REG,1aaaec28-7199-11e2-0800-262efcf60d52,4))lu (31) <= safe_seq(31), can't recover message
130208 6:42:29 [Note] WSREP: (20e20a59-7199-11e2-0800-5e53db9ec0dd, 'tcp://0.0.0.0:4567') turning message relay requesting off
130208 6:42:29 [Note] WSREP: declaring 1aaaec28-7199-11e2-0800-262efcf60d52 stable
130208 6:42:29 [Note] WSREP: declaring 277260fd-7199-11e2-0800-543840209148 stable
130208 6:42:29 [Note] WSREP: declaring 2dbec9c3-7199-11e2-0800-43d05e4549ec stable
130208 6:42:29 [Note] WSREP: declaring f94a9ef8-7198-11e2-0800-e28dc995a543 stable
130208 6:42:29 [Note] WSREP: view(view_id(PRIM,1aaaec28-7199-11e2-0800-262efcf60d52,6) memb {
1aaaec28-7199-11e2-0800-262efcf60d52,
20e20a59-7199-11e2-0800-5e53db9ec0dd,
277260fd-7199-11e2-0800-543840209148,
2dbec9c3-7199-11e2-0800-43d05e4549ec,
f94a9ef8-7198-11e2-0800-e28dc995a543,
} joined {
} left {
} partitioned {
})
130208 6:42:29 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 5
130208 6:42:29 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
130208 6:42:29 [Note] WSREP: STATE EXCHANGE: sent state msg: 2e77596e-7199-11e2-0800-d52ad88d221f
130208 6:42:29 [Note] WSREP: STATE EXCHANGE: got state msg: 2e77596e-7199-11e2-0800-d52ad88d221f from 0 (db003)
130208 6:42:29 [Note] WSREP: STATE EXCHANGE: got state msg: 2e77596e-7199-11e2-0800-d52ad88d221f from 2 (db002)
130208 6:42:29 [Note] WSREP: STATE EXCHANGE: got state msg: 2e77596e-7199-11e2-0800-d52ad88d221f from 4 (db004)
130208 6:42:29 [Note] WSREP: STATE EXCHANGE: got state msg: 2e77596e-7199-11e2-0800-d52ad88d221f from 1 (db001)
130208 6:42:30 [Note] WSREP: STATE EXCHANGE: got state msg: 2e77596e-7199-11e2-0800-d52ad88d221f from 3 (db005)
130208 6:42:30 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 4,
members = 3/5 (joined/total),
act_id = 22,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = f94b58b5-7198-11e2-0800-8bb1ce3646de
130208 6:42:30 [Note] WSREP: Flow-control interval: [36, 36]
130208 6:42:30 [Note] WSREP: New cluster view: global state: f94b58b5-7198-11e2-0800-8bb1ce3646de:22, view# 5: Primary, number of nodes: 5, my index: 1, protocol version 2
130208 6:42:30 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130208 6:42:30 [Note] WSREP: Assign initial position for certification: 22, protocol version: 2
130208 6:42:30 [Note] WSREP: 2 (db002): State transfer from 0 (db003) complete.
130208 6:42:30 [Note] WSREP: Member 2 (db002) synced with group.
130208 6:42:32 [Note] WSREP: Node 3 (db005) requested state transfer from '*any*'. Selected 0 (db003)(SYNCED) as donor.
130208 6:42:39 [Note] WSREP: 0 (db003): State transfer to 3 (db005) complete.
130208 6:42:39 [Note] WSREP: Member 0 (db003) synced with group.
130208 6:42:41 [Note] WSREP: 3 (db005): State transfer from 0 (db003) complete.
130208 6:42:41 [Note] WSREP: Member 3 (db005) synced with group. -
Hi,
Not sure if this could be the issue but can you check if you have a primary key on this table, prudhub.weather_forcast?
If you don't have a good PK candidate then add an auto increment PK column but you have to follow a few extra steps, see this post http://www.codership.com/content/5-tips-migrating-your-mysql-server-galera-cluster
==
If you ever need to add an auto_increment primary key on a running Galera cluster, you need to turn off this feature by setting wsrep_auto_increment_control=OFF. While in that state, make sure you direct all writes to a single master. When done with the alter table, you probably want to turn it back on again. Note that this is only an issue when adding a primary key to non-empty tables. All other ALTER TABLE operations are fine
==
BR,
-alex
-
Hi,
I see it is failing on a DELETE.
Looking at, http://support.severalnines.com/entries/21692388-Limitations-in-Galera-Replication-for-MySQL , do you have tables without an explicitly defined primary key ?
Best regards
J
Please sign in to leave a comment.
Comments
10 comments