Galera Cluster Keeps Rebuilding
I have been having an issue with Percona cluster rebuilding all the time. Once they are all rebuilt, one drops and the process starts all over again with all the nodes. We have F5 as our load balancer instead of HA Proxy. Can this be an issue?
-
Jason,
We don't think your problem is related to the load balancer - it's much more similar to the issues with workload, errors in SST process or InnoDB/Galera bugs. It's impossible to tell more without the access to MySQL's error logs, though. If you could share them (either here or internally, after opening a support ticket with us), we could try to give you more info about it.
-
MySQL132.55 KB11 May 2015 09:01:23
2015-05-07 21:05:31 33745 [Warning] 'user' entry 'syslogview@istwint01.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 21:05:31 33745 [Warning] 'db' entry 'gtocdr gtowrite@rudisbr.test' ignored in --skip-name-resolve mode.
2015-05-07 21:05:31 33745 [Warning] 'db' entry 'syslog syslogview@rudisbr.test' ignored in --skip-name-resolve mode.
2015-05-07 21:05:31 33745 [Warning] 'db' entry 'syslog syslogview@istwint01' ignored in --skip-name-resolve mode.
2015-05-07 21:05:31 33745 [Warning] 'db' entry 'syslog syslogview@istwint01.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 21:05:31 33745 [Warning] 'proxies_priv' entry '@ root@entmydnprd01.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 21:06:25 33745 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.70485S), skipping check
2015-05-07 21:08:05 33745 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.7161S), skipping check
2015-05-07 21:14:41 7f762dd25700 InnoDB: FTS Optimize Removing table bugs2/bugs_fulltext
2015-05-07 21:27:35 33745 [Warning] Aborted connection 7151 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 21:57:35 33745 [Warning] Aborted connection 7199 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 22:47:38 33745 [Warning] Aborted connection 7260 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 22:57:36 33745 [Warning] Aborted connection 7285 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 23:17:35 33745 [Warning] Aborted connection 7333 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 23:22:25 7f762dd25700 InnoDB: FTS Optimize Removing table bugs/bugs_fulltext
2015-05-07 23:24:16 33745 [Warning] 'user' entry 'syslogng@istnm.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'user' entry 'gtowrite@rudisbr.test' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'user' entry 'syslogview@rudisbr.test' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'user' entry 'syslogview@istwint01' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'user' entry 'syslogview@istwint01.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'db' entry 'gtocdr gtowrite@rudisbr.test' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'db' entry 'syslog syslogview@rudisbr.test' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'db' entry 'syslog syslogview@istwint01' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'db' entry 'syslog syslogview@istwint01.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 23:24:16 33745 [Warning] 'proxies_priv' entry '@ root@entmydnprd01.isllc.local' ignored in --skip-name-resolve mode.
2015-05-07 23:27:34 33745 [Warning] Aborted connection 7387 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 23:41:00 33745 [Warning] Aborted connection 7419 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 23:41:01 33745 [Warning] Aborted connection 7420 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 23:47:36 33745 [Warning] Aborted connection 7443 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-07 23:57:36 33745 [Warning] Aborted connection 7469 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 00:17:35 33745 [Warning] Aborted connection 7513 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 00:27:39 33745 [Warning] Aborted connection 7536 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 00:37:38 33745 [Warning] Aborted connection 7549 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 01:17:36 33745 [Warning] Aborted connection 7604 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 01:37:36 33745 [Warning] Aborted connection 7646 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 01:47:40 33745 [Warning] Aborted connection 7661 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 02:07:36 33745 [Warning] Aborted connection 7684 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 02:17:35 33745 [Warning] Aborted connection 7695 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 02:37:40 33745 [Warning] Aborted connection 7718 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 02:47:39 33745 [Warning] Aborted connection 7729 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 03:07:38 33745 [Warning] Aborted connection 7752 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 03:17:40 33745 [Warning] Aborted connection 7763 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 03:37:41 33745 [Warning] Aborted connection 7786 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 03:47:37 33745 [Warning] Aborted connection 7797 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 04:07:37 33745 [Warning] Aborted connection 7820 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 04:17:37 33745 [Warning] Aborted connection 7831 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 04:37:38 33745 [Warning] Aborted connection 7854 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 04:47:38 33745 [Warning] Aborted connection 7865 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 05:07:37 33745 [Warning] Aborted connection 7888 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 05:17:37 33745 [Warning] Aborted connection 7899 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 05:37:38 33745 [Warning] Aborted connection 7923 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 05:57:40 33745 [Warning] Aborted connection 7947 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 06:07:39 33745 [Warning] Aborted connection 7958 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 06:27:38 33745 [Warning] Aborted connection 7981 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 06:37:37 33745 [Warning] Aborted connection 7992 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 06:57:42 33745 [Warning] Aborted connection 8015 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 07:17:40 33745 [Warning] Aborted connection 8047 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 07:24:16 33745 [Warning] Aborted connection 7042 to db: 'mysql' user: 'root' host: 'localhost' (Got timeout reading communication packets)
2015-05-08 07:37:40 33745 [Warning] Aborted connection 8070 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 07:47:41 33745 [Warning] Aborted connection 8081 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 08:00:44 33745 [Warning] Aborted connection 7485 to db: 'unconnected' user: 'cmon' host: '10.213.161.90' (Got timeout reading communication packets)
2015-05-08 08:07:42 33745 [Warning] Aborted connection 8104 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 08:17:39 33745 [Warning] Aborted connection 8115 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 08:37:39 33745 [Warning] Aborted connection 8142 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 08:57:40 33745 [Warning] Aborted connection 8165 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 09:07:40 33745 [Warning] Aborted connection 8176 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 09:27:40 33745 [Warning] Aborted connection 8199 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 09:37:41 33745 [Warning] Aborted connection 8210 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 09:57:40 33745 [Warning] Aborted connection 8233 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 10:07:40 33745 [Warning] Aborted connection 8244 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 10:37:43 33745 [Warning] Aborted connection 8279 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 10:57:41 33745 [Warning] Aborted connection 8303 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 11:07:41 33745 [Warning] Aborted connection 8314 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 11:27:44 33745 [Warning] Aborted connection 8337 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 11:47:44 33745 [Warning] Aborted connection 8364 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 11:57:45 33745 [Warning] Aborted connection 8375 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 12:17:41 33745 [Warning] Aborted connection 8398 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 12:27:44 33745 [Warning] Aborted connection 8409 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 12:47:41 33745 [Warning] Aborted connection 8432 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 12:57:43 33745 [Warning] Aborted connection 8443 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 13:17:43 33745 [Warning] Aborted connection 8466 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 13:27:42 33745 [Warning] Aborted connection 8478 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 13:47:46 33745 [Warning] Aborted connection 8501 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 13:57:42 33745 [Warning] Aborted connection 8513 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 14:17:44 33745 [Warning] Aborted connection 8536 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 14:27:42 33745 [Warning] Aborted connection 8547 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 14:37:43 33745 [Warning] Aborted connection 8560 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 14:47:45 33745 [Warning] Aborted connection 8574 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 15:07:43 33745 [Warning] Aborted connection 8600 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 15:17:43 33745 [Warning] Aborted connection 8615 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 15:27:43 33745 [Warning] Aborted connection 8649 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 15:47:43 33745 [Warning] Aborted connection 8687 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 16:07:44 33745 [Warning] Aborted connection 8757 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 16:27:44 33745 [Warning] Aborted connection 8845 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 16:37:44 33745 [Warning] Aborted connection 8892 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 16:47:45 33745 [Warning] Aborted connection 8920 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 16:57:48 33745 [Warning] Aborted connection 8949 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 17:17:44 33745 [Warning] Aborted connection 8994 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 17:27:45 33745 [Warning] Aborted connection 9031 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 17:37:47 33745 [Warning] Aborted connection 9073 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 18:07:44 33745 [Warning] Aborted connection 9213 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 18:17:45 33745 [Warning] Aborted connection 9294 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 18:27:44 33745 [Warning] Aborted connection 9333 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 18:37:44 33745 [Warning] Aborted connection 9386 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 18:47:44 33745 [Warning] Aborted connection 9416 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 19:18:29 33745 [Warning] Aborted connection 9532 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 19:28:27 33745 [Warning] Aborted connection 9564 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 19:48:29 33745 [Warning] Aborted connection 9606 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 19:58:24 33745 [Warning] Aborted connection 9638 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 20:38:27 33745 [Warning] Aborted connection 9710 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 20:48:25 33745 [Warning] Aborted connection 9729 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 20:58:28 33745 [Warning] Aborted connection 9749 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 21:18:27 33745 [Warning] Aborted connection 9797 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 21:28:27 33745 [Warning] Aborted connection 9825 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 21:38:30 33745 [Warning] Aborted connection 9858 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 21:48:31 33745 [Warning] Aborted connection 9886 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-08 21:58:27 33745 [Warning] Aborted connection 9904 to db: 'istbops_wordpress' user: 'jmallory' host: '10.213.192.10' (Got an error reading communication packets)
2015-05-09 00:04:56 33745 [Note] WSREP: declaring 55d6c6cb at tcp://10.213.161.93:4567 stable
2015-05-09 00:04:56 33745 [Note] WSREP: forgetting 35368f1d (tcp://10.213.161.92:4567)
2015-05-09 00:04:56 33745 [Note] WSREP: Node 28d139c7 state prim
2015-05-09 00:04:56 33745 [Note] WSREP: view(view_id(PRIM,28d139c7,4) memb {
28d139c7,0
55d6c6cb,0
} joined {
} left {
} partitioned {
35368f1d,0
})
2015-05-09 00:04:56 33745 [Note] WSREP: save pc into disk
2015-05-09 00:04:56 33745 [Note] WSREP: forgetting 35368f1d (tcp://10.213.161.92:4567)
2015-05-09 00:04:56 33745 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2015-05-09 00:04:56 33745 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 06b039de-f5df-11e4-86df-e6a5a109d819
2015-05-09 00:04:56 33745 [Note] WSREP: STATE EXCHANGE: sent state msg: 06b039de-f5df-11e4-86df-e6a5a109d819
2015-05-09 00:04:56 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 06b039de-f5df-11e4-86df-e6a5a109d819 from 0 (10.213.161.91)
2015-05-09 00:04:56 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 06b039de-f5df-11e4-86df-e6a5a109d819 from 1 (10.213.161.93)
2015-05-09 00:04:56 33745 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 3,
members = 2/2 (joined/total),
act_id = 37056,
last_appl. = 36933,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 28d1a8db-ed2b-11e4-8563-f7de3faac1fe
2015-05-09 00:04:56 33745 [Note] WSREP: Flow-control interval: [23, 23]
2015-05-09 00:04:56 33745 [Note] WSREP: New cluster view: global state: 28d1a8db-ed2b-11e4-8563-f7de3faac1fe:37056, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3
2015-05-09 00:04:56 33745 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-05-09 00:04:56 33745 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-05-09 00:04:56 33745 [Note] WSREP: Service thread queue flushed.
2015-05-09 00:04:56 33745 [Note] WSREP: Assign initial position for certification: 37056, protocol version: 3
2015-05-09 00:04:56 33745 [Note] WSREP: Service thread queue flushed.
2015-05-09 00:05:01 33745 [Note] WSREP: cleaning up 35368f1d (tcp://10.213.161.92:4567)
2015-05-09 00:05:59 33745 [Note] WSREP: (28d139c7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2015-05-09 00:05:59 33745 [Note] WSREP: declaring 2c054e11 at tcp://10.213.161.92:4567 stable
2015-05-09 00:05:59 33745 [Note] WSREP: declaring 55d6c6cb at tcp://10.213.161.93:4567 stable
2015-05-09 00:05:59 33745 [Note] WSREP: Node 28d139c7 state prim
2015-05-09 00:05:59 33745 [Note] WSREP: view(view_id(PRIM,28d139c7,5) memb {
28d139c7,0
2c054e11,0
55d6c6cb,0
} joined {
} left {
} partitioned {
})
2015-05-09 00:05:59 33745 [Note] WSREP: save pc into disk
2015-05-09 00:05:59 33745 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
2015-05-09 00:05:59 33745 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2c3bd010-f5df-11e4-a8b4-5665ea7c4370
2015-05-09 00:05:59 33745 [Note] WSREP: STATE EXCHANGE: sent state msg: 2c3bd010-f5df-11e4-a8b4-5665ea7c4370
2015-05-09 00:05:59 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 2c3bd010-f5df-11e4-a8b4-5665ea7c4370 from 0 (10.213.161.91)
2015-05-09 00:05:59 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 2c3bd010-f5df-11e4-a8b4-5665ea7c4370 from 2 (10.213.161.93)
2015-05-09 00:05:59 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 2c3bd010-f5df-11e4-a8b4-5665ea7c4370 from 1 (10.213.161.92)
2015-05-09 00:05:59 33745 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 4,
members = 2/3 (joined/total),
act_id = 37116,
last_appl. = 36933,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 28d1a8db-ed2b-11e4-8563-f7de3faac1fe
2015-05-09 00:05:59 33745 [Note] WSREP: Flow-control interval: [28, 28]
2015-05-09 00:05:59 33745 [Note] WSREP: New cluster view: global state: 28d1a8db-ed2b-11e4-8563-f7de3faac1fe:37116, view# 5: Primary, number of nodes: 3, my index: 0, protocol version 3
2015-05-09 00:05:59 33745 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-05-09 00:05:59 33745 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-05-09 00:05:59 33745 [Note] WSREP: Service thread queue flushed.
2015-05-09 00:05:59 33745 [Note] WSREP: Assign initial position for certification: 37116, protocol version: 3
2015-05-09 00:05:59 33745 [Note] WSREP: Service thread queue flushed.
2015-05-09 00:06:01 33745 [Note] WSREP: Member 1.0 (10.213.161.92) requested state transfer from '*any*'. Selected 0.0 (10.213.161.91)(SYNCED) as donor.
2015-05-09 00:06:01 33745 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 37121)
2015-05-09 00:06:01 33745 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-05-09 00:06:01 33745 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.213.161.92:4444/xtrabackup_sst//1' --auth 'root:9bPOO03' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' '' --gtid '28d1a8db-ed2b-11e4-8563-f7de3faac1fe:37121''
2015-05-09 00:06:01 33745 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] Streaming with xbstream (20150509 00:06:02.205)
WSREP_SST: [INFO] Using socat as streamer (20150509 00:06:02.207)
WSREP_SST: [INFO] Using /tmp/tmp.vvHpBYQ7xg as xtrabackup temporary directory (20150509 00:06:02.254)
WSREP_SST: [INFO] Using /tmp/tmp.uxGmitACVn as innobackupex temporary directory (20150509 00:06:02.257)
WSREP_SST: [INFO] Streaming GTID file before SST (20150509 00:06:02.261)
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:10.213.161.92:4444; RC=( ${PIPESTATUS[@]} ) (20150509 00:06:02.273)
WSREP_SST: [INFO] Sleeping before data transfer for SST (20150509 00:06:02.391)
2015-05-09 00:06:02 33745 [Note] WSREP: (28d139c7, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] Streaming the backup to joiner at 10.213.161.92 4444 (20150509 00:06:12.394)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --no-version-check $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:10.213.161.92:4444; RC=( ${PIPESTATUS[@]} ) (20150509 00:06:12.396)
2015-05-09 00:11:30 33745 [Note] WSREP: 0.0 (10.213.161.91): State transfer to 1.0 (10.213.161.92) complete.
2015-05-09 00:11:30 33745 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 37171)
2015-05-09 00:11:30 33745 [Note] WSREP: Member 0.0 (10.213.161.91) synced with group.
2015-05-09 00:11:30 33745 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 37171)
2015-05-09 00:11:30 33745 [Note] WSREP: Synchronized with group, ready for connections
2015-05-09 00:11:30 33745 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20150509 00:11:30.039)
WSREP_SST: [INFO] Cleaning up temporary directories (20150509 00:11:30.042)
2015-05-09 00:11:51 33745 [Note] WSREP: 1.0 (10.213.161.92): State transfer from 0.0 (10.213.161.91) complete.
2015-05-09 00:11:51 33745 [Note] WSREP: Member 1.0 (10.213.161.92) synced with group.
2015-05-09 08:00:35 33745 [Warning] Aborted connection 10310 to db: 'unconnected' user: 'cmon' host: '10.213.161.90' (Got timeout reading communication packets)
2015-05-09 08:18:41 33745 [Warning] Aborted connection 10335 to db: 'mysql' user: 'root' host: 'localhost' (Got timeout reading communication packets)
2015-05-10 08:00:55 33745 [Warning] Aborted connection 11904 to db: 'unconnected' user: 'cmon' host: '10.213.161.90' (Got timeout reading communication packets)
2015-05-11 08:00:45 33745 [Warning] Aborted connection 13446 to db: 'unconnected' user: 'cmon' host: '10.213.161.90' (Got timeout reading communication packets)
2015-05-11 14:59:42 33745 [Note] WSREP: declaring 55d6c6cb at tcp://10.213.161.93:4567 stable
2015-05-11 14:59:42 33745 [Note] WSREP: forgetting 2c054e11 (tcp://10.213.161.92:4567)
2015-05-11 14:59:42 33745 [Note] WSREP: Node 28d139c7 state prim
2015-05-11 14:59:42 33745 [Note] WSREP: view(view_id(PRIM,28d139c7,6) memb {
28d139c7,0
55d6c6cb,0
} joined {
} left {
} partitioned {
2c054e11,0
})
2015-05-11 14:59:42 33745 [Note] WSREP: save pc into disk
2015-05-11 14:59:42 33745 [Note] WSREP: forgetting 2c054e11 (tcp://10.213.161.92:4567)
2015-05-11 14:59:42 33745 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2015-05-11 14:59:42 33745 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5aac96c5-f7ee-11e4-a229-9f6b5e3fe8ae
2015-05-11 14:59:42 33745 [Note] WSREP: STATE EXCHANGE: sent state msg: 5aac96c5-f7ee-11e4-a229-9f6b5e3fe8ae
2015-05-11 14:59:42 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 5aac96c5-f7ee-11e4-a229-9f6b5e3fe8ae from 0 (10.213.161.91)
2015-05-11 14:59:42 33745 [Note] WSREP: STATE EXCHANGE: got state msg: 5aac96c5-f7ee-11e4-a229-9f6b5e3fe8ae from 1 (10.213.161.93)
2015-05-11 14:59:42 33745 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 5,
members = 2/2 (joined/total),
act_id = 57553,
last_appl. = 57545,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 28d1a8db-ed2b-11e4-8563-f7de3faac1fe
2015-05-11 14:59:42 33745 [Note] WSREP: Flow-control interval: [23, 23]
2015-05-11 14:59:42 33745 [ERROR] Slave SQL: Could not execute Delete_rows event on table bugs2.tokens; Can't find record in 'tokens', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 593, Error_code: 1032
2015-05-11 14:59:42 33745 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 120, 57553
2015-05-11 14:59:42 33745 [Warning] WSREP: Failed to apply app buffer: seqno: 57553, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 2th time
2015-05-11 14:59:42 33745 [ERROR] Slave SQL: Could not execute Delete_rows event on table bugs2.tokens; Can't find record in 'tokens', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 593, Error_code: 1032
2015-05-11 14:59:42 33745 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 120, 57553
2015-05-11 14:59:42 33745 [Warning] WSREP: Failed to apply app buffer: seqno: 57553, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 3th time
2015-05-11 14:59:42 33745 [ERROR] Slave SQL: Could not execute Delete_rows event on table bugs2.tokens; Can't find record in 'tokens', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 593, Error_code: 1032
2015-05-11 14:59:42 33745 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 120, 57553
2015-05-11 14:59:42 33745 [Warning] WSREP: Failed to apply app buffer: seqno: 57553, status: 1
at galera/src/trx_handle.cpp:apply():351
Retrying 4th time
2015-05-11 14:59:42 33745 [ERROR] Slave SQL: Could not execute Delete_rows event on table bugs2.tokens; Can't find record in 'tokens', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 593, Error_code: 1032
2015-05-11 14:59:42 33745 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 120, 57553
2015-05-11 14:59:42 33745 [ERROR] WSREP: Failed to apply trx: source: 55d6c6cb-ed2b-11e4-ade6-9b50a08aa711 version: 3 local: 0 state: APPLYING flags: 1 conn_id: 14196 trx_id: 295040 seqnos (l: 58259, g: 57553, s: 57552, d: 57545, ts: 1448940270151059)
2015-05-11 14:59:42 33745 [ERROR] WSREP: Failed to apply trx 57553 4 times
2015-05-11 14:59:42 33745 [ERROR] WSREP: Node consistency compromized, aborting...
2015-05-11 14:59:42 33745 [Note] WSREP: Closing send monitor...
2015-05-11 14:59:42 33745 [Note] WSREP: Closed send monitor.
2015-05-11 14:59:42 33745 [Note] WSREP: gcomm: terminating thread
2015-05-11 14:59:42 33745 [Note] WSREP: gcomm: joining thread
2015-05-11 14:59:42 33745 [Note] WSREP: gcomm: closing backend
2015-05-11 14:59:43 33745 [Note] WSREP: view(view_id(NON_PRIM,28d139c7,6) memb {
28d139c7,0
} joined {
} left {
} partitioned {
55d6c6cb,0
})
2015-05-11 14:59:43 33745 [Note] WSREP: view((empty))
2015-05-11 14:59:43 33745 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-05-11 14:59:43 33745 [Note] WSREP: gcomm: closed
2015-05-11 14:59:43 33745 [Note] WSREP: Flow-control interval: [16, 16]
2015-05-11 14:59:43 33745 [Note] WSREP: Received NON-PRIMARY.
2015-05-11 14:59:43 33745 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 57553)
2015-05-11 14:59:43 33745 [Note] WSREP: Received self-leave message.
2015-05-11 14:59:43 33745 [Note] WSREP: Flow-control interval: [0, 0]
2015-05-11 14:59:43 33745 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2015-05-11 14:59:43 33745 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 57553)
2015-05-11 14:59:43 33745 [Note] WSREP: RECV thread exiting 0: Success
2015-05-11 14:59:43 33745 [Note] WSREP: recv_thread() joined.
2015-05-11 14:59:43 33745 [Note] WSREP: Closing replication queue.
2015-05-11 14:59:43 33745 [Note] WSREP: Closing slave action queue.
2015-05-11 14:59:43 33745 [Note] WSREP: /usr/sbin/mysqld: Terminated.
150511 14:59:43 mysqld_safe Number of processes running now: 0
150511 14:59:43 mysqld_safe WSREP: not restarting wsrep node automatically
150511 14:59:43 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
150511 15:00:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150511 15:00:33 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.JgroMd' --pid-file='/var/lib/mysql/entmydnprd01.isllc.local-recover.pid'
150511 15:00:39 mysqld_safe WSREP: Recovered position 28d1a8db-ed2b-11e4-8563-f7de3faac1fe:57552
Log of wsrep recovery (--wsrep-recover):
2015-05-11 15:00:33 50351 [Note] Plugin 'FEDERATED' is disabled.
2015-05-11 15:00:33 50351 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-05-11 15:00:33 50351 [Note] InnoDB: The InnoDB memory heap is disabled
2015-05-11 15:00:33 50351 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-05-11 15:00:33 50351 [Note] InnoDB: Memory barrier is not used
2015-05-11 15:00:33 50351 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-05-11 15:00:33 50351 [Note] InnoDB: Using Linux native AIO
2015-05-11 15:00:33 50351 [Note] InnoDB: Using CPU crc32 instructions
2015-05-11 15:00:33 50351 [Note] InnoDB: Initializing buffer pool, size = 8.7G
2015-05-11 15:00:34 50351 [Note] InnoDB: Completed initialization of buffer pool
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50353 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50354 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50356 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50355 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50357 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50358 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50359 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50360 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50361 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Warning] InnoDB: Setting thread 50362 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:34 50351 [Note] InnoDB: Highest supported file format is Barracuda.
2015-05-11 15:00:34 50351 [Note] InnoDB: Log scan progressed past the checkpoint lsn 28430605802
2015-05-11 15:00:34 50351 [Note] InnoDB: Database was not shutdown normally!
2015-05-11 15:00:34 50351 [Note] InnoDB: Starting crash recovery.
2015-05-11 15:00:34 50351 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-05-11 15:00:35 50351 [Note] InnoDB: Restoring possible half-written data pages
2015-05-11 15:00:35 50351 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 28430610549
2015-05-11 15:00:35 50351 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2015-05-11 15:00:36 50351 [Note] InnoDB: 128 rollback segment(s) are active.
2015-05-11 15:00:36 50351 [Warning] InnoDB: Setting thread 50368 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:36 50351 [Note] InnoDB: Waiting for purge to start
2015-05-11 15:00:36 50351 [Warning] InnoDB: Setting thread 50369 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:36 50351 [Warning] InnoDB: Setting thread 50370 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:36 50351 [Warning] InnoDB: Setting thread 50371 nice to 0 failed, current nice 10, errno 13
2015-05-11 15:00:36 50351 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 28430610549
2015-05-11 15:00:36 50351 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2015-05-11 15:00:36 50351 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2015-05-11 15:00:36 50351 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2015-05-11 15:00:36 50351 [Note] Server hostname (bind-address): '*'; port: 3306 -
Jason,
The most important bits are those:
2015-05-11 14:59:42 33745 [ERROR] Slave SQL: Could not execute Delete_rows event on table bugs2.tokens; Can't find record in 'tokens', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 593, Error_code: 1032
2015-05-11 14:59:42 33745 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 120, 57553
2015-05-11 14:59:42 33745 [ERROR] WSREP: Failed to apply trx: source: 55d6c6cb-ed2b-11e4-ade6-9b50a08aa711 version: 3 local: 0 state: APPLYING flags: 1 conn_id: 14196 trx_id: 295040 seqnos (l: 58259, g: 57553, s: 57552, d: 57545, ts: 1448940270151059)
2015-05-11 14:59:42 33745 [ERROR] WSREP: Failed to apply trx 57553 4 times
2015-05-11 14:59:42 33745 [ERROR] WSREP: Node consistency compromized, aborting...Galera detected a problem with replication - one of the transactions failed because InnoDB couldn't find a row to be deleted. It created a node inconsistency and triggered a safety check - Galera switched to SST to recover the node completely from the other member of the cluster.
It is hard to tell what exactly happened, most likely reason is the app (or user) executing some DML's locally on some of the nodes. It can be done when you use 'sql_log_bin=0' or 'wsrep_on=off' within a session scope or globally, on the whole node. While those options are useful sometimes, they may be dangerous too - in the worst case scenario it may lead to this kind of problems.
If the problem is indeed caused by such operations, you want to get rid of them - there's really no other solution.
Please sign in to leave a comment.
Comments
3 comments