Data Sync Among Cluster Nodes After A Node Joining Back the Cluster
I have 3 Galera cluster nodes running MariaDB. After shutting down node 3, I started it up as a standalone MariaDB. I inserted a new table into node 3. Then I restarted node 3 as a cluster node.
I found that the new table only existed in node 3. For node 1 and 2, there is not such new table, i.e. node 3 has a different set of tables. But for all nodes, wsrep_local_state_comment = Synced and wsrep_cluster_status = Primary.
Is it normal when a standalone node joining back the cluster?
-
Official comment
Hi,
It all depends on how the data was inserted. Galera tracks writesets executed on nodes and it might detect that the joining node is more advanced than the nodes in the cluster. In such case it will not let the new node join the cluster. If you inserted the data outside of the Galera, as long as SST is not deemed required to rejoin the cluster, that data discrepancy will not be detected unless it will be detected by some of the queries executed on the Galera cluster. For example, if one node does not contain some of the rows and on the other node such row was modified, the node without that row won't be able to perform that operation and it will be thrown out of the cluster due to data mismatch.
In short, Galera has some internal sanity checks that help to detect data drift but they will not cover 100% cases and a situation like you described (which is close to the worst case scenario for Galera to detect) is expected to happen. Synced state says nothing about the data. It just means the node joined the cluster and it is not in any other state (donor, joining, joiner, etc).
Comment actions
Please sign in to leave a comment.
Comments
1 comment