What happens when a database node fails?

When a database node fails, the database connections that have been opened on that node will also fail. It is important that HAProxy does not redirect new connection requests to the failed node.

It is possible to have HAProxy check that a server is up, but just making a connection to the MySQL port (usually 3306) is not good enough. The instance might be up, but the underlying storage engine might not be working properly. There are specific checks to be done, depending on whether the clustering type is Galera or MySQL Cluster.

Therefore, we use custom scripts, managed through the xinetd daemon, to make HAProxy check the status of a MySQL server. If a MySQL server is not available, it is automatically excluded from the list of available servers. From the HAProxy administration interface it is also possible to manually take out a server from the connection pool, e.g., for maintenance or for testing and validating new configuration parameters or optimized queries, before bringing it back into production.