Managing hundreds of cluster inside ClusterControl need some special adjustment in the controller node, database, and cmon service. Below are some configuration you might want to adjust for large scale deployment in ClusterControl
1. Tune the pool size in /etc/cmon.conf, the default connection pool size for cmon is 64. You might want to increase the pool size in cmon.conf configurations depend on how much the cluster you have. You can increase 1,5x - 2x from the cluster size.
cmon_pool_size=128
2. Tune the file descriptors in /etc/sysctl.conf, as we see in the logs. There are some errors related to the file descriptors with message "too many open files" on the Prometheus. You can add/increase fs.file-max parameter.
fs.file-max=1048576
3. Tune soft/hard limits in /etc/security/limits.conf, please set the soft/hard limits too in limits.conf related to open files.
* soft nproc 1048576* hard nproc 1048576* soft nofile 1048576* hard nofile 1048576
4. Tune database parameters in my.cnf on CMON database configuration (kindly please adjust based on your hardware specification, eg: CPU, Memory, Disk). The following just an example:
innodb_buffer_pool_size = 4G -> 36Ginnodb_buffer_pool_instances = 18innodb_log_file_size = 512M -> 1Ginnodb_io_capacity = 2000innodb_io_capacity_max = 4000innodb_read_io_threads = 16innodb_write_io_threads = 16max_connections = 51000 -> 5000
Some database parameters eg: buffer pool size, log buffer pool instance, io capacity, max connection). Also, use the UNIX socket instead of TCP/IP. Inside, cmon_*.cnf, and bootstrap.php, all MySQL hostname variables are changed from 127.0.0.1 to localhost (to make use of UNIX socket, instead of TCP/IP).
Comments
5 comments
Thanks
hello
Really helpful breakdown—tweaking pool size and file limits made a noticeable difference in our setup. Big thanks for sharing this, Agus!
Solid checklist for scaling ClusterControl. Along with pool/file-descriptor/DB tuning, also set
LimitNOFILEandLimitNPROCin the cmon systemd unit (ulimits inlimits.confdon’t apply to services). Verify MySQL uses the UNIX socket end-to-end (permissions &mysql.default_socket), and lowermax_connectionsto a realistic value to avoid OOM. Considerinnodb_flush_neighbors=0,table_open_cache, andperformance_schema=ON(with trimmed consumers) for better observability. Finally, monitor with Prometheus/Grafana and apply changes incrementally with a load test before rolling to prod.For **large ClusterControl deployments**, performance can be improved by tuning database monitoring, automation settings, resource usage, and infrastructure. Key areas include:
* **Increase Server Resources:** Provide enough CPU, RAM, and disk IOPS for ClusterControl and managed database nodes.
* **Tune Monitoring:** Reduce monitoring frequency and disable unnecessary metrics to lower overhead.
* **Optimize Database Settings:** Adjust MySQL/PostgreSQL/MongoDB parameters based on workload.
* **Use Separate Monitoring Nodes:** For very large clusters, run ClusterControl on a dedicated server.
* **Manage Retention Data:** Reduce historical data retention to control database growth.
* **Enable Load Balancing:** Use proper load balancers for high traffic environments.
* **Optimize Backups:** Schedule backups during low-traffic periods and use incremental options.
* **Review Alerts:** Keep only important alerts to avoid unnecessary processing.
* **Upgrade ClusterControl:** Use the latest stable version for better scalability and fixes.
Proper tuning helps ClusterControl handle larger database clusters with better speed, reliability, and monitoring performance.
Please sign in to leave a comment.