Managing hundreds of cluster inside ClusterControl need some special adjustment in the controller node, database, and cmon service. Below are some configuration you might want to adjust for large scale deployment in ClusterControl
1. Tune the pool size in
/etc/cmon.conf, the default connection pool size for cmon is 64. You might want to increase the pool size in cmon.conf configurations depend on how much the cluster you have. You can increase 1,5x - 2x from the cluster size.
2. Tune the file descriptors in
/etc/sysctl.conf, as we see in the logs. There are some errors related to the file descriptors with message "too many open files" on the Prometheus. You can add/increase
3. Tune soft/hard limits in
/etc/security/limits.conf, please set the soft/hard limits too in limits.conf related to open files.
* soft nproc 1048576
* hard nproc 1048576
* soft nofile 1048576
* hard nofile 1048576
4. Tune database parameters in
my.cnf on CMON database configuration (kindly please adjust based on your hardware specification, eg: CPU, Memory, Disk). The following just an example:
innodb_buffer_pool_size = 4G -> 36G
innodb_buffer_pool_instances = 18
innodb_log_file_size = 512M -> 1G
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
innodb_read_io_threads = 16
innodb_write_io_threads = 16
max_connections = 51000 -> 5000
Some database parameters eg: buffer pool size, log buffer pool instance, io capacity, max connection). Also, use the UNIX socket instead of TCP/IP. Inside, cmon_*.cnf, and bootstrap.php, all MySQL hostname variables are changed from 127.0.0.1 to localhost (to make use of UNIX socket, instead of TCP/IP).