Articles in this section

See more

Tuning ClusterControl for Large Deployment

Avatar
Agus Syafaat
  • Updated
Follow

Managing hundreds of cluster inside ClusterControl need some special adjustment in the controller node, database, and cmon service. Below are some configuration you might want to adjust for large scale deployment in ClusterControl

 

1. Tune the pool size in /etc/cmon.conf​, the default connection pool size for cmon is 64. You might want to increase the pool size in cmon.conf configurations depend on how much the cluster you have. You can increase 1,5x - 2x from the cluster size.

cmon_pool_size=128


2. Tune the file descriptors in /etc/sysctl.conf​, as we see in the logs. There are some errors related to the file descriptors with message "too many open files" on the Prometheus. You can add/increase fs.file-max parameter.

fs.file-max=1048576


3. Tune soft/hard limits in /etc/security/limits.conf​, please set the soft/hard limits too in limits.conf related to open files.

* soft    nproc        1048576
* hard    nproc        1048576
* soft    nofile       1048576
* hard    nofile       1048576


4. Tune database parameters in my.cnf on CMON database configuration (kindly please adjust based on your hardware specification, eg: CPU, Memory, Disk). The following just an example:

innodb_buffer_pool_size = 4G -> 36G
innodb_buffer_pool_instances = 18
innodb_log_file_size = 512M -> 1G
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
innodb_read_io_threads = 16
innodb_write_io_threads = 16
max_connections = 51000 -> 5000

Some database parameters eg: buffer pool size, log buffer pool instance, io capacity, max connection). Also, use the UNIX socket instead of TCP/IP. Inside, cmon_*.cnf, and bootstrap.php, all MySQL hostname variables are changed from 127.0.0.1 to localhost (to make use of UNIX socket, instead of TCP/IP). 

Related articles

Comments

4 comments

Sort by Date Votes
  • Avatar
    robert

    Thanks

    0
    Comment actions Permalink
  • Avatar
    Richard Andrews

    hello

    0
    Comment actions Permalink
  • Avatar
    Henry Ashford

    Really helpful breakdown—tweaking pool size and file limits made a noticeable difference in our setup. Big thanks for sharing this, Agus!

    1
    Comment actions Permalink
  • Avatar
    jerskin

    Solid checklist for scaling ClusterControl. Along with pool/file-descriptor/DB tuning, also set LimitNOFILE and LimitNPROC in the cmon systemd unit (ulimits in limits.conf don’t apply to services). Verify MySQL uses the UNIX socket end-to-end (permissions & mysql.default_socket), and lower max_connections to a realistic value to avoid OOM. Consider innodb_flush_neighbors=0, table_open_cache, and performance_schema=ON (with trimmed consumers) for better observability. Finally, monitor with Prometheus/Grafana and apply changes incrementally with a load test before rolling to prod. 

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk