Adjust Open Files Limit

On some occasions, ClusterControl backup might fail or stall with the following errors:

InnoDB: Error number 24 means 'Too many open files'.

It indicates that you have reached the OS's maximum number of open file descriptors. The default value (1024) is too low for Xtrabackup. To increase open files limitation to a recommended value, which is 16384, you can use our script s9s_ulimit available in our Git repository.

Clone the Git repo in ClusterControl host (run this on the controller) and navigate into "cluster" directory :

$ git clone https://github.com/severalnines/s9s-admin.git
$ cd s9s-admin/cluster

Execute the script:

$ ./s9s_ulimit

Explanation

This script will automatically insert two lines into /etc/security/limits.conf, the soft limit and hard limit respectively. Example of output of the file if above command has executed:

# /etc/security/limits.conf
* soft nofile 16384
* hard nofile 16384

** The settings will be applied in each host in the cluster including ClusterControl host.

Please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session. The script will insert a command into /root/.bashrc which will run and set the specified limit every time the root user login. To activate the changes, you need to re-login the SSH or console session. 

Use following command to get the summary of current limits (as root):

$ ulimit -a

Other Parameters

This script is also able to accept two parameters; -t or --item and -n or --value which can be used to adjust any other items related to ulimit.

$ ./s9s_ulimit -t core -n 0

You may refer to this manual for supported items and values: http://ss64.com/bash/limits.conf.html

Notes

This script will not accept "unlimited" value as it could cause problem in some areas as reported in this bug report: https://bugzilla.redhat.com/show_bug.cgi?id=485955. Nevertheless, you can still change the value to "unlimited" using your own way.