Upgrading the cluster with apt-get update/upgrade or yum
This article describes the procedure for running apt-get / yum to upgrade a node manually (e.g if you want to update the OS) or if you use the Community version of ClusterControl.
General
- Update to the latest CMON version
- Start by with updating the database servers and move upwards the stack.
APT (debian/ubuntu)
For each node run the following commands, it may require 'sudo'
Manual operations on a NODE, such as apt-get update requires, the controller to be shutdown, so it does not try to do recovery at a time when you are in the middle of upgrading.
For each NODE in the cluster do:
- CONTROLLER: service cmon stop
- NODE: apt-get update && apt-get upgrade
(if you use PXC, then the upgrade will stop/start the MySQL Server at this stage, hence we shutdown the CONTROLLER first - NODE: update-rc.d -f mysql remove && update-rc.d -n mysql stop 20 0 1 6 .
This is really only an optimization. If not disabling auto-start, the mysql server will be started, and then ClusterControl will have to shut it down, in order to join it into the cluster.
Make sure only the stop part of the mysql server is installed so it can be shutdown gracefully on e.g server reboot.
We should leave recovery to ClusterControl as the /etc/init.d/mysql script has no notion of the MySQL Server being part of a cluster. - NODE: /sbin/reboot (only if needed)
- CONTROLLER: service cmon start
As soon as the reboot command is launched you can start the controller again. - Let the NODE recover.
YUM (centos/redhat)
For each node run the following commands, it may require 'sudo'
Manual operations on a NODE, such as apt-get update requires, the controller to be shutdown, so it does not try to do recovery at a time when you are in the middle of upgrading.
For each NODE in the cluster do:
- CONTROLLER: service cmon stop
-
NODE: yum update
(if you use PXC, then the upgrade will stop/start the MySQL Server at this stage, hence we shutdown the CONTROLLER first - NODE: /sbin/reboot (only if needed)
- CONTROLLER: service cmon start
As soon as the reboot command is launched you can start the controller again. - Let the NODE recover.
-
Johan,
Can you explain step #3 in the above sequences a little bit more? I'm not sure I understand completely what 'update-rc.d -f mysql remove && update-rc.d -n mysql stop 20 0 1 6' means. I've looked at the /etc/rc.d./mysql script and I do not see a 'remove' option so this has thrown me off a bit. I've ignored this step and have upgraded three nodes in a cluster, however each time I upgraded a node it required a recovery process. I simply waited until everything was "green" before I continued on.
Please advise if possible.
Thanks in advance.
-
Brian,
The instructions have been updated. The idea of the update-rc.d stuff (for Ubuntu, Centos only), is to shutdown the mysql server gracefully, on node reboot, but to not start it automatically (we let the CMON controller do that part). You don't have to do this step, but if you do, then the node will start to recover slight faster than if you don't.
But yes, otherwise, take one node at a time, let it be green then proceed to the next one.
When doing manual operations on a node that affects the mysql server (e.g upgrading it which has as a side effect that the mysql server will be shutdown), we do recommend that you shutdown the controller, so it does not try to recover it before you have finished upgrading the packages on the node that you need to have updated. The controller does not know that your are doing an upgrade. It just sees that the node has timed out and will start recovery, which may be or not may be what you want in that situation.
If shutting down the controller is needed or not really depends on the system, the database size, the load on the server when it is being upgraded etc etc, so we take a carte blanche and just recommend you shutdown the controller, do the upgrade, restart the controller, let the controller recover the node and then take the next one.
I hope this clarifies somewhat!
Good luck,
Johan
-
Hi Niko,
Kindly review following blog post for details:
http://www.severalnines.com/blog/patch-updates-and-new-version-upgrades-your-database-clusters
Regards,
Ashraf
-
Hi Olivia,
The steps are still relevant to date. However, instead of shutting down CMON service when performing database upgrade, you could also schedule maintenance mode for all nodes in the cluster so ClusterControl won't try to recover the nodes while under maintenance.
Then, proceed with upgrading data nodes, SQL/API nodes and then Management nodes. Do perform this exercise during the lowest peak hours.
Regards,
Ashraf -
Hi,
I would, in case of MySQL Cluster to do it in this order:
1) upgrade the management servers first (this way you know the config.ini still works with the new version of MySQL Cluster).
2) upgrade the data nodes in a rolling fashion (one at a time).
3) upgrade the mysql server / sql nodes in a rolling fashion
Also, take a backup before upgrading.
BR
johan
Please sign in to leave a comment.
Comments
10 comments