HomeTechnologyZero Downtime Upgrades Made Easy with ClusterControl

Zero Downtime Upgrades Made Easy with ClusterControl

“Keep your database upgraded to the latest version – it’s for your safety” is something you may frequently hear as sound advice and best practice when it comes to database management. On the other hand, upgrading your database can be a time-consuming task. Even a minor version upgrade requires that you thoroughly test the upgrade in a staging environment before upgrading your manufacturing setup. So what’s the huge deal? If you’re only lagging behind 1 minor version, it shouldn’t matter, right? Well, it might not…until it does. And are you really prepared to take that kind of risk?

Earlier this year, a new probably risky vulnerability was identified in Galera Cluster (CVE-2021-27928). At first glance, we see that the severity was marked as high, and when we start digging into the issue further, it does indeed gaze severe. It appears that a SUPER user may execute any arbitrary code by changing wsrep_provider and wsrep_notify_cmd variables at the runtime. It allows the user to load the .so library and point towards a script that the server will execute. As you can imagine, this is not a fine situation. Sure, you need to have entry to the SUPER user, and you would need to have something available to execute on the database node, but the fact that Galera can be configured to execute arbitrary code as a ‘mysql’ user is bad enough on its own.

As usual, in cases such as these, the fixes have been created, and new versions of the software, unaffected by the vulnerability, have been pushed. This particular issue has been fixed in MariaDB 10.5.9, 10.4.18, 10.3.28, and 10.2.37, as well as Percona XtraDB Cluster 5.6.51-28.46, Percona XtraDB Cluster 5.7.33-31.49, and Percona XtraDB Cluster 8.0.22-13.1. All seems to be back to normal. Right?

Wrong. There are countless systems operating on manufacturing that have not yet been upgraded to the new, unaffected version. Severalnines support team is in touch with many database environments in the wild, and we are constantly working with prospects to help them migrate to an environment managed by ClusterControl. We see all kinds of MySQL (and not only MySQL) operating in outdated versions, sometimes even versions that have reached their End Of Life and are no longer getting security updates. That should not be the case, especially if you are a ClusterControl user.

ClusterControl comes with a set of features that will help you to stay up to date with all security fixes. Let’s take a gaze:

First of all, ClusterControl comes with Operational Reports, 1 of them being the Package Upgrade Report:

Zero Downtime Upgrades Made Easy with ClusterControl

Like all of ClusterControl’s operational reviews, the Package Upgrade Report can be scheduled to be executed regularly and then delivered via email. It will contain information about the package versions installed on the nodes and if there are any kind of upgrades that should be performed:

1634997764 606 Zero Downtime Upgrades Made Easy with ClusterControl

The Package Upgrade Report presents a list of packages that should be updated for all databases, loadbalancers, security fixes, and any other packages installed on the node. For all of the system packages, the resolution is to upgrade them using standard methods (apt, yum). When it comes to the databases and loadbalancers, ClusterControl comes with functionality that allows you to perform the minor version upgrade immediately from the UI.

Before we head there, let’s assume that the database has to be updated. You do not want to just proceed and run the upgrade blindly – it might probably cause problems for your application. It shouldn’t – minor versions do not break backwards compatibility (except when you use MySQL 8.0 – then yes, you may expect anything when going from 8.0.x to 8.0.x+1); however, there is always some risk involved. What you should do first is test the upgrade in a separate environment.

We have a simple MariaDB Galera cluster with ProxySQL and Keepalived:

1634997764 573 Zero Downtime Upgrades Made Easy with ClusterControl

We would like to construct a test cluster so that we can test the upgrade process. With ClusterControl, it is as easy as using Create Replica Cluster job:

1634997765 581 Zero Downtime Upgrades Made Easy with ClusterControl

We can get the fresh data from the existing cluster, or we can use the data from a backup.

1634997765 790 Zero Downtime Upgrades Made Easy with ClusterControl

We also have to pick a source node in the manufacturing cluster:

1634997766 636 Zero Downtime Upgrades Made Easy with ClusterControl

Then we have to go through a regular deployment wizard, choosing the version and vendor of the database, defining root password, and so on. We conclude by passing the nodes on which the cluster will be installed.

1634997767 149 Zero Downtime Upgrades Made Easy with ClusterControl

As a result, you will see a new cluster on the list with a clear mark that it is replicating off the manufacturing cluster. One thing worth mentioning, in the default setup, ClusterControl will use the latest versions of the packages to create the replica cluster. If you want to double-check just the queries, this is enough. If you want to go through the whole upgrade process, you would need to pin down older versions of the MySQL packages in order to install an old version (and then unpin them and test the upgrade).

One way or the other, after successful checks, you will ultimately want to perform the upgrade. ClusterControl can help you to accomplish this:

1634997767 27 Zero Downtime Upgrades Made Easy with ClusterControl

In Manage -> Upgrades, you will find a UI to perform the upgrade.

1634997768 666 Zero Downtime Upgrades Made Easy with ClusterControl

You can use “Check For New Packages” to refresh the database of available packages. We can also pick which nodes we want to upgrade and which services: 

1634997769 529 Zero Downtime Upgrades Made Easy with ClusterControl

Simply confirm and that’s it – ClusterControl will perform the upgrade and get you the latest version of the packages.

As you can see, ClusterControl makes keeping your databases up to date easy and straightforward. The only step that you should handle manually is the proper testing. Otherwise – everything else can be performed for you by ClusterControl. Interested in learning more about how ClusterControl can help you effectively handle your database? Try it free for 30 days.


Most Popular