Is your Helm chart promotion process complicated and difficult to automate? Are rapidly increasing Helm chart versions making your head spin? Do you want you had a way to quickly and easily see the differences between deployments across all of your environments?
If you answered “yes” to any of these questions, then read on! My purpose for writing this article is to share a few of the techniques that I’ve seen make the biggest impact for Codefresh and our clients. That said, if what you’re looking for is a more comprehensive list of Helm best practices, then we’ve got you covered over in our documentation.
I’ve been lucky to see a lot of worthy ideas and approaches regarding Helm, so in order to choose the ones I think are most impactful, I’ll start by defining what I think success looks like.
My goals for simpler Helm deployments and promotions are to be able to:
- Easily find and gaze at any chart version
- Minimize my Helm chart version sprawl
- Easily compare the differences between deployments across all my environments (test, QA, staging, manufacturing, etc.)
- Easily automate promotions from 1 environment to another
Of course, your definition of “simpler” may differ from mine, which is why this isn’t really a “best practices” article. I’m just presenting some cool ideas, which you can take or leave relying on whether or not you find them helpful in your particular situation. Let’s take it 1 goal at a time, starting with…
Easily find any chart version
If you’re deploying charts immediately from a Git repository, then you know that while it’s easy to gaze at the HEAD of your repo/department to examine a currently-deployed chart, inspecting older versions of that chart is not so easy. I’m referring to the tedious task of sifting through 1’s Git history, 1 commit at a time, trying to find a specific chart version. UGH – this is never a fine time!
One way that I’ve seen folks attempt to make this scenario more workable is to put the Helm chart version in their commit message. I’ve found this method unreliable in practice, and looking for a specific chart version requires a lot of cryptic grepping and git log instructions. A identical approach is to tag 1’s Git commits with the Helm chart version. This is nonetheless rather cryptic to deal with, and it creates an additional dispute with tag sprawl.
Furthermore, there are other problems past just looking for chart versions. When deploying charts from Git to multiple environments, a common approach is to create a lengthy-lived department for each environment. In this scenario, when I promote a chart from 1 environment to another, I’m essentially copying the chart from 1 department to another, usually with a Pull Request. This means that I now have 2 copies of the same chart. Not only are we placing a lot of belief in my Pull Request to ensure that I’ve captured every single file from the chart version I’m promoting, but we’ve also lost our single source of truth for this version: placing it into 2 different lengthy-lived branches introduces the possibility of divergence.
In quick, managing chart versions in Git alone is painful. And it’s for this reason that Codefresh recommends…
Technique #1: Use a Helm Chart Repository
Storing your Helm charts as packages in a repo makes it way simpler to find whichever version(s) you’re looking for. For example, if I have a Helm chart called “buslog”, I can quickly search for all available versions:
Then I can pull the specific version that I want to examine:
helm pull myrepo/buslog —version 1.3.16
Don’t get me inaccurate – we should nonetheless develop and update our Charts in Git. I’m just saying that whenever we update the chart and increment its version, we should push it as a package to a Helm chart repository (via a CI pipeline) where it can be easily searched and used/reused for deployments. Furthermore, we are now free to place our charts wherever we like in Git – they don’t necessarily need to be tied to the Git repository we use for deployments any more.
Speaking of deployments, let’s take a moment to talk about 3 mechanisms that you can use to deploy your Helm chart repository-based packages from your CD.
- Helmfile tool – lets you deploy a helmfile.yaml list of charts, including the specific version and repository where each chart lives. You can also have 1 helmfile.yaml per environment/department, and since their charts all refer back to the same repository, our single source of truth is maintained. Simply call the helmfile binary from your CD pipeline to apply a helmfile.yaml file.
- Argo CD application spec – if you’re using Argo CD, then 1 option is to simply update your application specs to point to charts in a Helm chart repository instead of a Git repository. As with Helmfile, we can have Argo CD applications that deploy charts to lots of different environments, and since they all refer to the same Helm chart repo, we get our single source of truth. Here are the fields you’ll need to update in your application specs:
chart: buslog # the chart attribute replaces the path attribute
- Helm Umbrella Chart – This mechanism is really flexible. You can use it to deploy a single chart, or a group of related charts. You can use it in conjunction with a Helmfile or an Argo CD application to group related charts together, or you can just deploy it by itself from a basic CD pipeline with the helm command. Umbrella charts also have some advanced features which we’ll talk about later in this article. Here is an example:
apiVersion: v2 # v2 = Helm3
– name: buslog
And 1 final note on Helm chart repositories… One of the obstacles to adopting a repository in Helm 2 was that it typically required 1 to handle an additional application or VM in which to house this repository. Now with Helm 3, you can use any OCI-based container registry as your Helm chart repository. This includes most of the major Docker registries that you’re probably already using, like Amazon ECR, Google Artifact Registry, Azure ACR, Harbor, Artifactory, etc.
Let’s move on to our next goal…
Minimize Helm chart version sprawl
I know there are lots of different chart versioning strategies out there (some of which use the appVersion field, which I don’t really discuss here), so this goal may or may not resonate with you. That said, 1 very common versioning strategy is to increment a chart’s version every time it gets a new Docker image. Since new Docker images tend to get constructed for every application commit, then this could very well mean that you’re also producing a new chart version for every commit. And if we’re versioning on every commit, then the versions have lost not only their meaning, but also their usefulness.
One way to bring meaning back to chart versioning is to only version a chart when it’s structure changes (i.e. the Chart.yaml, values.yaml, or templates), and not for each time its image changes. “But wait,” you say, “when I change my image, won’t I at the very least need to change the chart’s values.yaml??” Well, not necessarily. And this brings us to our next technique…
Technique #2: Use Umbrella Charts
Helm charts have the ability to include other charts, referred to as subcharts, via their dependencies section. When a chart is created for the purpose of grouping together related subcharts/services, such as to compose a whole application or deployment, we call this an umbrella chart.
When using an umbrella chart, it is really easy to override values in its subcharts, such as the tag of a new image. We do this in the umbrella chart’s values.yaml, by placing the new value at
<name-of-subchart>.path.to.image.tag. For example, to override the
image.tag value in the buslog subchart our previous umbrella chart example, we would place this set of attributes into the umbrella chart’s
So the thought here is that you would have an umbrella chart in the Git repo/department that you use for deploying 1 or more services (usually a group of related services) to an environment. Then for most changes, your CI or promotion pipeline is just going to update 1 line in this umbrella chart’s values.yaml, and it will leave the subchart (buslog in this example) completely alone. Your subchart would only need to have its chart version incremented on those rare occasions when its non-image elements change (usually as part of a CI pipeline).
In this scenario, you may feel that it makes sense to update the version of the umbrella chart, since we are changing its values.yaml with the subchart’s new image. If so, then that’s OK – we’ve nonetheless massively lowered the amount of frivolous version updates in all of our subcharts, which is a huge improvement. And if not, then that’s OK too – I haven’t found very many situations in which I’ve been able to derive value from versioning umbrella charts that represent the deployment of a whole environment. One exception to that might be if you needed to deploy duplicate environments. In that case, versioning the umbrella chart (and maybe also storing it in a Helm chart repo) could be very useful.
At this point, I think a visible aid might help illustrate how this all comes together. Here is a construct from an example CI pipeline for a microservice, which processes both its Helm chart and its Docker image.
In the PREP stage, you can see that there is a Detect Changes step, which checks to see if any of the Helm chart’s files have changed. You can check for this with a git show command, identical to:
git show —name–only —oneline | grep “$DIRECTORY_OF_CHART”
&& export CHART_CHANGED=true || export CHART_CHANGED=untrue
If the chart’s files have not changed (as will be the case most of the time), you can see that the full PUSH CHART TO REPO stage is greyed out, indicating that it is skipped. Then you can see that the pipeline moves on to construct the new Docker image. Finally, in the PROMOTE TO TEST stage, it updates the umbrella chart for the Test environment with the new image tag. Afterwards, our CD process should pick up the updated umbrella chart, and deploy it to Test. You can view the whole pipeline YAML for this example in GitHub.
If you’re a fan of Helmfile, you may be considering at this point that you can achieve all the things I’ve just described with Helmfile as well. And you’d be right! Helmfile is a solid tool. However, I’m recommending umbrella charts here because of their excellent compatibility with other tools. Right now the GitOps movement is gaining a ton of momentum and industry focus/development. The major GitOps sync tools like Argo CD (to which Codefresh contributes) and Flux CD are not suitable with Helmfile. If you’re already an Argo CD user, then you can easily point your Argo CD applications at Git deployment repos containing umbrella charts. Your CI and promotion pipelines can concentrate their changes on the umbrella charts, and shouldn’t ever need to touch the Argo CD application specs.
Now on to the next goal…
Easily compare differences between environments
If we use an umbrella chart and Helm chart repo to specify the whole deployment for each of our environments, then we can shrink the number of files to compare per environment down to 2: Chart.yaml and values.yaml. Not bad, eh!? If we’re using a separate Git department for each of our deployment environments, then we can use
git diff to compare these 2 files between branches/environments. Now in my opinion, keeping environments separated into branches like this is nonetheless kind of a pain, which takes me to the next technique…
Technique #3: Use a deployment monorepo
Many of us have spent years working with various Git branching strategies that have inspired us to create a separate department for each deployment environment. So when we transitioned into doing CD for microservices, it seemed only natural to hold using branches for environments. But for most deployment repositories, separating environments into branches has very tiny value. Most of the time, it just provides restrictions, like needing to use
git diff for comparisons, as mentioned above. Instead, it would be a lot more straightforward if we used subdirectories for each environment, thus creating a sort of monorepo for deployments.
For example, here is what 1 of my Git deployment repos with subdirectories looks like:
You can see that I have directories for 4 environments: test, qa, stage, and prod. And because I don’t have to traverse different Git branches to see everything, the task of evaluating and manipulating these files is a lot simpler. For example, I can now gaze at all my environments in my Git system’s web UI, or compare them with any standard text editor. It took me about 10 seconds to compare my Test and QA environments from my trusty VS Code window (below), and no esoteric Git or VS Code knowledge was required!
As you can see, I have 1 chart version and 3 image tags that are different between my environments.
Or if I fancy diff, I can use that just as easily. I love options!
Using a Git deployment repo with subdirectories for environments also helps us accomplish the final goal of this article…
Easily automate promotions from 1 environment to another
Since all of our environments are easily readable / updateable without having to change branches, the process of promoting an image or chart from 1 environment to the next is super simple. We can do this with just 2 lines of shell code, using the excellent yq v4 tool. For example, to promote my buslog image from Test to QA:
# Paste image tag into the destination environment (qa)
yq eval –inplace .buslog.image.tag=”$IMAGE_TO_PROMOTE” qa/values.yaml
# Copy image tag from the source environment (test)
IMAGE_TO_PROMOTE=$(yq eval .buslog.image.tag test/values.yaml)
# Paste image tag into the destination environment (qa)
yq eval —inplace .buslog.image.tag=“$IMAGE_TO_PROMOTE“ qa/values.yaml
And since the promotion process is now simplified, it’s also really straightforward to automate it via a pipeline in your CI/CD tool of choice – hopefully Codefresh! In which case, you could simply use the helm-promote step from our Step Marketplace. While you’re at it, you might also want to gate your promotion pipeline with an approval step, or have it generate a Pull Request.
In conclusion, we looked at 3 different techniques to help us simply our Helm deployments:
- Use a Helm Chart Repository
- Use Umbrella Charts
- Use a Deployment Monorepo
Taken individually, they each provide value, and help us solve some of the goals presented at the beginning of this article. Taken together, however, I think these techniques can have a synergistic effect on 1 another, and can go a lengthy way towards simplifying 1’s Helm deployments. I hope you’ve been able to find at least 1 of them, if not all 3, useful in your situation!
New to Codefresh? Create your free account today!