HomeDockerHow to Handle Secrets Like a Pro Using Gitops -

How to Handle Secrets Like a Pro Using Gitops –

One of the foundations of GitOps is the utilization of Git as the source of truth for the whole system. While most people are acquainted with the practice of storing the application source code in version control, GitOps dictates that you should also store all the other parts of your application, such as configuration, kubernetes manifests, db scripts, cluster definitions, etc.

But what about secrets? How can you use secrets with GitOps? This has been 1 of the most wellliked questions from teams that adopt GitOps.

The truth is that there is no single accepted practice on how secrets are managed with GitOps. If you already have a solid resolution in place such as HashiCorp vault, then it would make sense to use that even though technically it is against GitOps.

If you are starting from scratch on a new project, there are ways to store secrets in Git so that you can handle them using GitOps principles as well. It goes without saying that you should never ever commit raw secrets in Git.

All solutions that handle secrets using Git are storing them in an encrypted form. This means that you can get the best of both worlds. Secrets can be managed with GitOps, and they can also be positioned in a secure manner in any Git repository (even public repositories).

How Kubernetes secrets work

In the context of this article we will talk about 2 kinds of secrets, the constructed-in Kubernetes secrets (present in every Kubernetes cluster) and the sealed secrets as introduced by the Bitnami Sealed secrets controller.

Before we talk about sealed secrets, let’s talk about normal/plain secrets first. Kubernetes includes a native secret resource that you can use in your application. By default, these secrets are not encrypted in any way and the base64 encoding used should be never seen as a security feature. While there are ways to encrypt the Kubernetes secrets inside the cluster itself, we are more interested in encrypting them outdoors the cluster, so that we can store them externally in Git as required by 1 the founding principles of GitOps (everything stored in Git).

Using Kubernetes application secrets is straightforward. You can use the same mechanisms as configmaps, namely mounting them as files on your application or passing them as environment variables.

Plain Kubernetes secrets
Plain Kubernetes secrets

Sealed secrets are just an extension constructed on top of Kubernetes native secrets. This means that after the encryption/decryption takes place, all secrets function as plain Kubernetes secrets, and this is how your application should entry them. If you don’t like how plain Kubernetes secrets work, then you need to find an alternative security mechanism.

An example application with secrets

As a operating example, we will use a simple application found at https://github.com/codefresh-contrib/gitops-secrets-sample-app. This is a web application that reads several dummy secrets and displays them (without actually using them in any way).

We have chosen to make the application read the secrets as files (from /secrets/) instead of using environment variables. Here are the paths that are used:

It is necessary to note that the application is very simple. It only reads secrets from these paths. It doesn’t know anything about Kubernetes, secret sources, volume mounts, or anything else. You could run it on a Docker container (outdoors of Kubernetes), and if the correct paths have secret files, it would just work.

For illustration purposes, the application masses different kinds of secrets such as username/password, public/private key, and certificates. We will handle all of them in a identical way.

The Bitnami sealed secrets controller

The Bitnami Sealed secrets controller is a Kubernetes controller that you install on the cluster and performs a single task. It converts sealed secrets (that can be dedicated to Git) to plain secrets (that can be used in your application).

Installing the controller is straightforward:

Once installed, the controller creates 2 keys on its own:

  1. The private key is used for secret decryption. This key should stay inside the cluster, and you should never give it to anyone.
  2. The public key is used for secret encryption. This can (and will) be used outdoors the cluster, so it is ok to give to somebody else.

Once the controller is installed, you install and use your application in the standard way. You don’t need to change your application code or tamper with your Kubernetes manifests. If your application can use vanilla Kubernetes secrets, then it can work with sealed secrets as well.

Decryption of secrets
Decryption of secrets

It should be clear that the controller does not approach in direct contact with your application. It converts sealed secrets to Kubernetes secrets, and afterwards it is up to your application how to use them. The application does not even know that its secrets were initially encrypted in Git.

Encrypting your secrets

We have seen how the controller decrypts secrets. But how do we encrypt secrets in the first place? The controller comes with the associated kubeseal executable that is created for this purpose.

It is a single binary, so you can install it by copying it to your favorite directory (probably in your PATH variable)

If you have a Mac you can also get it via brew install kubeseal.

Kubeseal does the opposite operation from the controller. It takes an existing Kubernetes secret and encrypts it. Kubeseal requests the public key that was created during the installation process from the cluster and encrypts all secrets with that key.

This means that:

  • Kubeseal needs entry to the cluster in order to encrypt secrets. (It expects a kubeconfig like kubectl.)
  • Encrypted secrets can only be used in the cluster that was used for the encryption process.

The last point is very necessary as it means that all secrets are cluster specific. The namespace of the application is also used by default, so secrets are cluster and namespace specific.

If you want to use the same secret for different clusters, you need to encrypt it for each cluster individually.

To use kubeseal, just take any existing secret in yaml or json format and encrypt it:

This creates a SealedSecret which is a custom Kubernetes resource specific to the controller. This file is safe to commit in Git or store in another external system.
You can then apply the secret on the cluster

The secret is now part of the cluster and will be decrypted by the controller when an application needs it.

Here is the full diagram of encryption/decryption:

Full process
Full process

The full process is the following:

  1. You create a plain Kubernetes secret regionally. You should never commit this anyplace.
  2. You use kubeseal to encrypt the secret in a SealedSecret.
  3. You delete the original secret from your workstation and apply to the cluster the sealed secret.
  4. You can optionally commit the Sealed secret to Git.
  5. You deploy your application that expects normal Kuberentes secrets to function. (The application needs no modifications of any kind.)
  6. The controller decrypts the Sealed secrets and passes them to your application as plain secrets.
  7. The application works as usual.

Using sealed secrets with Codefresh GitOps

By using the Sealed Secrets controller, we can finally store all our secrets in Git (in an encrypted form) right alongside the application configuration.

In the example repository, you can gaze at the folder https://github.com/codefresh-contrib/gitops-secrets-sample-app/tree/main/safe-to-commit and it has all manifests of the application, including secrets.

You can simply point the Codefresh GitOps UI to this folder and deploy an application in a single step:

deploy GitOps application
deploy GitOps application

After the deployment is completed, you can see the components of the application in the GitOps dashboard:

GitOps dashboard
GitOps dashboard

And if you launch the application, you will see it is properly reading all secrets:

Application secrets
Application secrets

Adopting the Sealed Secrets controller in manufacturing

Secret rotation is a complex process that should not be taken flippantly. We have defined the fundamentals of Sealed Secrets in this article, but if you want to use the controller in manufacturing you need to read the documentation and take into account other aspects such as secret rotation and key handling.

For more details on using the controller with Codefresh GitOps, please see our example page.

New to Codefresh? Create your free account today!

Source

Most Popular