HomeDockerUnlimited Preview Environments with Kubernetes Namespaces -

Unlimited Preview Environments with Kubernetes Namespaces –

In our huge series of Kubernetes anti-patterns, we briefly defined that static test environments are no longer desired if you are using Kubernetes. They are expensive, hard to maintain, and hard to clean up.

Static environments
Static environments

Instead, we suggested the adoption of temporary environments that are created on demand when a pull request is opened.

Dynamic environments
Dynamic environments

In this article, we will see the practical explanations on how to achieve limitless temporary environments using Kubernetes namespaces.

Choosing a naming strategy

Since the preview environments will be created and destroyed in a dynamic manner, you need to select a strategy for their names. While several solutions exist for naming, the 2 most common variations are:

  1. Using the name of the department as a context URL. This means example.com/feature1, example.com/feature2, example.com/feature3, and so on
  2. Using the name of the department as a host subdomain. This means feature1.example.com, feature2.example.com, feature3.example.com

The context-URL-based strategy is very easy to set up since it doesn’t need any special DNS settings (or TLS certs/wildcards). On the other hand, not all applications are designed to run with a different root context.
If you are sure that your application will not have issues with the context URL, then that strategy is the easiest to start.

The host-based naming strategy is much more robust, but it requires some configuration in your DNS provider to catch all subdomains and send them to the cluster that will hold all your preview namespaces.

In both cases, we also use an underlying Kubernetes namespace with the same name as the department. We take advantage of the fact that Git branches have unique names, making sure that there are no clashes between environment names or Kubernetes namespaces.

It is also very common for teams to create department names that represent specific issues (e.g. with JIRA). This makes it very easy to understand what developers are implementing for each feature environment.

For example, if a developer begins working on “issue 45 for billing”, then:

  1. A Git department is created with name issue-45-billing
  2. A temporary environment is deployed at Kubernetes namespace issue-45-billing
  3. The environment is uncovered at example.com/issue-45-billing or at issue-45-billing.example.com

Using a Kubernetes Ingress for traffic management

You can create a preview environment in a Kubernetes namespace using any of the available deployment mechanisms (e.g. Helm or Kustomize). In order to distinguish traffic between different pods, you also need to install a Kubernetes Ingress. An Ingress is a special Kubernetes resource responsible for routing requests inside the cluster.

There are several implementations available, and for our example, we will use Ambassador gateway. We will use Ambassador Edge stack 1.13.8 but the open source Emissary Ingress should work as well. Both host-based and context-based naming strategies are supported natively by the Ingress specification.

You can see how we set up our Ingress in the example application for the context-based naming strategy.

The most necessary property is the “path” property that tells the Ingress what URL context to honor when a request comes in the cluster (e.g. example.com/feature1, example.com/feature2, and so on).

We use Helm for making this path property configurable. This means we can pass a Helm value for each deployment that represents the URL of that preview environment.

Apart from the configurable Ingress, our example application is a vanilla Kubernetes application. You can see the full Helm chart in GitHub.

Creating an environment for a pull request

With the application manifests in place and an Ingress installed in the cluster, we are now ready to set up the workflow for the temporary environments.

First we need a pipeline that creates a temporary environment when a pull request is opened (or synced/updated).

Codefresh comes with a rich set of triggers that allow you to define exactly which events will launch the pipeline. Here is the trigger dialog:

catch all pull request events
catch all pull request events

We are only interested in the initial event of opening a pull request alongside with the “sync” event. In GitHub terms, a pull request is synced when somebody pushes another commit to an already open pull request. We want to update the environment in this case as well.

As for the pipeline itself, it is very simple with just 4 steps:

The 4 steps are:

  1. A clone step for checking out the source of the application
  2. A construct step to create a container image and also push it to Docker Hub
  3. Another clone step for checking out the Helm chart
  4. The Helm deploy step to deploy the application to a new namespace

The key point here is the CF_BRANCH_TAG_NORMALIZED variable. This variable is provided by Codefresh and represents the Git department that triggered this pipeline.

We use the variable in the deploy step in the namespace property as well as the ingress_path property.

As an example, if I create a pull request for a GitHub department named “demo” and run this pipeline:

  1. A namespace called demo will be created on the cluster
  2. Helm will deploy a version of the application on that namespace
  3. The Ingress of the cluster will be instructed to redirect all traffic at /demo/ to this deployment

Here is the result deployment in the browser:

Example deployment
Example deployment

And that’s it! Now each time a new pull request is opened, a new deployment will take place in the respective namespace.

Because we also catch the PR sync event in our Git trigger, we can also commit again on a department where a pull request is already open. Another deployment will take place with all our changes.

Automatic feedback on the pull request with the environment URL

Even if you have a naming convention for preview environments that is easy to remember, it would be very helpful for all members of your team to actually have a written history log of the creation of a temporary environment.

One of the most common patterns is adding the environment URL as a comment in the same pull request that created it.

Pull request comment
Pull request comment

In the example above, I am working on feature 2345 or a department called pr-2345. After I created the pull request, the environment was deployed to my Kubernetetes cluster, and a comment on the pull request has the exact URL.

This way, anybody who is responsible for reviewing the pull request has, in a single place, both the file changes and the temporary environment for checking how the application looks after the changes.

To achieve this pattern, you can use the Codefresh plugin for adding feedback to pull requests.
You can add the following snippet in your Codefresh pipeline:

With this pipeline step, we add a comment on a pull request. For the comment itself, we again use the predefined CF_BRANCH_TAG_NORMALIZED variable that provides the name of the pull request.

The plugin knows which pull request will be used for the comment by routinely fetching the pull request from the trigger of the pipeline. This is why we have no need to specify which pull request will be commented on.

Quality checks and smoke checks

Creating automated preview environments for each pull request is a capability that is also offered by several other products in the Kubernetes ecosystem. The huge power of Codefresh comes with the flexibility to add any verification steps before or after the creation of the environment.

For example, it would be wise to run unit and integration checks before an environment is deployed. After all, if unit checks fail, does it really make sense to create a temporary environment? The developer should instead fix the unit checks and then try to deploy again.

On the other hand, maybe you want to use the temporary environment for integration checks and possible security scans. This way when a pull request is created, the reviewer will have all the information desired at hand:

  • The code that was changed
  • How the application looks
  • If the new code introduces security issues or not
  • If the new code passes unit and integration checks.

Making all this information available in a single place results in a much faster review process.

It is also possible to add extra steps in the pipeline that check things after the environment is created. A very common example is operating a set of smoke checks on the newly created temporary environment. This gives you even higher confidence about the correctness of the changes.

You can also include other steps after the deployment such as posting a message to a Slack channel, sending an email, updating a dashboard, and so on.

Here is our final pipeline for creating a preview environment when a pull request is opened.

Pull Request preview pipeline
Pull Request preview pipeline

This pipeline has the following steps:

  1. A clone step to fetch the source code of the application
  2. A freestyle step that runs Maven for compilation and unit checks
  3. A construct step to create the docker image of the application
  4. A step that scans the source code for security issues with Snyk
  5. A step that scans the container image for security issues with trivy
  6. A step that runs integration checks by launching the app in a service container
  7. A step for Sonar analysis
  8. A step that clones a second Git repository that has the Helm chart of the app
  9. A step that deploys the source code to a new namespace.
  10. A step that provides a comment on the pull request with the URL of the temporary environment
  11. A step that runs smoke checks against the temporary test environment

Here is the whole pipeline definition:

Now when a preview environment is created, you have the guarantee that it passed the checks defined by your team (quality and security), leaving only the actual business logic as a review item.

This makes the process of reviewing pull requests as quickly as possible, since all the common checks are fully automated and reviewers can focus solely on how the application works.

Destroying a preview environment

Creating a preview environment can be a costly operation in a huge team. Reducing cloud costs is 1 of the biggest challenges when it comes to Kubernetes and cloud infrastructure.

You need to have a way to clean up preview environments when they are no longer used. Even though some teams have an automated job (e.g. via cron) to waste preview environments that are no longer used, the most cost efficient option is to delete a preview environment immediately after the respective pull request is closed.

We can setup a trigger for this event using the Git dialog of Codefresh:

Closed pull request events
Closed pull request events

For this pipeline, we capture the pull request closed events. It is not really necessary if the pull request was merged or not. Since it is closed, we assume that the respective preview environment is no longer desired.

The pipeline that deletes the environment is trivial; it has only 1 step:

destroy environment
waste environment

Here is the full definition of the delete pipeline:

In the pipeline, we uninstall the Helm application and also delete the respective namespace with the same name.

Adopting the mindset of preview environments

We hope you loved this tutorial for preview environments. Adopting Kubernetes is both a technical challenge and a paradigm shift, as several traditional practices are no longer desired. Stop using predefined test environments and switch to dynamic preview environments today!

For more details, see our documentation for preview environments.

Note that preview environments can affect your billing if they are not properly configured and managed. If you are operating an open source project with public infrastructure you need to take precautions to forestall abuse of this mechanism.

New to Codefresh? Create your free account today!

Go to the source

Most Popular