Lucas van Lieropfreelance software developer

Truly Immutable deployments with Docker or Kubernetes.

Just a random picture of a locked door

Target audience: developers who (are planning to) run containers in production.

TL;DR Deploy your containers with an immutable file system, its more secure and predictable!


Deploying should be boring a.k.a. predictable, what does that look like?

For me boring means it's predictable, which means the deployment process provides some guarantees, such as:

  • it's obvious which code has been deployed (e.g. can be related to a commit in version control)
  • deployed code behaves as expected (asserted by tests)
  • deployed code is fully compatible with the stack it runs on (asserted by tests).
  • all of the above do not change once deployed

For now I'd like to focus on the last item, I'll probably write an article on setting up an automated build and test pipeline later.

How predictable are deployment processes nowadays?

Deployment practices have come along way since 'the old days' where developers used to edit (hack) files directly on servers without any form of testing.

In today's container era all of above guarantees can be provided by CI pipelines that build, test and deploy code + infrastructure automatically from version control. This does not mean they ARE always provided, in fact many deployment processes guarantee:

  • it's obvious which code has been deployed
  • deployed code behaves as expected
  • deployed code is fully compatible with the stack it runs on.

But still don't guarantee:

  • all of the above do not change once deployed

How to make a deployment even more predictable with immutable containers

Let's take a look a some reasons that can cause changes in code or configuration after a deployment.

  • Hacking attempts that change the application or even download scripts
  • Bugs in the application that write files to an incorrect path
  • Frameworks that generate code for caching purposes (common practice with non compiled languages)
  • Automatic updates

Even if some of these changes are desired, they are not tested and therefore unpredictable. Any guarantees provided by the build/test/deployment process can no longer be trusted. If possible these should be part of the build process rather than happen after deployment. Especially updates since they would be gone when a new version is deployed anyway.

The good news is that changes can be prevented fairly easy by just starting the container with an immutable (read only) file system. In practice this means an application, its configuration and basically everything else that's in the container can NOT be changed. Let's see some examples on how to configure this in Docker & Kubernetes.

Example #1 Configuring immutable containers in Docker Compose (version 3) and Docker Swarm

With Docker all you have to do is mark the service as 'read only'.

Note that this example is for version 3 of the Docker Compose file format which can also be used for deploying to Docker Swarm. For the 'classic' Docker Compose version 2 format check the example below.

version: '3'

services:
  app:
    ...
    read_only: true

The example above assumes the application does not require any changes to the file system. In practice that might not always be possible because some processes require some files or directories to be writable. For example a temporary directory for storing file uploads might be necessary. Of course the final uploaded files should be stored in a persistent volume. Another example might be processes like Apache HTTP server that want to store their process id in a pid file.

This of course could be solved by mounting a writable file or directory from the local file system into the container. However since there's no need to persist these files they can be written to memory instead of disk.

Docker Compose supports temporary volumes in memory. In this example a writable directory named /var/run will be available in the container.
Note: it goes without saying that these kind of exclusions should be used as little as possible!

version: '3'

volumes:
  apache-run:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs

services:
  app:
    ...
    read_only: true
    volumes:
      - apache-run:/var/run

Example #2 Configuring immutable containers in the classic Docker Compose version 2 format

The Docker Compose version 2 file format is a bit different and has a separate tmpfs configuration that creates a temporary volume in memory.

Note you have to specify at least the uid (user id) that should own the volume (unless the process runs as root which is not recommended for security reasons!).

version: "2"

app:
    read_only: true
     tmpfs:
        - /var/run:uid={uid-of-the-process}

Example #3 Configuring immutable file systems in Kubernetes (v1.7+)

In Kubernetes an immutable file system can be configured as part of a deployment security context. Instead of Docker's tmpfs a volume of the type emptyDir must be configured.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
...
spec:
  ...
  template:
    ...
    spec:
      containers:
      - name: app
        ...
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
          - mountPath: /var/run
            name: apache-run
      volumes:
      - name: apache-run
        emptyDir: {}

Rounding up

  • Do all containers need to have an immutable file system? Of course this is up to you. I would recommend that at least containers where scripts/binaries can be executed should be immutable. So web and application servers and maybe even database servers.

  • Should development containers have an immutable file system too? Not necessarily since it's a mainly security measure for production situations. However it's better to have a similar setup across all environments so possible issues can be spotted before going to production.

FYI: The examples above are taken from a legacy WordPress project. Since I'm not a WordPress expert at all I intentionally run it in an immutable container to reduce the attack vector and to prevent unplanned automatic updates from happening. For some more context check the full Docker Compose config at the time of writing for both development and production


Thanks Jeroen, Annelies and Caroline for reviewing this post

Categories: software-development

Tags: docker, docker-compose, docker-festival, immutability, kubernetes, security