Docker2Caddy - An automatic Reverse Proxy for Docker containers

So you have a number of Docker containers running web services which you would like to expose to the outside? Well, you probably will at least have considered a reverse proxy already. Doing this manually for one, two or even five containers may be feasible, but everything above that will be a PITA for sure. At the FSFE we ran into the same issue with our own distributed container infrastructure at and crafted a neat solution that I would like to present to you in the next few minutes.

The result is Docker2Caddy that provides a workflow in which you can spin up new containers anytime (e.g. via a CI) and the reverse proxy will just do the rest for you magically.

The assumptions

Let’s assume you want to go with reverse proxies to make your web services accessible via ports 80 and 443. There are other possibilities, and in more complex environments there may be already integrated solutions, but for this article we’ll wade in a rather simple environment spun up with docker-compose¹.

Let’s also assume you care about security and go with a rootless installation of Docker. So the daemon will run as an unprivileged user. That’s possible but much more complex than the default rootful installation². Because of this, a few other solutions will not work, we’ll check that later.

Finally, each container shall at least have one separate domain assigned to it for which you obviously want to have a valid certificate, e.g. by Let’s Encrypt.

In the examples below, we have two containers running, each running a webserver listening to port 8080. The first container shall be available via first.com, the second via second.net. The latter shall also be available via www.second.net.

The problems

In the described scenario, there are a number of problem for automating the configuration of the reverse proxy in order to direct a domain to the correct container, starting with container discovery to IPv6 routing to handling offline containers.

The reverse proxy has to be able to discover the currently running containers and ideally monitor for changes regularly so that a newly created container with a new domain is reachable within a short time without manual intervention.

Before Docker2Caddy we have used nginx-proxy combined with acme-companion (formerly known as docker-letsencrypt-nginx-proxy-companion). These are Docker containers that query all containers connected to the bridge Docker network. For this to work, the containers have to run with environment variables indicating the desired domains and local ports that shall be proxied.

In a rootless Docker setup this finally reaches its limits although discovery still works. But already before that we did not like the fact that we had to connect containers to the bridge network upon creation and therefore lost a bit more isolation (which is dubious in Docker anyway).

Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups, IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt. But with rootless, the FSFE System Hackers team did not manage to get IPv6 working in containers to the degree that we needed. While IPv6 traffic reached the nginx-proxy, it was then treated as IPv4 traffic with the internal Docker IP address. That bits you ultimately if you limit requests based on IP addresses, e.g. for signups or payments. All traffic via IPv6 will be treated as the same internal IPv4 address, therefore triggering the limits regularly.

The easiest solution therefore is to use a reverse proxy running on the host system, not as a Docker container with its severe limitations. While the first intuition lead us to nginx, we decided to go with Caddy. The main advantages we saw are that a virtual host in Caddy is very simple to configure and that TLS certificates are generated and maintained automatically without extra dependencies like certbot.

In this setup, containers would need to open their webserver port to the host. This “public” port has to be unique per host, but the internal port can stay the same, e.g. port 1234 could be mapped to port 8080 inside the container. In Caddy you would then configure the domain first.org to forward to localhost:1234. A more or less identical second example container could then expose the port 5678 to the host, again listen on 8080 internally, and Caddy would redirect second.net and www.second.net to localhost:5678.

But how does Caddy know about the currently running containers and the ports via which they want to receive traffic? And how can we handle containers that are unavailable, for instance because they crashed or have been deleted for good? Docker2Caddy to the rescue!

The solution

I already concluded that Caddy is a suitable reverse proxy for the outlined use case. But in order to be care-free, the configuration has to be generated automatically. For this to work, I wrote a rather simple Python application called Docker2Caddy that is kept running in the background via a systemd service and writes proper logs that are also rotated nicely.

This is how it works internally: it queries (in a configurable interval) the Docker daemon for running containers. For each container it looks for specific labels (that are also configurable), by default proxy.host, proxy.host_alias and proxy.port. If one or multiple containers are found – in our case two – one Caddy configuration file per container is created. This is based on a freely configurable Jinja2 template. If the configuration changed, e.g. by a new host, Caddy will be reloaded and will create a TLS certificate if needed.

But what happens if a container is unavailable? In Docker2Caddy you can configure a grace period. Until this is reached, the Caddy configuration for the container in question is not removed but could forward to a local or remote error page. Only afterwards, the configuration is removed, and Caddy reloaded subsequently.

So, what makes Docker2Caddy special? I am biased but see a number of points:

1. Simplicity: fundamentally it’s a 188 pure lines of code Python script.
2. Configurability: albeit it’s simplicity, it’s easy to configure for various needs thanks to the templates and the support for rootless Docker setups.
3. Adaptability: it should be rather simple to make Docker2Caddy also work for Podman, or even use different reverse proxies. Feel free to extend it before I’ll do it myself someday ;)
4. Performance: while I did not perform before/after benchmarks, Caddy is blazingly fast and will surely perform better on the host than in a limited Docker container.

If you’re facing the same challenges in your setup, please feel free to try it out. Installation is quite simple and there’s even a minimal Ansible playbook. If you have feedback, I appreciate reading it via comments on Mastodon (see below), email, or, if you have an FSFE account, as a new issue or patch at the main repo.

---