Shipping a load of containers requires a reliable infrastructure
So you have a number of Docker containers running web services which you would like to expose to the outside? Well, you probably will at least have considered a reverse proxy already. Doing this manually for one, two or even five containers may be feasible, but everything above that will be a PITA for sure. At the FSFE we ran into the same issue with our own distributed container infrastructure at and crafted a neat solution that I would like to present to you in the next few minutes.
The result is Docker2Caddy that provides a workflow in which you can spin up new containers anytime (e.g. via a CI) and the reverse proxy will just do the rest for you magically.
Let’s assume you want to go with reverse proxies to make your web services
accessible via ports 80 and 443. There are other possibilities, and in more
complex environments there may be already integrated solutions, but for this
article we’ll wade in a rather simple environment spun up with
Let’s also assume you care about security and go with a rootless installation of Docker. So the daemon will run as an unprivileged user. That’s possible but much more complex than the default rootful installation2. Because of this, a few other solutions will not work, we’ll check that later.
Finally, each container shall at least have one separate domain assigned to it for which you obviously want to have a valid certificate, e.g. by Let’s Encrypt.
In the examples below, we have two containers running, each running a webserver
listening to port
8080. The first container shall be available via
first.com, the second via
second.net. The latter shall also be available via
In the described scenario, there are a number of problem for automating the configuration of the reverse proxy in order to direct a domain to the correct container, starting with container discovery to IPv6 routing to handling offline containers.
The reverse proxy has to be able to discover the currently running containers and ideally monitor for changes regularly so that a newly created container with a new domain is reachable within a short time without manual intervention.
Before Docker2Caddy we have used
nginx-proxy combined with
acme-companion (formerly known
as docker-letsencrypt-nginx-proxy-companion). These are Docker containers that
query all containers connected to the
bridge Docker network. For this to work,
the containers have to run with environment variables indicating the desired
domains and local ports that shall be proxied.
In a rootless Docker setup this finally reaches its limits although discovery still works. But already before that we did not like the fact that we had to connect containers to the bridge network upon creation and therefore lost a bit more isolation (which is dubious in Docker anyway).
Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups,
IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt.
But with rootless, the FSFE System Hackers team did not manage to get IPv6
working in containers to the degree that we needed. While IPv6 traffic reached
nginx-proxy, it was then treated as IPv4 traffic with the internal Docker
IP address. That bits you ultimately if you limit requests based on IP
addresses, e.g. for signups or payments. All traffic via IPv6 will be treated
as the same internal IPv4 address, therefore triggering the limits regularly.
The easiest solution therefore is to use a reverse proxy running on the host
system, not as a Docker container with its severe limitations. While the first
intuition lead us to nginx, we decided to go with
Caddy. The main advantages we saw are that a virtual
host in Caddy is very simple to configure and that TLS certificates are
generated and maintained automatically without extra dependencies like
In this setup, containers would need to open their webserver port to the host.
This “public” port has to be unique per host, but the internal port can stay the
same, e.g. port
1234 could be mapped to port
8080 inside the container. In
Caddy you would then configure the domain
first.org to forward to
localhost:1234. A more or less identical second example container could then
expose the port
5678 to the host, again listen on
8080 internally, and Caddy
But how does Caddy know about the currently running containers and the ports via which they want to receive traffic? And how can we handle containers that are unavailable, for instance because they crashed or have been deleted for good? Docker2Caddy to the rescue!
I already concluded that Caddy is a suitable reverse proxy for the outlined use case. But in order to be care-free, the configuration has to be generated automatically. For this to work, I wrote a rather simple Python application called Docker2Caddy that is kept running in the background via a systemd service and writes proper logs that are also rotated nicely.
This is how it works internally: it queries (in a configurable interval) the
Docker daemon for running containers. For each container it looks for specific
labels (that are also configurable), by default
proxy.port. If one or multiple containers are found – in our case two –
one Caddy configuration file per container is created. This is based on a freely
configurable Jinja2 template. If the configuration changed, e.g. by a new host,
Caddy will be reloaded and will create a TLS certificate if needed.
But what happens if a container is unavailable? In Docker2Caddy you can configure a grace period. Until this is reached, the Caddy configuration for the container in question is not removed but could forward to a local or remote error page. Only afterwards, the configuration is removed, and Caddy reloaded subsequently.
So, what makes Docker2Caddy special? I am biased but see a number of points:
If you’re facing the same challenges in your setup, please feel free to try it out. Installation is quite simple and there’s even a minimal Ansible playbook. If you have feedback, I appreciate reading it via comments on Mastodon (see below), email, or, if you have an FSFE account, as a new issue or patch at the main repo.