Docker reverse proxy with HTTP/2, LetsEncrypt and on-demand container start support
Did you know Docksal can automatically start your existing stopped projects upon a web request? It’s an amazing feature and it is heavily used in CI/CD sandbox environments powered by Docksal. This keeps the server RAM usage low, as very few sandboxes are actually actively used at the same time. Inactive project instances (containers) are stoped after a certain period, but can be woken back up simply by visiting the environment’s URL. In Docksal, the docksal/vhost-proxy service is handling this workflow.
A basic reverse proxy for containers
Why do we even need a reverse proxy?
There is only one possible endpoint per IP:Port. If you have multiple containers exposing the same service (a web server in this case), they then either have to use unique IPs, unique ports or sit behind a reverse proxy. The proxy then listens on the primary endpoint and routes requests to many containers based on some logic (usually, the Host header).
Docksal’s vhost-proxy service was inspired by the excellent jwilder/nginx-proxy project. It uses Nginx as a reverse proxy server to route requests to multiple running containers on a host. docker-gen is used to interact with the Docker API. When docker-gen receives a container start/stop event from Docker, it regenerates a configuration template and reloads Nginx. Now the proxy knows how to route requests to newly launched containers and stopped ones are removed from the routing at the same time.
Optimizing sandbox density
Docker based stacks require a lot more resources, compared to plain old shared hosting approach (same stack for all guest). At the same time, individual sandbox environments don’t have to stay up for long. You push changes to Git, a sandbox environment is provisioned for your branch, tests are run. A few hours or days later someone may want to do a manual review or testing. In-between those events the sandbox is doing nothing except using server resources (mostly RAM). We wanted to improve the situation and maximize resource use on sandbox servers. So we taught our plain reverse proxy some new tricks.
First, we added a cron job, which checks web container logs and stops the entire sandbox stack if there was nothing happening for a certain period of time. That was relatively easy.
Now, how do we have those containers started back when needed again? One way would be to just have the sandbox be rebuilt, but that takes time and is lame. We need a way for this to happen automatically and on-demand.
The on-demand start magic
We used some Lua magic (powered by OpenResty) to teach Nginx to start containers upon a web request to an existing, but stopped sandbox environment.
nginx.conf
...
server {
listen 80;
server_name _; # This is just an invalid value which will never trigger on a real hostname.
location / {
rewrite_by_lua_file conf/lua/proxyctl.lua;
}
}
...Here, we pass requests for undefined hosts to a Lua script.
proxyctl.lua
...
os.execute("sudo proxyctl start \"" .. ngx.var.host .. "\"")
...Inside that Lua script, we trigger a custom control utility via os.execute. That utility takes the requests’s Host header (ngx.var.host) as an argument, looks up and then wakes up a matching project (using Docker CLI to interact with the Docker daemon). All of this happens real-time in in the context of the original HTTP request. The request is delayed for several seconds (the time it takes to wake up the containers), but then it continues as normal.
Matching is done based on container labels. All web containers have a label where their desired host names are stored.
Note, that active sandboxes have their own virtual host routing configuration (generated by docker-gen) and thus would never end up going through this workflow and experience a delay upon each request. No. While active, they are proxied by Nginx directly.
Calling a shell script upon a HTTP request is a dirty hack, but it has been doing a decent job for us so far. While hacks do work, they should be replaced with more solid solutions eventually.
We’ll want to replace os.execute call with pure Lua. Unfortunately, there is no official Docker API Lua library, so we may have to write one. There is this Luarocks module, which may be useful. We may also want to implement our own Lua module to interact with Docker via its HTTP API.
We could switch to a different solution altogether (not Nginx+Lua), so long as it covers the following high level requirements and is open source:
- be a reverse-proxy
- support scripting/plugins
- support HTTP/2
- support LetsEncrypt
- talk to Docker via API
The current docksal/vhost-proxy implementation handles the first 3 in the list above and LetsEncrypt integration is in a working POC phase (using https://github.com/GUI/lua-resty-auto-ssl).
Here’s a list of potential alternatives to consider:
- Traefik (ones Go plugin support is implemented)
- OptimalBits/redbird
- TODO: research more
Interested to learn more? Follow this issue on Github:
Have a need for similar use case or already have a solution that works for you? Please shared in comments!
And clap, clap, clap, … to help this post reach a larger audience on Medium.
