Docker Compose scale with Dynamic Configuration - Part 1

TL;DR;

Docker-compose allows scaling specific services from the composition. Assume we have a server and worker model. Consider the below compose file for the same

version: '3'

services:
  server:
    image: alpine
    command: sh -c "echo Launching server && exec tail -f /dev/null"
  worker:
    image: alpine
    command: sh -c "echo Launching worker >> /data/worker.log && exec tail -f /dev/null"
    volumes:
      - ./data:/data

Now if we do docker-compose up the server will echo Launching server and worker will print Launching worker to the /data/worker.log file.

Now let us run the same config with the scale option

$ docker-compose up -d --scale worker=5

Above command instructs docker-compose to launch 1 server container and 5 worker container. Now if we print our ./data/worker.log

$ cat data/worker.log
Launching worker
Launching worker
Launching worker
Launching worker
Launching worker

We get logs of all workers mixed into the same log file. The worker container has no idea that it is being scaled and can’t adjust its log filename or folder. Now if check the worker containers using docker ps

$ docker ps | grep worker
0520ea120ee0        alpine               "sh -c 'echo Launc..."   32 seconds ago      Up 31 seconds                                          scale1_worker_5
5d08bf33d88d        alpine               "sh -c 'echo Launc..."   32 seconds ago      Up 30 seconds                                          scale1_worker_4
34971d89ccdc        alpine               "sh -c 'echo Launc..."   32 seconds ago      Up 31 seconds                                          scale1_worker_3
b789c7806aac        alpine               "sh -c 'echo Launc..."   32 seconds ago      Up 31 seconds                                          scale1_worker_2
1468e77231b0        alpine               "sh -c 'echo Launc..."   32 seconds ago      Up 31 seconds                                          scale1_worker_1

Let is pick our worker scale1_worker_3 and inspect it.

$ docker inspect scale1_worker_3

....
....
    "Labels": {
        "com.docker.compose.config-hash": "6b42ee8ad919a3581c7342e91900ac3984fcf5b7c69274e63bc655461e9a0b57",
        "com.docker.compose.container-number": "3",
        "com.docker.compose.oneoff": "False",
        "com.docker.compose.project": "scale1",
        "com.docker.compose.service": "worker",
        "com.docker.compose.version": "1.15.0"
    }
....            
....

As we can see that compose will label the container using com.docker.compose.container-number which indicates a dynamic number for the worker.

If we can get this label inside docker container it would allow to use this number and change our logging or the log filename.

How do we get labels value inside a container? unfortunately docker does not provide anything of this sort.

Cloud providers like AWS provide a metadata url which a instance and hit query details about itself

$ curl http://169.254.169.254/latest/meta-data/local-hostname
ip-10-251-50-12.ec2.internal

So what we need is a docker metadata service which can return details about a container.

Building a Docker metadata service

Docker runs on /var/run/docker.sock by default and one can query Docker API directly on this socket

Now we can also get this using curl

$ curl -sgG --unix-socket /var/run/docker.sock 'http:::/containers/json' --data-urlencode 'filters={"name":["scale1_worker_3"]}' | jq  '.[].Labels."com.docker.compose.container-number"'
"3"

But to do this inside the docker container requires two things. One is to have curl inside, which is ok to have. Second is to share the /var/run/docker.sock.

Sharing the docker socket is ok if we are doing some testing or don’t care much about security. But if we need to put something in production then this is not a good way.

Building a Lua Nginx based Docker metdata server

Let’s build a service with /metadata/<label>/<labelname> as the format for the url that a container should hit. Unlike AWS metadata service we can’t determine which container the request generated from. So we need to also get the container id from the container.

So you url should be /metadata/<containerid>/<label>/<labelname>.

Now the simplest nginx config that can do that would be as below

user root;

events {
   worker_connections 8000;
}

http {
  server {
     listen 80;

     location / {
       deny all;
     }

     location ~* "/metadata/(?<cid>[a-zA-Z0-9]{20,})(/(?<command>[^/]+))?(/(?<subcommand>[^/]+))?" {

     }

   }
}

Now we need a way to query the docker socket but at the same time limit the querys to internal requests only. So we can add

    location /containers/json {
      internal;
      proxy_pass http://unix:/var/run/docker.sock:/containers/json;
    }

This exposes only /containers/json API endpoint to our nginx. Next we need some lua code inside our metadata location block to fetch the details about container and return the response filtered.

local cjson = require("cjson")

local cid = ngx.var.cid
local command = ngx.var.command
local subcommand = ngx.var.subcommand
ngx.log(ngx.ERR, "cid="..cid.. ", command=".. command .. ", subcommand=" .. subcommand)
local filters = cjson.encode({id = {cid}})
local res = ngx.location.capture("/containers/json", {args = {filters = filters}})

First we get the cid, command, subcommand from our location and then create filter parameter based on the container id. We hit our /containers/json endpoint which is internal, and get the response object.

Next we need to check if container exist or not, if we don’t get a response back then we just send a 404 back

if not res then
   ngx.status = 404
   return ngx.exit(ngx.HTTP_NOT_FOUND)
end

If we get a reponse then we want to load that string as a json variable

  local data = cjson.decode(res.body);
  ngx.header.content_type = "application/json"

Next we create a jsonpath query based on the labels

if command == "labels" then
   json_filter = "$..Labels"
   if subcommand ~= '' then
      json_filter = "$..['Labels']['".. subcommand  .. "']"
   end
end

Next we apply this json filter and send the response back

  local jp = require("jsonpath")
  local result = jp.value(data, json_filter)
  if type(result) == "string" then
     ngx.say(result)
  else
     ngx.say(cjson.encode(result))
  end

The jsonpath module need to be installed on openresty image, so we created a new Dockerfile

FROM openresty/openresty:alpine-fat
RUN apk update && apk add git && /usr/local/openresty/luajit/bin/luarocks install jsonpath

To make it easy for any docker container to reach our metadata service we will use a easy to remember IP 172.172.172.172. Now let us put everything into a compose file

version: '3'
services:
  metadata:
    build:
      context: .
    user: root
    volumes:
      - ./nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf
      - ./lua:/lua
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      metadataserver:
        ipv4_address: 172.172.172.172

networks:
  metadataserver:
    driver: bridge
    ipam:
      config:
      -
        subnet: 172.172.172.0/24

After doing docker-compose up -d we can start checking our metadata service

$ docker inspect -f '{{.Id}}' scale1_worker_3
34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754

$ curl -s "172.172.172.172/metadata/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754/labels" | jq
{
  "com.docker.compose.oneoff": "False",
  "com.docker.compose.config-hash": "6b42ee8ad919a3581c7342e91900ac3984fcf5b7c69274e63bc655461e9a0b57",
  "com.docker.compose.version": "1.15.0",
  "com.docker.compose.container-number": "3",
  "com.docker.compose.service": "worker",
  "com.docker.compose.project": "scale1"
}

$ curl -s "172.172.172.172/metadata/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754/labels/com.docker.compose.container-number"
3

So our metadata service is working great. Now the only issue is that the service requires the container id. This container id needs to be determined inside the container itself.

Determing container id from inside the container

This is where cgroup becomes our friend. Extracting container id is easy by exploring the cgroup of PID 1 or self

$ cat /proc/1/cgroup
11:freezer:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
10:pids:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
9:perf_event:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
8:net_cls,net_prio:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
7:blkio:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
6:cpu,cpuacct:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
5:hugetlb:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
4:memory:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
3:cpuset:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
2:devices:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754
1:name=systemd:/docker/34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754

We can easily filter out the container ID using

$ cat /proc/1/cgroup | grep cpuset | awk -F"/" '{print $NF}'
34971d89ccdc2f58e29f4387ef107714682d69abd06c1d6d1cbb5961ac8b4754

Metadata service can be download from below repo

https://github.com/tarunlalwani/docker-container-metdata-service

TARUN LALWANI

Building a Docker metadata service

Building a Lua Nginx based Docker metdata server

Determing container id from inside the container