Docker 1.13 introduced a set of features that allow us to centrally manage secrets and pass them only to services that need them. They provide a much-needed mechanism to provide information that should be hidden from anyone except designated services.
A secret (at least from Docker’s point of view) is a blog of data. A typical use case would be a certificate, SSH private keys, passwords, and so on. Secrets should stay secret meaning that they should not be stored unencrypted or transmitted over a network.
With all that being said, let’s see them in action and continue our discussion through practical examples.
Creating Secrets
Since a single node is more than enough to demonstrate Docker secrets, we’ll start by creating a one node Swarm cluster based on Docker Machines.
A note to Windows users
The recommendation is to run all the examples from Git Bash (installed through Docker Toolbox as well as Git). That way the commands you’ll see throughout the article will be same as those that should be executed on OS X or any Linux distribution.
docker-machine create \
-d virtualbox \
swarm
eval $(docker-machine env swarm)
docker swarm init \
--advertise-addr $(docker-machine ip swarm)We created a Docker Machine node called swarm and used it to initialize the cluster.
Now we can create a secret.
A note to Windows users
For mounts (a secret is a mount as well) used in the next command to work, you have to stop Git Bash from altering file system paths. Set this environment variable.
export MSYS_NO_PATHCONV=1
The format of the command that creates a secret is as follows (please do not run it).
docker secret create [OPTIONS] SECRET file|-
The secret create command expects a file that contains a secret. However, creating a file with unencrypted secret defies the purpose of having secrets in the first place. Everyone can read that file. We could, delete the file after pushing it to Docker but that would only create unnecessary steps. Instead, we’ll use - that will allow us to pipe standard output.
echo "I like candy" \
| docker secret create my_secret -The command we just executed created a secret called my_secret. That information was sent to the remote Docker Engine using TLS connection. If we had a bigger cluster with multiple managers, the secret would be replicated among all.
We can inspect the newly created secret.
docker secret inspect my_secret
The output is as follows.
[
{
"ID": "9iqwc8zb7xum7krgm183t4mym",
"Version": {
"Index": 11
},
"CreatedAt": "2017-02-20T23:00:48.983267019Z",
"UpdatedAt": "2017-02-20T23:00:48.983267019Z",
"Spec": {
"Name": "my_secret"
}
}
]The value of the secret is hidden. Even if a malicious person gains access to Docker Engine, the secret would still be unavailable. Truth be told, in such a case, our worries would be much greater that protection of a Docker secret but I’ll leave that discussion for some other time.
Now that we have encrypted the secret and stored in Swarm managers, we should explore ways to utilize it within our services.
Consuming Secrets
A new argument --secret was added to the docker service create command. If a secret is attached, it will be available as a file in the /run/secrets directory inside all the containers that form a service.
Let’s see it in action.
docker service create --name test \
--secret my_secret \
--restart-condition none \
alpine cat /run/secrets/my_secretWe created a service called test and attached the secret called my_secret. The service is based on alpine and will output the content of the secret. Since it is a one-shot command that will terminate quickly, we set --restart-condition to none. Otherwise, the service would terminate a moment after it’s created, Swarm would reschedule it, only to see it terminate again, and so on. We would enter a never-ending loop.
Let’s take a look at the logs.
docker logs $(docker container ps -qa)
The output is as follows.
I like candy
The secret is available as the /run/secrets/my_secret file inside the container.
Before we start discussing a more real-world example, let us remove the service and the secret we created.
docker service rm test docker secret rm my_secret
A Real-World Example of Using Secrets
Docker Flow Proxy project exposes statistics that should be reserved for internal use only. Therefore, it needs to be protected with a username and password. Before Docker v1.13, situations like that one would be handled by allowing users to specify username and password through environment variables. Docker Flow Proxy is no exception and, indeed, has the environment variables STATS_USER and STATS_PASS.
The command that would create the service with custom username and password would be as follows.
docker network create --driver overlay proxy
docker service create --name proxy \
-p 80:80 \
-p 443:443 \
-p 8080:8080 \
-e STATS_USER=my-user \
-e STATS_PASS=my-pass \
--network proxy \
-e MODE=swarm \
vfarcic/docker-flow-proxyWhile that would protect the statistics page from ordinary users, it would still leave it exposed to anyone capable of inspecting the service. A simple example is as follows.
docker service inspect proxy --pretty
The relevant part of the output is as follows.
... ContainerSpec: Image: vfarcic/docker-flow-proxy:latest@sha256:b1014afa9706413818903671086e484d98db669576b83727801637d1a3323910 Env: STATS_USER=my-user STATS_PASS=my-pass MODE=swarm ...
The same result that does not reveal confidential information could be accomplished with the commands that follow.
echo "secret-user" \
| docker secret create dfp_stats_user -
echo "secret-pass" \
| docker secret create dfp_stats_pass -
docker service update \
--secret-add dfp_stats_user \
--secret-add dfp_stats_pass \
proxyWe created two secrets (dfp_stats_user and dfp_stats_pass) and updated our service. From now on, those secrets would be available inside service containers as files /run/secrets/dfp_stats_user and /run/secrets/dfp_stats_pass. If a secret is named the same as the environment variable, is in lower case, and has the dpf_ prefix, it will be used instead.
If you inspect the container one more time, you’ll notice that there is no trace of the secrets.
We could stop here. After all, there’s not much more to be said for Docker secrets. However, we got used to Docker stacks and it would be great if secrets would work in the new YAML Compose format.
Before we move on, let’s remove the proxy service.
docker service rm proxy
Using Secrets With Docker Compose
True to the mission to have the same features available in all supported flavours, Docker introduced secrets in Compose YAML format version 3.1.
We’ll continue using Docker Flow Proxy to demonstrate how secrets work inside Compose files.
curl -o dfp.yml \
https://raw.githubusercontent.com/vfarcic/
docker-flow-stacks/master/proxy/docker-flow-proxy-secrets.ymlWe downloaded the docker-flow-proxy-secrets.yml stack from the vfarcic/docker-flow-stacks repository.
The relevant parts of the definition of the stack are as follows.
version: "3.1"
...
services:
proxy:
image: vfarcic/docker-flow-proxy:${TAG:-latest}
ports:
- 80:80
- 443:443
networks:
- proxy
environment:
- LISTENER_ADDRESS=swarm-listener
- MODE=swarm
secrets:
- dfp_stats_user
- dfp_stats_pass
deploy:
replicas: 3
...
secrets:
dfp_stats_user:
external: true
dfp_stats_pass:
external: trueThe version of the format is 3.1. The proxy service has the two secrets attached. Finally, there is a separate secrets section that defines the secrets as external entities. The alternative would be to specify secrets internally. An example would be as follows.
secrets:
dfp_stats_user:
external: true
dfp_stats_pass:
external: true
secrets:
dfp_stats_user:
file: ./dfp_stats_user.txt
dfp_stats_pass:
file: ./dfp_stats_pass.txtI prefer the first option that specifies secrets externally since that does not leave any trail. In some other cases, secrets might be used for non-secretive information (we’ll discuss it soon) and using internal secrets specified as files would probably be a better option.
Let’s run the stack and check whether it works.
docker stack deploy -c dfp.yml proxy
Statistics themselves are useless if there is no data so we’ll deploy another service that will be reconfigured in the proxy and start generating some stats.
curl -o go-demo.yml \
https://raw.githubusercontent.com/vfarcic/
go-demo/master/docker-compose-stack.yml
docker stack deploy -c go-demo.yml go-demoPlease wait a few moments until the services from the go-demo stack are running. You can check their status by executing docker stack ps go-demo. You might see go-demo_main replicas in the failed status. Do not panic. They will continue failing only until the go-demo_db is starts running.
Now we can, finally, confirm that the proxy is configured to use secrets for authentication.
curl -u secret-user:secret-pass \
"http://$(docker-machine ip swarm)/admin?stats;csv;norefresh"It works! With only a single additional step (docker service create), we made our system more secured.
Common Ways to Use Secrets
Until secrets were introduced, a common way to pass information to containers was through environment variables. While that will continue being the preferable way for non-confidential information, part of the setup should involve secrets as well. Both should be combined. The question is which method to choose and when.
The obvious use case for Docker secrets are secrets. That was obvious, wasn’t it. If there is a piece of information that should remain invisible to anyone but specific containers, it should be provided through Docker secrets. A commonly used pattern is to allow the same information to be specified as either environment variable and a secret. In case that both a set, secrets should take precedence. You already saw this pattern through Docker Flow Proxy. Every piece of information that can be specified through environment variables can be specified as a secret as well.
In some cases, you might not be able to modify code of your service and adapt it to use secrets. Maybe it’s not a question of ability but lack of desire to modify your code. If you fall into the latter case, I will, for now, restrain myself from explaining why code should be continuously refactored and imagine that you have a very good reason for it. In either case, the solution is usually to create a wrapper script that transforms secrets into whatever your service needs and then invoke the service. Put that script as CMD instruction in Dockerfile and you’re done. Secrets stay secrets and you don’t get fired from refactoring your code. To some this last sentence sounds silly but it’s not uncommon for companies to consider refactoring a waste of time.
What should be a secret? No one can truly answer that question for you since it differs from one organization to another. Some of the examples would be usernames and passwords, SSH keys, SSL certificates, and so on. If you don’t want others to know about it, make it a secret.
We should strive for immutability and do our best to run containers that are exactly the same no matter where they run. True immutability means that even the configuration is always the same across all environments. However, that is not always easy and is sometimes even impossible to accomplish. Such a situation could be a good candidate for Docker secrets. They do not necessarily have to be used only as means of specifying confidential information. We can use secrets as a way to provide information that differs from one cluster to another. In such a case, pieces of configuration that should differ from one environment to another (e.g. staging and production clusters) can be stored as secrets.
I am certain that there are quite a few other use cases I didn’t even think about. After all, secrets are a new feature (a few weeks old from the day of this writing).
I would like to get your feedback. Do you think that secrets would be helpful and if you do, what would be your use case? Please join slack.devops20toolkit.com/ Slack channel and let me know.
What Now?
Remove your Docker Machine VM and start applying secrets to your own Swarm cluster. There’s not much more to be said (for now).
docker-machine rm -f swarm
The DevOps 2.1 Toolkit: Docker Swarm
If you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.
You can get a copy from Amazon.com (and the other worldwide sites) or LeanPub. It is also available as The DevOps Toolkit Series bundle.
Give the book a try and let me know what you think.
| Reference: | Managing Secrets In Docker Swarm Clusters from our JCG partner Viktor Farcic at the Technology conversations blog. |
