In this article, we’ll discuss a way to forward logs from containers created as Docker Swarm services inside our clusters. We’ll use the ELK stack. They’ll be forwarded from containers to LogStash and, from there, to ElasticSearch. Once in the database, they will be available through Kibana.
Environment Setup
We’ll start by creating a Docker Swarm cluster. I will assume you already have at least a basic knowledge how Docker Swarm Mode works and that you know how to create Docker services. If you don’t, I suggest you read the Docker Swarm Introduction (Tour Around Docker 1.12 Series) article or fetch The DevOps 2.1 Toolkit: Docker Swarm book.
Some of the files will be shared between the host file system and Docker Machines we’ll create soon. Docker Machine makes the whole directory that belongs to the current user available inside the VM. Therefore, please make sure that the code is cloned inside one of the user’s sub-folders.
A note to Windows users
The recommendation is to run all the examples from Git Bash (installed through Docker Toolbox as well as Git). That way the commands you’ll see throughout the book will be same as those that should be executed on OS X or any Linux distribution.
git clone https://github.com/vfarcic/cloud-provisioning.git cd cloud-provisioning scripts/dm-swarm.sh
We cloned the cloud-provisioning repository and executed the scripts/dm-swarm.sh script that created the production cluster.
Let’s confirm that the cluster was indeed created correctly.
eval $(docker-machine env swarm-1) docker node ls
The output of the node ls command is as follows (IDs are removed for brevity).
HOSTNAME STATUS AVAILABILITY MANAGER STATUS swarm-2 Ready Active Reachable swarm-1 Ready Active Leader swarm-3 Ready Active Reachable
Now that the production cluster is up and running, we can create ELK services.
scripts/dm-swarm-services-elk.sh
That’s it. We should have a few services running inside the Swarm cluster. Let’s double check it.
docker service ls
The output is as follows (IDs are removed for brevity).
NAME REPLICAS IMAGE COMMAND swarm-listener 1/1 vfarcic/docker-flow-swarm-listener logstash 1/1 logstash:2.4 logstash -f /conf/logstash.conf elasticsearch 1/1 elasticsearch:2.4 kibana 1/1 kibana:4.6 proxy 1/1 vfarcic/docker-flow-proxy
We are, finally, ready to explore how to ship logs from our Swarm services to LogStash, and from there ElasticSearch.
Forwarding Logs To LogStash
How can we forward logs from all the containers no matter where they’re running? One possible solution would be to configure logging drivers. We could use --log-driver argument to specify a driver for each service. The driver could be syslog or any other supported option. That would solve our log shipping problem. However, using the argument for each service is tedious and, more importantly, we could easily forget to specify it for a service or two and discover the omission only after we encounter a problem and are in need for logs. Let’s see if there is another option to accomplish the same result.
We could specify a log driver as a configuration option of the Docker daemon on each node. That would certainly make the setup easier. After all, there are probably fewer servers than services. If we were to choose between setting a driver when creating a service or as the daemon configuration, I’d choose the later. However, we managed to get thus far without changing the default daemon configuration and I’d prefer if we can continue working without involving any special provisioning tools. Luckily, we still did not exhaust all our options.
We can ship logs from all our containers with the project called logspout.
LogSpout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever we want. It also has an extensible module system. It’s a mostly stateless log appliance. It’s not meant for managing log files or looking at history. It is just a tool to get your logs out to live somewhere else, where they belong.
If you go through the project documentation, you’ll notice that there are no instructions how to run it as a Docker service. That should not matter since, by this time, you can consider yourself an expert in creating services.
What do we need from a service that should forward logs from all the containers running inside all the nodes that form a cluster? Since we want to forward them to LogStash that is already attached to the elk network, we should attach LogSpout to it as well. We need it to ship logs from all the nodes so the service should be global. It needs to know that the destination is the service called logstash and that it listens on the port 51415. Finally, one of the LogSpout’s requirements is that Docker socket from the host is mounted inside the service containers. It’ll use it to monitor logs.
The command that creates the service that fulfills all those objectives and requirements is as follows.
docker service create --name logspout \
--network elk \
--mode global \
--mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
-e SYSLOG_FORMAT=rfc3164 \
gliderlabs/logspout syslog://logstash:51415We created a service called logspout, attached it to the elk network, set it to be global, and mounted the Docker socket. The command that will be executed once containers are created is syslog://logstash:51415. It tells LogSpout that we want to use syslog protocol to send logs to logstash running on port 51415.
This project is an example of a usefulness behind Docker Remote API. The logspout containers will use it to retrieve the list of all currently running containers and stream their logs. This is already the second product inside our cluster that uses the API (the first being Docker Flow: Swarm Listener).
Let’s see the status of the service we just created.
docker service ps logspout
The output is as follows (IDs are removed for brevity).
NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR logspout gliderlabs/logspout swarm-3 Running Running 9 seconds ago \_ logspout gliderlabs/logspout swarm-2 Running Running 9 seconds ago \_ logspout gliderlabs/logspout swarm-1 Running Running 9 seconds ago
The service is running in the global mode resulting in an instance inside each node.
Let’s test whether the logspout service is indeed sending all the logs to LogStash. All we have to do is create a service that generates some logs and observe them from LogStash’ output. We’ll use the registry to test the setup we made so far.
docker service create --name registry \
-p 5000:5000 \
--mount "type=bind,source=$PWD,target=/var/lib/registry" \
--reserve-memory 100m \
registryBefore we check the LogStash logs, we should wait until the registry is running.
docker service ps registry
If the current state is still not running, please wait a few moments.
Now we can take a look at logstash logs and confirm that logspout sent it log entries generated by the registry.
LOGSTASH_NODE=$(docker service ps logstash | tail +2 | awk '{print $4}')
eval $(docker-machine env $LOGSTASH_NODE)
LOGSTASH_ID=$(docker ps -q --filter "ancestor=logstash:2.4")
docker logs $LOGSTASH_IDOne of the entries from the output is as follows.
{
"message" => "time=\"2016-10-19T23:14:19Z\" level=info msg=\"listening on [::]:5000\" go.version=go1.6.3 instance.id=87c31e30-a747-4f70-b7c2-396dd80eb47b version=v2.5.1 \n",
"@version" => "1",
"@timestamp" => "2016-10-19T23:14:19.000Z",
"host" => "10.0.0.7",
"priority" => 14,
"timestamp8601" => "2016-10-19T23:14:19Z",
"logsource" => "c51c177bd308",
"program" => "registry.1.abszmuwq8k3d7comu504lz2mc",
"pid" => "4833",
"severity" => 6,
"facility" => 1,
"timestamp" => "2016-10-19T23:14:19Z",
"facility_label" => "user-level",
"severity_label" => "Informational"
}As before when we tested LogStash input with logger, we have the message, timestamp, host, and a few other syslog fields. We also got logsource that holds the ID of the container that produced the log as well as program that holds the container name. Both will be useful when debugging which service and container produced a bug.
If you go back to the command we used to create the logstash service, you’ll notice the environment variable LOGSPOUT=ignore. It tells LogSpout that the service or, to be more precise, all containers that form the service, should be ignored. If we did not define it, LogSpout would forward all logstash logs to logstash thus creating an infinite loop. As we already discussed, in production we should not output LogStash entries to stdout. We did it only to get a better understanding how it works. If stdout output is removed from the logstash configuration, there would be no need for the environment variable LOGSPOUT=ignore. As a result logstash logs would also be stored in ElasticSearch.
Now that we are shipping all the logs to LogStash and from there to ElasticSearch, we should explore the ways to consult them. You’ll find them in the kibana service.
open http://$(docker-machine ip swarm-1)/app/kibana
Before leaving, please make sure to remove the machines we created and free your resources for some other tasks.
docker-machine rm -f swarm-1 swarm-2 swarm-3
The article you just finished reading is an extract from the Defining Logging Strategy chapter of The DevOps 2.1 Toolkit: Docker Swarm book.
The DevOps 2.1 Toolkit: Docker Swarm
If you liked this article, you might be interested in The DevOps 2.1 Toolkit: Docker Swarm book. Unlike the previous title in the series (The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices) that provided a general overlook of some of the latest DevOps practices and tools, this book is dedicated entirely to Docker Swarm and the processes and tools we might need to build, test, deploy, and monitor services running inside a cluster.
The book is still under “development”. You can get a copy from LeanPub. It is also available as The DevOps Toolkit Series bundle. If you download it now, before it is fully finished, you will get frequent updates with new chapters and corrections. More importantly, you will be able to influence the direction of the book by sending me your feedback.
I choose the lean approach to book publishing because I believe that early feedback is the best way to produce a great product. Please help me make this book a reference to anyone wanting to adopt Docker Swarm for cluster orchestration and scheduling.
| Reference: | Forwarding Logs From All Containers Running Anywhere Inside A Docker Swarm Cluster from our JCG partner Viktor Farcic at the Technology conversations blog. |
