This article is part of our Academy Course titled Docker Tutorial for Java Developers.
In this course, we provide a series of tutorials so that you can develop your own Docker based applications. We cover a wide range of topics, from Docker over command line, to development, testing, deployment and continuous integration. With our straightforward tutorials, you will be able to get your own projects up and running in minimum time. Check it out here!
Table Of Contents
1. Introduction
From the previous parts of the tutorial we already know that Docker not only has an awesome command line tooling, but exposes a feature-rich Docker Engine API as well. As of now, the officially supported clients are provided for Go and Python languages. Certainly, not very encouraging news for Java developer, but there is a light at the end of the tunnel.
Quite a long time ago Spotify started its own journey to develop a Java client for the Docker Engine API (more widely known as Spotify Docker Client). The project is being actively maintained ever since to follow all the recent developments in Docker / Engine API space and is the de-facto choice for JVM based applications nowadays. It does the job really well but has one caveat you should be aware of: because this is a community effort, not the official Java client from Docker, it takes some time to catch up with all the changes introduced by frequent Docker releases.
In this section of the tutorial we will go over a few key Docker Engine APIs and demonstrate how those could be consumed from your Java applications using Spotify Docker Client.
2. Versioning
Before getting started, we have to understand how Docker Engine API is versioned and how to match the Docker release and the version of the API it supports. The version command of the docker command line tool is what we need.
$ docker version Client: Version: 17.06.1-ce API version: 1.30 Go version: go1.8.3 Git commit: 874a737 Built: Thu Aug 17 22:48:20 2017 OS/Arch: linux/amd64 Server: Version: 17.06.1-ce API version: 1.30 (minimum version 1.12) Go version: go1.8.3 Git commit: 874a737 Built: Thu Aug 17 22:54:55 2017 OS/Arch: linux/amd64 Experimental: false
The output is pretty straightforward. We could easily spot that our version of Docker is 17.06.1-ce and its supported API version is 1.30. From that, let us bookmark the reference to Docker Engine API v1.30 documentation and roll up the sleeves.
3. Getting Started
To begin with, we would need to introduce the respective Spotify Docker Client dependency to our projects. In case of Apache Maven, it is fairly easy:
<dependency> <groupId>com.spotify</groupId> <artifactId>docker-client</artifactId> <version>8.9.1</version> </dependency>
As everything else in Docker‘s ecosystem, the Engine API evolves pretty fast so it is highly recommended to rely on the most recent Spotify Docker Client release. As of now, 8.9.1 is the latest one we could use.
Awesome, dependency is there, so we could go ahead and create an instance of DockerClient class. There are a couple of ways to do that and, essentially, it really depends on which operating system (or systems) you are using (either in development, continuous integration or production environments).
Probably, the quickest, safest and most portable approach would be to use DefaultDockerClient.fromEnv method which will gather the required details from the environment variables and will work across a range of Docker installations, including older Mac and Windows systems were Docker is not natively supported (things like Docker Toolbox or boot2docker).
final DockerClient client = DefaultDockerClient .fromEnv() .build();
Here we just assume the default Docker configuration is in place but you have a total control over customizing it, including connection URIs, registry authentication, connection pooling, etc. It is important to remember that when you are done with the instance of the client, please close it, for example:
client.close();
Great, we have our Docker client instance. It is a single entry point to call numerous Docker Engine APIs, let us take a look what we could accomplish with it.
4. Images
The first Engine API we are going to look at is Images API. The Spotify Docker Client supports most of its features through a family of methods. To illustrate that, let us borrow the Dockerfile from the previous section of the tutorial and store its content in the src/main/resources folder, inside the file with name Dockerfile:
FROM openjdk:8u131-jdk-alpine CMD ["java", "-version"]
And now, let us build the image from it using Spotify Docker Client:
final URL dockerfile = getClass().getResource("/");
final String imageId = client.build(Paths.get(dockerfile.toURI()),
BuildParam.name("base:openjdk-131-jdk"));
Easy, but how could we check if the image is there? Our old buddy docker command line tool comes into mind first, but why not to use Spotify Docker Client for that?
final ImageInfo imageInfo = client.inspectImage(imageId);
Nice, what about fetching all available images?
final List allImages = client.listImages();
What if we need to add additional tags to the image?
client.tag(imageId, "openjdk");
It is also possible to get image history, for example:
final List history = client.history(imageId);
In case we do not need this image anymore, it could be easily removed:
final List removedImages = client.removeImage(imageId, true, false);
We can also search images in the Docker Hub registry. For example, let us look up for all available JDK images:
final List jdkImages = client.searchImages("jdk");
To interface with registries, there are pull and push methods. The first one fetches the image from a registry while the latter uploads it to a registry.
client.pull("base:openjdk-131-jdk");
client.push("base:openjdk-131-jdk");
In order to get more details related to images manipulation, please check out the the official Spotify Docker Client documentation.
5. Containers
The next Engine API we are going to talk about is Containers API, arguably the richest one from the functionality perspective.
To make an use case a bit more realistic, let us spawn MySQL container instance from mysql:8.0.2 image, passing some environment variables and exposing at least one port.
final ContainerCreation container = client.createContainer(ContainerConfig
.builder()
.image("mysql:8.0.2")
.env(
"MYSQL_ROOT_PASSWORD=p$ssw0rd",
"MYSQL_DATABASE=my_app_db"
)
.exposedPorts("3306")
.hostConfig(
HostConfig
.builder()
.portBindings(
ImmutableMap.of(
"3306",
ImmutableList.of(
PortBinding.of("0.0.0.0", 3306)
)
)
)
.build()
)
.build()
);
In order for container creation to succeed, we would need to have mysql:8.0.2 available locally. If you are not sure you have one, it is always better to pull the image before using it.
client.pull("mysql:8.0.2");
At this point the container is created but not started yet, so let us run it:
client.startContainer(container.id());
Having the container up and running, there are quite a few things which could be done with it. We could get all the details about the container by inspecting it:
final ContainerInfo info = client.inspectContainer(container.id());
Pretty straightforward, but what if we need to capture the logs or the standard output from the container? This could be accomplished by attaching to the running container, for example:
client .attachContainer(container.id(), AttachParameter.values()) .attach(System.out, System.err, false);
There is one caveat though, the last attach invocation will block the calling thread, waiting for the output from the container. This is probably not what most of us expect so we should better offload the invocation to dedicated thread.
executor.submit(
() -> {
client
.attachContainer(container.id(), AttachParameter.values())
.attach(System.out, System.err, false);
return null;
});
You have the control over the state of the processes within the container by pausing / unpausing them, for example:
client.pauseContainer(container.id()); client.unpauseContainer(container.id());
There are also the APIs exposed to fetch the runtime statistics about the container or to get the list of running container processes.
final ContainerStats stats = client.stats(container.id()); final TopResults top = client.topContainer(container.id());
Finally, there is a dedicated API to fetch and tail the logs of the container, quite similar to the attachContainer method we have discussed above.
client
.logs(container.id(), LogsParam.stdout(), LogsParam.stderr(), LogsParam.tail(10))
.attach(System.out, System.err, false);
Once the container is not needed anymore, it could be stopped and terminated right after.
client.stopContainer(container.id(), 5 /* wait 5 seconds before killing */); client.removeContainer(container.id());
In order to get more details related to containers manipulation, please check out the official Spotify Docker Client documentation.
So we know how to deal with images and containers using purely Docker Engine APIs with a help of Spotify Docker Client. Let us talk about equally important topics like managing ports, networks, volumes and resource limits.
6. Ports
Every container could expose ports to listen to at runtime, either through image Dockerfile instructions or through the arguments of the createContainer method invocation. The Spotify Docker Client does not provide the specialized method to fetch the ports and their mappings but this information could be easily extracted from ContainerInfo, for example:
final ContainerInfo info = client.inspectContainer(container.id()); final ImmutableMap<String, List> mappings = info.hostConfig().portBindings();
In fact, a more reliable way would be to use info.networkSettings().ports() as the container may not necessarily use port mappings, but only expose some ports.
final ContainerInfo info = client.inspectContainer(container.id()); final ImmutableMap<String, List> mappings = info.networkSettings().ports();
7. Volumes
In the previous section of the tutorial we have learned about volumes as the preferred mechanism for persisting data used by containers. The Spotify Docker Client comprehensively wraps Volumes API through the family of methods. Let us start by creating a new volume:
final Volume volume = client.createVolume();
It is possible to get the metadata about the existing volume:
final Volume info = client.inspectVolume(volume.name());
Also, you could fetch all available volumes (and filter them if necessary), for example:
final VolumeList volumes = client.listVolumes();
Once the volume is not needed anymore, it could be removed.
client.removeVolume(volume.name());
In order to get more details related to volumes manipulation, please check out the official Spotify Docker Client documentation.
8. Networks
The importance of the user-defined networks in Docker is hard to overestimate as they play a critical role in how containers communicate which each other. Not surprisingly, the Networks API is supported mostly in full power by Spotify Docker Client.
The creation of a new network is just one method away:
final NetworkCreation network = client.createNetwork(
NetworkConfig
.builder()
.name("test-network")
.driver("bridge")
.build()
);
If you know the network identifier, you could fetch its metadata:
final Network info = client.inspectNetwork(network.id());
If you do not know the network identifier or just interested in querying (and filtering) all networks, you could do that as well:
final List networks = client.listNetworks();
Once the network is not needed anymore, it could be easily removed.
client.removeNetwork(network.id());
Spotify Docker Client provides the means to connect to the network or disconnect from the network for any running (or just created) container, for example:
final ContainerCreation container = client.createContainer(ContainerConfig
.builder()
.image("mysql:8.0.2")
.env(
"MYSQL_ROOT_PASSWORD=p$ssw0rd",
"MYSQL_DATABASE=my_app_db"
)
.exposedPorts("3306")
.build()
);
client.startContainer(container.id());
client.connectToNetwork(container.id(), network.id());
Or:
client.disconnectFromNetwork(container.id(), network.id());
As of now, the official Spotify Docker Client documentation does not include a section dedicated to networks manipulation.
9. Resource Limits
The Spotify Docker Client also supports the ability to dynamically adjust the container configuration (primary, resource limits like memory, CPU, or block I/O), for example:
final ContainerUpdate update = client.updateContainer(container.id(), HostConfig .builder() .memory(268435456L /* limit to 256Mb */) .build() );
10. The Primer
To finish up this section, it would be great to have a realistic example where all the APIs we have looked at so far work seamlessly together. As we have been playing mostly with MySQL, quite popular choice as a relational data store for Java (and not only) applications, the end-to-end demo would be based on it. So what are our goals?
- Create a container instance using
mysql:8.0.2image - Expose TCP port
3306and bind it to a random host port (so we could run as many containers as we want, without having a port conflicts) - Make sure the container is started and MySQL server process is ready to serve our queries
- Connect to the MySQL instance using JDBC driver for MySQL and list all catalogs (databases)
Sounds like a lot of work, but … likely we learned about Spotify Docker Client, so let us take one step at a time to make the miracle happen.
final DockerClient client = DefaultDockerClient
.fromEnv()
.build();
// Pull the image first
client.pull("mysql:8.0.2");
// Create container
final ContainerCreation container = client.createContainer(ContainerConfig
.builder()
.image("mysql:8.0.2")
.env(
"MYSQL_ROOT_PASSWORD=p$ssw0rd",
"MYSQL_DATABASE=my_app_db"
)
.exposedPorts("3306")
.healthcheck(
Healthcheck
.create(
// MySQL image doesn't have `nc` available
Arrays.asList(
"CMD-SHELL", "ss -ltn src :3306 | grep 3306"
),
5000000000L, /* 5 seconds, in nanoseconds */
3000000000L, /* 3 seconds, in nanoseconds */
5
)
)
.hostConfig(
HostConfig
.builder()
.portBindings(
ImmutableMap.of(
"3306",
ImmutableList.of(
PortBinding.of("0.0.0.0", 0 /* use random port */)
)
)
)
.build()
)
.build()
);
This snippet should be looking familiar except probably the heath check part. Why do we need it? Well, it would be great if Docker tells us not only when container has started but when the application inside the container is fully up and running. This is exactly what the health checks are about. With respect to MySQL, the by the book recipe is to verify if there is a process listening on port 3306, and this is precisely what we are doing here with ss -ltn src :3306 | grep 3306 shell command.
// Start the container
client.startContainer(container.id());
// Inspect the container's health
ContainerInfo info = client.inspectContainer(container.id());
while (info.state().health().status().equalsIgnoreCase("starting")) {
// Await for container's health check to pass or fail
Thread.sleep(1000);
// Ask for container status
info = client.inspectContainer(container.id());
// Along with health, better to check the container status as well
if (info.state().status().equalsIgnoreCase("exited")) {
LOG.info("The container {} has exited unexpectedly ...", container.id());
break;
}
}
Nothing complicated in this snippet above as well. We are periodically polling the container and checking its health status, the values we may get here are starting, healthy, and unhealthy. Please notice that we also consult the container status as the health check may not reflect the actual state of the container in some circumstances.
// Check if container is healthy
if (info.state().health().status().equalsIgnoreCase("healthy")) {
// ...
}
Once the Docker reports to us that the container is healthy, we are ready to open a JDBC connection to MySQL instance. But before that, we have to figure out which port to use.
final PortBinding binding = info
.networkSettings()
.ports()
.get("3306/tcp")
.get(0);
final int port = Integer.valueOf(binding.hostPort());
We have it, let us pass the port directly to JDBC connection string and we are almost there! We just need to put in the last piece of the puzzle, the name of the host to connect. Or should it be localhost? The answer is: it really depends. On most operating systems with native Docker support the localhost is pretty safe bet. However, it is better to ask Spotify Docker Client what is the proper one to use by calling client.getHost(), it may not return localhost all the time (like in case of boot2docker for example).
final String url = String.format(
"jdbc:mysql://%s:%d/my_app_db?user=root&password=p$ssw0rd&verifyServerCertificate=false",
client.getHost(), port);
try (Connection connection = DriverManager.getConnection(url)) {
try(ResultSet results = connection.getMetaData().getCatalogs()) {
while (results.next()) {
LOG.info("Found catalog '{}'", results.getString(1));
}
}
} catch (SQLException ex) {
LOG.error("MySQL connection problem", ex);
}
Awesome, if everything works perfectly fine (as it should), you would see something like that in the console output (please notice that our database my_app_db is also here):
... 10:25:19.439 [main] INFO Found catalog 'information_schema' 10:25:19.439 [main] INFO Found catalog 'my_app_db' 10:25:19.439 [main] INFO Found catalog 'mysql' 10:25:19.439 [main] INFO Found catalog 'performance_schema' 10:25:19.439 [main] INFO Found catalog 'sys'
Last but not least, please do not forget to terminate the container and close the Spotify Docker Client instance at the very end.
// Stop the container client.stopContainer(container.id(), 5); client.removeContainer(container.id()); client.close();
And we are done! Could you accomplish all that using just Docker Engine API directly? Definitely, but the amount of supporting Java code you would need to write for that would unpleasantly surprise you.
11. Conclusions
Fairly speaking, Docker Engine API is just a set of REST(ful) services, very rich ones though. For sure, anyone could access them from any Java application just by using generic HTTP client. So what the deal?
Personally, I would emphasize the explicit contracts, maintainability and type-safety. Instead of dealing with URIs, query or path parameters, raw JSON for request and response payloads, you have a stable Java-based API, with strict set of capabilities and expectations.
As we are going to see later on, Spotify Docker Client serves as a solid foundation for automation and advanced testing techniques, particularly in context of Java development. Thank you, Spotify, for giving back to the community!
12. What’s next
In the next section we are going to talk about the ways to migrate your build pipelines to Docker, as such eliminating the need to replicate and maintain the tooling on each host or in every environment.
The complete project sources are available for download.
