Yesterday at DockerCon Europe, Andrey Sibiryov, a senior engineer at Uber Technologies, demonstrated how to improve load-balancing performance using an open-source technology that’s been part of the Linux kernel for more than a decade — IPVS.
Andrey Sibiryov discusses IPVS at DockerCon Europe.
What Is IPVS?
To be precise, IPVS (IP Virtual Server) has been in the kernel for 15 years, and it’s based on netfilter. It supports TCP, SCTP, and UDP and can achieve incredibly fast speeds, often within 5 percent of direct connection speeds. Other features include NAT, tunneling, and direct routing.
It’s not a proxy — it’s a forwarder that runs on Layer 4. IPVS forwards traffic from clients to back-ends, meaning you can load balance anything, even DNS! Modes it can use include:
- IPIP. Encapsulates IP and is routable anywhere. The response skips the load balancer, so only the request goes through IPVS.
- DNAT. Rewrites DST IP, uses the same L4 and behaves like a NAT, rewriting the ip packets and forwarding request and response traffic.
- DSR. Rewrites DST MAC to forward at L2.
In testing, IPVS’ performance was nearly the same TPS as a normal connection. HAProxy and MySQL proxy were around 60 percent of its speed.
Why Do We Need IPVS?
Sibiryov said that if you have more than one instance of an app, you need load balancing. But if NGINX/HAProxy, hipache, and Vulcan work fine, why would you need IPVS?
The simple answer is speed. But there are other benefits:
- UDP support
- Dynamically configurable
- More than eight balancing methods
- Health checking
(https://no-cache.hubspot.com/cta/default/1169977/224ef658-b146-4569-b00b-3c0ad87be198.png)
How Do We Use IPVS?
As Sibiryov said, “Linux kernel things are crazy magical tools from space.” So how do you put something like that to good use?
To make IPVS easier to use, he wrote GORB. It’s a REST API for IPVS inside a Docker Container to provide IPVS routing for Docker. It’s still pretty beta, but it looks really cool. GORB has a bunch of features:
- REST API Daemon Container to control IPVS
- GORB Docker Link Container to automatically detect Docker containers and route to them
- TCP + HTTP health checks built-in
- Written in Go and LGPL licensed
- Docker + GORB + MacVLAN network plugin + IPVS DR mode = dream setup
Sibiryov pointed out that small companies don’t necessarily need IPVS, but it does become useful once traffic scales. In that situation, the MacVLAN Docker network plugin enables you to give containers their own MAC addresses to utilize the fastest IPVS mode by forwarding at L2.
Next, Sibiryov went into a demo of GORB. He ran Docker via Virtualbox on his Mac. The Dockerfile for the GORB container is super simple because it’s a Go app. You just start with the golang base and go get github.com/kobolog/gorb. The ENTRYPOINT is simply gorb.
He ran GORB in privileged mode because it has to access the host kernel’s IPVS module (that also means you have to have IPVS in your host kernel to use GORB). When he ran the GORB container, he had to give direct access to his host network interface eth1 so the traffic could be accessible and routed.
Next, he ran the GORB Docker Link container. This container has access to the host Docker daemon so it can listen to Docker events. The link container also needs access to the host network interface. The link container notices when containers are started, checks what ports are exposed, and forwards traffic on the host interface on that port to any containers running with that port exposed.
He ran five NGINX containers, each exposing port 80. Docker automatically allocates different ports for each container externally, but the GORB Docker link container detects them and routes them all via the host’s port 80 with IPVS. Then, he ran Apache Bench, and of course it was ridiculously fast.
Conclusion
To conclude his talk, Sibiryov briefly compared IPVS to a $25,000 enterprise hardware load balancer. While GORB is still relatively new (he cautioned to watch out for bugs), IPVS is open-source, configurable, and like GORB, it’s also free. It requires no special software or tools, and can leverage a 1 GBPS line with only 1 percent CPU usage in DR mode (MAC L2).
Plus, IPVS has a 1,000-word man page instead of a vendor manual, and let’s not forget that it’s 15 years old. Simplicity and stability, Sibiryov stressed, make IPVS a serious alternative to enterprise load-balancing hardware.
| Reference: | Kernel Load-Balancing for Docker Containers Using IPVS from our JCG partner Nick Gauthier at the Codeship Blog blog. |
