I recently saw a great presentation by Joe Armstrong called “Systems that run forever self-heal and scale” . Joe Armstrong is the inventor of Erlang and he does mention Erlang quite a lot, but the principles are very much universal and applicable with other languages and tools.
The talk is well worth watching, but here’s a few quick notes for a busy reader or my future self.
General remarks
- If you want to run forever, you have to have more than one instance of everything. If anything is unique, then as soon as that service or machine goes down your system goes down. This may be due to unplanned outage or routine software update. Obvious but still pretty hard.
- There are two ways to design systems: scaling up or scaling down. If you want a system for 1,000 users, you can start with design for 10 users and expand it, or start with 1,000,000 users and scale it down. You will get different design for your 1,000 users depending on where you start.
- The hardest part is distributing data in a consistent, durable manner. Don’t even try to do it yourself, use known algorithms, libraries and products.Data is sacred, pay attention to it. Web services and such frameworks? Whatever, anyone can write those.
- Distributing computations is much easier. They can be performed anywhere, resumed or retried after a failure etc. There are some more suggestions hints on how to do it.
Six rules of a reliable system
- Isolation – when one process crashes, it should not crash others. Naturally leads to better fault-tolerance, scalability, reliability, testability and comprehensibility. It all also means much easier code upgrades.
- Concurrency – pretty obvious: you need more than one computer to make a non-stop system, and that automatically means they will operate concurrently and be distributed.
- Failure detection – you can’t fix it if you can’t detect it. It has to work across machine and process boundaries because the entire machine and process can’t fail. You can’t heal yourself when you have a heart attack, it has to be external force.It implies asynchronous communication and message-driven model.Interesting idea: supervision trees. Supervisors on higher levels of the tree, workers in leaves.
- Fault identification – when it fails, you also need to know why it failed.
- Live code upgrade – obvioius must have for zero downtime. Once you start the system, never stop it.
- Stable storage – store things forever in multiple copies, distributed across many machines and places etc.With proper stable storage you don’t need backups. Snapshots, yes, but not backups.
Others: Fail fast, fail early, let it crash. Don’t swallow errors, don’t continue unless you really know what you’re doing. Better crash and let the higher level process decide how to deal with illegal state.
Actor model in Erlang
We’re used to two notions of running things concurrently: processes and threads. The difference? Processes are isolated, live in different places in memory and one can’t screw the other. Threads can.
Answer from Erlang: Actors. They are isolated processes, but they’re not the heavy operating system processes. They all live in the Erlang VM, rely on it for scheduling etc. They’re very light and you can easily run thousands of them on a computer.
Conclusion
Much of this is very natural in functional programming. Perhaps that’s what makes functional programming so popular nowadays – that in this paradigm it’s so much easier to write reliable, fault-tolerant scalable, comprehensible systems.
A while back a wrote an article on process supervision and how best to overcome shortcomings in the approach taken by Erlang and other Java frameworks which look to emulate it. http://www.jinspired.com/site/going-beyond-actor-process-supervision-with-simulation-signaling “This all sounds good until you realize that supervision in Erlang, and much like other actor systems including Scala/Akka, is largely focused on process lifecycle management and not the internal workings of the worker process (or actor). The supervisor does not actually monitor, or is aware of, the software execution behavior of the worker process other than whether it has terminated and the cause of such termination. It… Read more »