Back to the drawing board: The lost lines of code on the road to Akka Streams, Akka HTTP and Akka Typed
If you’ve ever been curious about how new core features are being developed and why your favorite technologies take on a certain path, you’ve reached the right place. In the new design decisions series on the Takipi blog you’ll discover the driving forces behind the tools and frameworks you use on a daily basis. You’ll see what makes them tick, how the gears fall into place, meet the developers behind all those lines of code and get a chance to pitch in and ask the tough questions that you want answered.
New Post: The Bet With Typesafe That Got Me Into Building Akka http://t.co/yqtpFahIMB pic.twitter.com/H5oTlhD9Uz
— Takipi (@takipid) May 11, 2015
For the first interview we reached out to Dr. Roland Kuhn, Akka Tech Lead at Typesafe, to get a unique inside view of the work behind Akka. At Typesafe Kuhn and the team are in charge of deciding where to go next with the framework, the semantics of new operations, and determining the right answers for the problems it aims to solve. A little known fact is that Roland became a core contributor in Akka just before the 1.0 release in January 2011. Over christmas he suggested to Viktor Klang, which is now Typesafe’s Deputy CTO, the idea of the CallingThreadDispatcher, to which he replied: ”Nah you won’t be able to do it – I challenge you to write one!” – And that’s exactly what happened. “Working on an open source project has been my dream for I don’t know how long” says Roland, and that’s what got him in there.
The Akka team is made of 5 developers and the only precondition for joining is that you need to be in or move to Europe. Although a distributed team, they meet regularly and remain in pretty much the same timezone. But that’s not the full story, since in Akka’s case, the community plays a big role as well and is probably the biggest motivator for the creation of the framework.
The Problems that Akka Tackles are Those of Distributed Computing
“The Actor model has been described as a concurrency construct; but that is a very narrow viewpoint”, says Roland. “It actually is about distribution. Independent distributed agents that communicate using messages and that’s precisely what we focus on – Everything in Akka is geared towards that. You can run your application in a single process, and you can run it on a cluster of a few thousand nodes if need be. All features that we have built in are centered around enabling actors to do something”.
“The foremost concern that we always keep in mind is to keep the model clean and simple. So ease of use as typically defined is not our primary driver. Our viewpoint is that we need to expose the essential complexity of the domain that we are modelling. And in terms of distributed computing, that means, if you really want to write a distributed application, you will have to understand what that means, Akka does not take that aspect away from you. What we do is provide an abstraction, an Actor in this case, that distills the essence of what it means to be distributed and that then just works. You send a message to an Actor and the message will be delivered to that Actor no matter where where on a cluster the recipient lives”.
“We are a strong believer in using the right tool for a job so we try to build something that is simple and thereby easy to understand, but you must be willing to think about what you are using. And it must work without compromises in all situations, even if that costs a little bit performance. And within these constraints we still try to make it as fast as we can”.
6 Versions of Akka Streams Were Thrown Away Before Deciding on This One
“We let ourselves to be guided by the community, but that is not the entire truth because sometimes we just need to walk ahead a few steps and then some people will follow and validate, or, well, just tell us what is wrong with our ideas. Akka Streams, for example, was conceived after seeing on our mailing list that developers who use Akka usually have two kinds of problems:
1. They send messages from one Actor to another, and the recipient Actor has an unbounded mailbox so they don’t deal with back pressure, or just don’t care for it, and then the mailbox grows without bound until you have an OutOfMemoryError. So the pattern that you need to implement here is flow control.
2. Actors are currently not “usefully typed” as I like to call it, they are uni-typed. You can send any message to any actor. We are trying to change that and I think we have good chances of succeeding with the Akka Typed project. But at the same time, this has a lot of freedom so you can model any messaging pattern with it”.
“Akka Streams is something the community basically screamed that it needed”
“So, we have seen these issues and we thought, well, let’s just package that up and create an abstraction that solves them, which is Akka Streams. We thought it must do back pressure, it must not run out of memory, and it should be properly type checked, the whole set up. These are pretty vague constraints which explains why it took us 1.5 years of experimenting, building new solutions and throwing it away I think six times, at least that was when I stopped counting, before we settled on a design that we now think will be successful.”
The State of Akka Streams: Are They Ready for Production?
Akka Streams are designed as a generic solution to the streams processing problem. They let you describe any sort of data stream, model all sorts of combinations and not only rely on transformations like map or filter, but also modify the rate at which the elements are flowing. This means you can conflate multiple elements into a single one if the input is too fast, or extrapolate, turning one element into many. More operations include modeling junctions so you can merge, or route to multiple destinations.
Moreover, Roland says “We try to be as good and thorough as we can in our reviews. So the codes should work, but it’s still a development branch, so it has bugs, but people are already using it in production systems. It solves these problems of back pressure and having more type safety and people are really really hungry for that, so this is why they’re using even these development previews. We are aiming for the upcoming release to satisfy this demand.”
“Building on Akka Streams we are re-envisioning how HTTP should be handled”
The next thing on the road map is Akka HTTP. Traditionally HTTP was handled so that a request is one object and the response is another object. You receive a request, and then you generate a response that’s passed back to the HTTP layer and back to the client. According to Roland, that’s becoming now the old way of doing things: “It has become more and more apparent that you want to exchange not only small request and response pairs but sometimes you just want to upload or download a few gigabytes of data that does not really fit this model. Since we’re starting completely fresh, Akka HTTP is built completely on Akka Streams. So when you get an HTTP request in Akka HTTP, it’s actually a stream of byte strings and you can compose it, you can transform it, you can pass it to a different server or to a file without having to have the bytes in memory. And that I think is a new thing that we’re adding, we do fully streamified HTTP handling”.
Akka Typed – A Working Solution is Not Necessarily the Right Solution
Another project that’s on Akka’s roadmap is Akka Typed, making type checking available for Actor interactions. Roland started with that in Akka 1.2 by trying to unify Futures and ActorRefs under a channel abstraction: “That didn’t really work and I removed it. Then when Scala macros came out, we created another implementation that worked but wasn’t really successful because macros are an experimental feature and many people don’t dare using that in commercial products. So this is why Typed Channels were also removed. They worked but were too complex so it was a failed experiment. After seeing these failures, I finally gave up and simplified the whole proposal by removing Akka features of our implementation of the Actor model that are not in the original model at all”.
“Akka Typed will be part of our next version – In an even more experimental than usual preview mode. It will take a long while until this part will be called stable”.
“Instead of ActorRefs it’s now ActorRefs of some type T and you cannot send other messages than that of type T to this ActorRef. This is the basic change, but it changes so much under the hood. I am not sure when or if we will remove the untyped Actors, they might just live side-by-side for quite a while and we will all experiment with Akka Typed and wait for community feedback to see where it should be going”.
“One thing that we have still not completely fixed is Akka Persistence and it still has the experimental marker. With the community’s input we concluded that we need to release Akka Streams first so that we can then use them in Akka persistence. That is something we will work on this year as well”. We also asked if there are any other fixes waiting to be introduced to Akka:
“We committed some sins in the design that are now basically impossible to fix because they break all the code”.
“One of them is that you can call System.ActorOf and get back an ActorRef, synchronously. This is not how it’s supposed to be as witnessed by the code that was necessary to write to support this. It’s really subtle and way too complex. Since we cannot really remove System.ActorOf, the way by which we plan to amend this is not to include it in Akka Typed. In Akka Typed, there’s a new kind of Actor system that serves the same purpose but it doesn’t have ActorOf anymore. And then my cunning plan is to offer this, show it to the community and see whether anyone screams, “I NEED MY SYSTEM ACTOROF!”.
“The funny thing is, with all these changes we are removing aspects that we have added on top of the original Actor model. So we are compacting, we’re getting back closer to the Actor model than we have previously had”.
“We continue to be inspired by early ideas so what Carl Hewitt wrote in ’73 is still accurate and we implemented that, and we’re also inspired by recent progress on the theoretical side of computer science. For example CRDTs (Conflict-free Replicated Data Types) are extremely interesting. We happened across one when implementing the Akka cluster, we implemented our own CRDT before the paper that describes them was actually published and I think we will continue to work on that topic as well”.
As to supporting Java together with Scala, Roland says “Our general philosophy has been that we offer both the Java and the Scala API on the same objects. Everything we do works equally well from Java and from Scala and we try to offer the same power”.
Debugging Actors at Scale – The World is Now Distributed and You Can’t Stop it
The old scheme of attaching a debugger, hitting a breakpoint and then stopping the world does not work anymore: “Other systems will be running while you are inspecting your data, timeouts will typically fire and the behavior of the program under a debugger in development will be significantly different from the one that you actually try to understand in production. This is why in order to debug reactive systems of any kind, you will need to rely on tracing, monitoring and logging on a live system. You cannot place a breakpoint in your development environment and see what’s happening because that usually does not tell you what you need to know”.
How to Debug Distributed Systems?
Large-scale production environments can produce millions of errors a day from hundreds of different locations in the code. While some errors may be trivial, others break critical features and affect end-users without you knowing it. Traditionally, to identify and solve these errors you’d have to rely on your log files or a log management tool to even know an error occurred, let alone how to fix it. BUT… If you’re debugging a Scala or a Java application, things are a bit different. With Takipi in your monitoring stack you can detect caught and uncaught exceptions, Actor and log errors and see the code and variable state that caused them. You check it out right here.
To conclude, we’d like to thank Roland for joining us on this interview and hope you’ve found it interesting as much as we did! It’s definitely been a fun ride, and we hope you have a better understanding of how Akka works, what it’s based on and the forces that keep shaping it as we speak. It’s also been great to learn about Roland’s journey, from following up on a message on the Scala mailing list, to becoming Akka’s tech lead and fulfilling his dream of working on an open source project. It’s exciting to think of how this journey will keep on going and where it will take us next.
If you have more questions or curious to clear up a few topics that weren’t covered in this interview, please feel free to post your questions in the comment section below.
| Reference: | The Bet With Typesafe That Got Me Into Building Akka from our JCG partner Alex Zhitnitsky at the Takipi blog. |
