Lambda Architecture is a simple, powerful, though limited example of a Microservice. As it is so simple, you want to use it as much as possible, to expose the more complex services/component in your system which cannot support this interaction model.
Lambda Architecture depends on a data model with an append-only, immutable data source that serves as a system of record. It is intended for ingesting and processing timestamped events that are appended to existing events rather than overwriting them. State is determined from the natural time-based ordering of the data.
A translation function
The microservice takes messages in and produces messages out. It’s input is an append only data structure, and the function has no state of it’s own which is not derived from it’s inputs.
This simple translator is useful for taking messages from external systems and normalising those messages into an internal format. Conversely, you can convert messages from an internal message format to an external format.
Working with history
Notionally, all the messages to date form an input of the function. Replaying every message on every new message is too expensive. You can instead cache key information derived from those messages to provide a real time solution.
This is a variation on the standard function where state is retained based on it’s inputs. Provided this is only an optimisation, rather than a change in behaviour, this is still a function. i.e. a function of all the messages, rather than just one message.
Having a function with private state gives it memory and makes it practical for more complex operations such as control (permissioning, turning functionality on and off), matching of messages over time (e.g. order matching)
Putting these together
By having translators for messages in and out, as well as a central control system, you can build a more complex processing system.
The functionality here should form your critical path. You want a simple message to be able to pass from one end of your system to the other in the shortest time possible. In Java this can mean as low as 10 micro-seconds 99% of the time. As well as being fast it is simple to test each function independently, as well as test this without transports for ease of debugging.
Non-critcial path functions.
Not everything can or should be on the critical path. There will be operations which either use a lot of CPU, or need to obtain data from external systems such as databases. You need a way to interact with these systems without impacting the critcial path. Using feedback is a simple way to achieve this.
In this diagram, the slower functions are CPU bound strategies, but you could have components which obtain data from a database for example, without slowing the critical path.
Reading more
You can read my earlier posts on how to implement these Lambda Architecture components in Java and easily test them. Micro services for Performance as well has how you can implement a simple service to asynchronously access a JDBC database A JDBC Gateway Microservice
For low latency, high throughput persistence we recommend using Chronicle Queue which is Apache 2.0 open source project available at https://github.com/OpenHFT/Chronicle-Queue and on Maven Central
| Reference: | Simple Asynchronous Microservices using Lambda Architecture from our JCG partner Peter Lawrey at the Vanilla Java blog. |
I don’t get it. Why its call “Lambda” ? It is an old architecture that applied since the start of Java. Nothing related to Java 8 and “Lambda expression”
Correct. Alonzo Church invented lambda calculus in 1936. Lambdas was used to mean functions long before “Lambdas” were added to Java 8.
From: https://en.wikipedia.org/wiki/Lambda_architecture
Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods. … Lambda architecture depends on a data model with an append-only, immutable data source that serves as a system of record. It is intended for ingesting and processing timestamped events that are appended to existing events rather than overwriting them. State is determined from the natural time-based ordering of the data.