Kafka and MQTT are two complementary technologies. Together they allows to build IoT end-to-end integration from the edge to the data center. No matter if on premise or in the public cloud. I talk about this topic at Kafka Summit SF in San Francisco in October 2018: “Processing IoT Data from End to End with MQTT and Apache Kafka“.
As preparation for the talk, I created two demo projects on Github. I want to share these here. A more detailed blog post including slides and video recording will come after the conference.
Motivation for using Apache Kafka and MQTT together
MQTT is a widely used ISO standard (ISO/IEC PRF 20922) publish-subscribe-based messaging protocol. MQTT has many implementations such as Mosquitto or HiveMQ. For a good overview, see Wikipedia’s comparison of MQTT brokers. MQTT is mainly used in Internet of Things scenarios (like connected cars or smart home). But it is also used more and more in mobile devices due to its support for WebSockets.
However, MQTT is not built for high scalability, longer storage or easy integration to legacy systems. Apache Kafka is a highly scalable distributed streaming platform. Kafka ingests, stores, processes and forwards high volumes of data from thousands of IoT devices.
Therefore, MQTT and Apache Kafka are a perfect combination for end-to-end IoT integration from edge to data center (and back, of course, i.e. bi-directional).
Let’s take a look at two different alternatives for building this integration. Both solutions allow highly scalable and mission-critical integration of IoT scenarios. Thought, they use different concepts (including trade-offs).
MQTT Broker + Apache Kafka + Kafka Connect MQTT Connector
This project focuses on the integration of MQTT sensor data into Kafka via MQTT Broker and Kafka Connect for further processing:
In this approach, you pull the data from the MQTT broker via Kafka Connect to the Kafka broker. You can leverage any features of Kafka Connect such as built-in fault tolerance, load balancing, Converters, Simple Message Transformations (SMT) for routing / filtering / etc., scaling different connectors in one Connect worker instance, and other Kafka Connect related benefits.
Here is the Github project including demo script to try it out by yourself:
Apache Kafka + Kafka Connect + MQTT Connector + Sensor Data
MQTT Proxy + Apache Kafka (no MQTT Broker)
As alternative to using Kafka Connect, you can leverage Confluent MQTT Proxy: Integrate IoT data from IoT devices directly without the need for a MQTT Broker:
In this approach, you push the data directly to the Kafka broker via the Confluent MQTT Proxy. You can scale MQTT Proxy easily with a Load Balancer (similar to Confluent REST Proxy). The huge advantage is that you do not need an additional MQTT Broker in the middle. This reduces efforts and costs significantly. It is a better approach for pushing data into Kafka.
Here is the Github project including demo script to try it out by yourself:
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data
Both approaches have their trade-offs. The good thing is that you have the choice. Choose the right tool and architecture for your use case.
Conclusion: MQTT and Apache Kafka are complementary and integrate well
Both, MQTT and Apache Kafka, have great benefits for their own use cases. But none of them is the single allrounder for everything. The combination of both makes them very powerful and a great solution to build IoT end-to-end scenarios from edge to data center and back.
At Kafka Summit, I will share many more details and content:
- How to integrate MQTT and Apache Kafka end-to-end (not just talking about connectivity, but also about data processing, filtering, routing, etc.)
- When to choose Kafka Connect and MQTT Broker vs. MQTT Proxy without MQTT Broker
- Slides and video recording of talk and live demo
By the way: If you want to see an IoT example with Kafka sink applications like Elasticsearch / Grafana, please take a look at the project “Kafka and KSQL for Streaming IoT Data from MQTT to the Real Time UI“. This example shows how to realize the integration with ElasticSearch and Grafana via Kafka Connect.
| Published on Java Code Geeks with permission by Kai Waehner, partner at our JCG program. See the original article here: Apache Kafka + MQTT = End-to-End IoT Integration (Github Examples) Opinions expressed by Java Code Geeks contributors are their own. |
