Apache Kafka GroupId vs ConsumerId vs ClientId

Apache Kafka’s consumer groups are a powerful feature that enables parallel processing of messages from topics. When working with Kafka consumers and managing group subscriptions, it’s essential to understand the distinction between key identifiers used within the Kafka ecosystem: GroupId, ConsumerId, and ClientId. This article breaks down the key differences between GroupId, ConsumerId, and ClientId.

1. What is a Consumer Group?

A consumer group acts as a unit, with multiple consumers working together to consume messages from one or more topics. This allows for efficient message processing by distributing the workload across multiple instances. Consumer groups allow messages from multiple partitions to be processed concurrently by different consumer instances within the same group, thereby maximizing throughput.

1.1 Consumer Group Example

Let’s consider an example where we have a Kafka topic named orders with three partitions. We create a consumer group named orderProcessors with two consumer instances (consumer-1 and consumer-2). Kafka automatically assigns partitions to these consumers as follows:

• consumer-1 -> partitions 0, 2
• consumer-2 -> partition 1

Now, each consumer instance within the orderProcessors group processes messages from its assigned partitions concurrently, allowing efficient and parallel message processing.

In the next sections, we will delve into the important identifiers used within consumer groups—GroupId and ConsumerId—and how they are configured and managed in Apache Kafka using Spring Kafka and Kafka CLI tools. Understanding these identifiers is key to effectively managing consumer groups and optimizing Kafka applications for scalability and reliability.

2. Identifiers in Apache Kafka Consumer Groups

In Apache Kafka, identifiers such as GroupId, ConsumerId, and ClientId play critical roles in managing consumer groups and individual consumer instances. Identifiers in Kafka are unique labels assigned to different components within the Kafka ecosystem. They serve specific purposes in organizing and managing message consumption and producer-client interactions.

2.1 Key Identifiers in Kafka

Understanding these identifiers is essential for optimizing Kafka applications for performance and scalability.

• GroupId:
  • The GroupId identifies a consumer group, which is a logical collection of consumer instances that work together to consume messages from one or more topics.
  • Consumers within the same GroupId coordinate to process messages from assigned partitions, ensuring parallelism and fault tolerance.
• ConsumerId:
  • The ConsumerId uniquely identifies an individual consumer instance within a consumer group.
  • Each consumer instance in a group has a distinct ConsumerId, which helps Kafka track its progress and state in consuming messages.
• ClientId:
  • The ClientId is an identifier assigned to a Kafka client application (producer or consumer).
  • It represents a logical grouping of related producer or consumer instances within an application.
  • Multiple consumer instances or producers can share the same ClientId if they belong to the same application.

2.2 Importance of Identifiers

• Dynamic Partition Assignment: Kafka uses identifiers like GroupId and ConsumerId to dynamically assign partitions to consumer instances within a group, ensuring load balancing and fault tolerance.
• Resource Management: Identifiers help Kafka manage resources and state for each consumer instance and client application, facilitating efficient message processing and scalability.
• Fault Recovery: Identifiers enable Kafka to track the progress of consumers and recover from failures by reassigning partitions and redistributing workload among active instances.

2.3 Tabular Difference: GroupId, ConsumerId, and ClientId

3. Configuring GroupId and ConsumerId with Spring Kafka

3.1 Using Spring Boot and Spring Kafka

Spring Kafka simplifies the integration of Kafka into Spring-based applications. Below is an example of how to configure GroupId and ConsumerId in Spring Kafka:

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class KafkaConsumer {

    @KafkaListener(
            topics = "myTopic",
            groupId = "myConsumerGroup",
            id = "consumer1",
            clientIdPrefix = "myApp-"
    )
    public void listen(String message) {
        // Process received message
        System.out.println("Received message: " + message);
    }
}

In the KafkaListener annotation:

• topics: Specifies the topic to subscribe to.
• groupId: Sets the consumer group identifier (GroupId).
• id: Sets the unique consumer instance identifier (ConsumerId).
• clientIdPrefix: The clientIdPrefix property is used to prepend a unique identifier (myApp-) to the ClientId for each consumer instance.

3.1.1 Configuring Properties in application.properties

spring.kafka.consumer.group-id=myConsumerGroup
spring.kafka.consumer.client-id=myKafkaApp
spring.kafka.consumer.client-id-prefix=myApp-
spring.kafka.consumer.auto-offset-reset=earliest

In the above properties file:

• spring.kafka.consumer.group-id: Configures the default consumer group (GroupId).
• spring.kafka.consumer.client-id: Sets the client identifier (ClientId).
• The spring.kafka.consumer.client-id-prefix property prepends a unique identifier (such as myApp-) to the ClientId for each consumer instance.
• Other properties like auto-offset-reset control consumer behaviour.

3.2 Configuring GroupId and ConsumerId Using Kafka CLI

Configuring and setting GroupId, ConsumerId, and ClientId in Apache Kafka using the Kafka Command Line Interface (CLI) involves specifying properties and options when running the consumer commands. Let’s demonstrate configuring and setting these identifiers using Kafka CLI commands.

3.2.1 Using Kafka Consumer Groups CLI

The Kafka CLI provides tools to manage consumer groups directly from the command line. Enter the following command to inspect or configure consumer group details:

List Consumer Groups:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Describe Consumer Group:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group myConsumerGroup

3.2.2 Configuring Consumer Group Using CLI

To launch a Kafka consumer with a specific GroupId, ClientId and ConsumerId using Kafka CLI, enter the following command:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --group myConsumerGroup --consumer-property group.instance.id=consumer1 --consumer-property "client.id=myApp"

Explanation:

• --bootstrap-server localhost:9092: Specifies the Kafka broker(s) to connect to.
• --topic myTopic: Specifies the topic from which to consume messages.
• --group myConsumerGroup: Specifies the consumer group (GroupId) to join.
• --consumer-property group.instance.id=consumer1: Sets the unique identifier (ConsumerId) for this consumer instance within the consumer group.
• --consumer-property "client.id=myApp": Sets the logical identifier (ClientId) for the Kafka client application.

When we examine the consumer group using the --describe --group command shown above, we will observe the unique GroupId, ConsumerId and ClientId generated by Kafka for each individual consumer as shown below:

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
myConsumerGroup myTopic         0          0               0               0               consumer1-04db76e3-8c5a-4ec4-abb1-e56a6e327898 /192.168.0.200  myApp

4. Conclusion

In this article, we have explored important identifiers linked with Kafka consumers: GroupId, ClientId, and ConsumerId. In summary, configuring GroupId, ConsumerId, and ClientId is crucial for managing and scaling Kafka consumer instances effectively. These identifiers help Kafka coordinate message consumption within consumer groups, manage state and ensure scalability in distributed systems.

5. Download the Source Code

This was an article on Apache Kafka Groupid vs Consumerid.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.