Software Development
Achieving Strong Consistency in Distributed Spring Boot Applications with Redpanda
Redpanda is a modern, Kafka-compatible streaming platform designed for high performance and low latency. As a drop-in replacement for Apache Kafka, it offers several advantages:
- Simplified architecture (single binary, no ZooKeeper dependency)
- Improved performance (lower latency, higher throughput)
- Full Kafka API compatibility (works with existing Kafka clients and tools)
- Strong consistency guarantees out of the box
1. Strong Consistency Patterns with Spring Boot and Redpanda
1. Transactional Messaging Pattern
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | public class KafkaConfig { @Bean public ProducerFactory<String, String> producerFactory() { Map<String, Object> configProps = new HashMap<>(); configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "redpanda:9092"); configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); configProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "tx-1"); configProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); return new DefaultKafkaProducerFactory<>(configProps); } @Bean public KafkaTransactionManager<String, String> kafkaTransactionManager( ProducerFactory<String, String> producerFactory) { return new KafkaTransactionManager<>(producerFactory); }}@Servicepublic class OrderService { @Autowired private KafkaTemplate<String, String> kafkaTemplate; @Autowired private OrderRepository orderRepository; @Transactional("kafkaTransactionManager") public void processOrder(Order order) { // Database operation orderRepository.save(order); // Kafka operation in the same transaction kafkaTemplate.send("orders", order.getId(), order.toJson()); }} |
2. Outbox Pattern for Strong Consistency
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | @Configuration@Entitypublic class OutboxEvent { @Id @GeneratedValue private Long id; private String aggregateType; private String aggregateId; private String eventType; private String payload; private LocalDateTime timestamp; private boolean published;}@Servicepublic class OrderService { @Transactional public void createOrder(Order order) { // 1. Save order to database orderRepository.save(order); // 2. Save event to outbox table in the same transaction OutboxEvent event = new OutboxEvent(); event.setAggregateType("Order"); event.setAggregateId(order.getId()); event.setEventType("OrderCreated"); event.setPayload(order.toJson()); event.setTimestamp(LocalDateTime.now()); event.setPublished(false); outboxRepository.save(event); }}@Componentpublic class OutboxProcessor { @Scheduled(fixedRate = 1000) @Transactional public void processOutbox() { List<OutboxEvent> events = outboxRepository.findByPublishedFalse(); events.forEach(event -> { kafkaTemplate.send("orders", event.getAggregateId(), event.getPayload()); event.setPublished(true); outboxRepository.save(event); }); }} |
2. Redpanda-Specific Optimizations
1. Leveraging Redpanda’s Linearizable Reads
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 | @Configurationpublic class ConsumerConfig { @Bean public ConsumerFactory<String, String> consumerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "redpanda:9092"); props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-group"); props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); // Redpanda specific optimization props.put("enable.linearizable.reads", "true"); return new DefaultKafkaConsumerFactory<>(props); }} |
2. High-Performance Consumer Configuration
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 | @KafkaListener(topics = "orders", groupId = "order-group")public void listen(Order order) { // Process order with strong consistency guarantees orderProcessingService.process(order);}@Beanpublic ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() { ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(consumerFactory()); factory.setConcurrency(4); // Tune based on Redpanda partition count factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE); return factory;} |
3. Monitoring and Observability
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 | @Configurationpublic class MetricsConfig { @Bean public MicrometerProducerListener<String, String> micrometerProducerListener(MeterRegistry registry) { return new MicrometerProducerListener<>(registry); } @Bean public MicrometerConsumerListener<String, String> micrometerConsumerListener(MeterRegistry registry) { return new MicrometerConsumerListener<>(registry); }}// In application.properties:management.endpoints.web.exposure.include=health,info,metricsmanagement.metrics.export.prometheus.enabled=true |
4. Deployment Considerations
- Redpanda Cluster Sizing: Start with 3 nodes for production deployments
- Resource Allocation: Redpanda benefits from more CPU than Kafka
- Storage: Use direct-attached storage for best performance
- Networking: Low-latency networking is crucial for strong consistency
5. Comparison with Traditional Kafka
| Feature | Redpanda | Apache Kafka |
|---|---|---|
| Architecture | Single binary, no ZooKeeper | Requires ZooKeeper |
| Performance | Lower latency, higher throughput | Good performance with tuning |
| Consistency | Strong by default | Configurable |
| Operational Complexity | Simplified | More complex |
| Compatibility | Full Kafka API compatibility | N/A |
6. Conclusion
Redpanda provides an excellent alternative to Kafka for Spring Boot applications requiring strong consistency in distributed environments. Its simplified architecture and performance characteristics make it particularly suitable for high-throughput, low-latency systems while maintaining the familiar Kafka API that Spring developers are accustomed to.
