Apache Kafka, a distributed messaging system, is gaining very much attraction today. Spring is a very popular framework for Java developer. Getting Apache Kafka to work with Spring smoothly will be a very good thing for many Java developers. In this Spring Kafka tutorial, we will get to know about Spring Kafka, the Spring for Kafka, how to use KafkaTemplate to produce messages to Kafka brokers, and how to use “listener container” to consume messages from Kafka as well.

Spring Kafka Tutorial

Spring Kafka

1. Basic Spring Kafka

In this section, we will get through some various components that comprise the Spring Kafka

1.1. Sending Messages

In the same style with JmsTemplate or JdbcTemplate, Spring Kafka provides us a “template” for Kafka called KafkaTemplate. It wraps a Kafka producer and provides us many convenience methods to send messages to Kafka brokers. Below are some of those methods

You can find more information about the template here.

1.2. Receiving Messages

To receive the messages, we have to:

  • Configure the MessageListenerContainer
  • Provide a Message Listener, or use the @KafkaListener annotation

1.2.1. MessageListenserContainer

There are 2 implementation of the MessageListenserContainer in Spring Kafka:

  • KafkaMessageListenerContainer
  • ConcurrentMessageListenerContainer

The KafkaMessageListenerContainer allows us to consume messages from Kafka topics in a single thread while the ConcurrentMessageListenerContainer allows us to consume messages in multi-threaded style.

1.2.1. @KafkaListener annotation

Spring Kafka provides @KafkaListener annotation marks a method to be the target of a Kafka message listener on the specified topics, for example:

2. Spring Kafka Example

In this part of the Spring Kafka tutorial, we will get through an example which use Spring Kafka API to send and receive messages to/from Kafka topics.

2.1. Preparation

  • Apache Kafka 0.9/0.10 broker installed on local machine or remote. You can refer to this link for setting up.
  • JDK 7/8 installed on your development PC.
  • IDE (Eclipse or IntelliJ)
  • Build tool (Maven  or Gradle)

2.2. Source Code Structure

The source code was added to the Github or you can download it here

After having the source code, you can import the source code into Eclipse and run the test.

To import:

  • Menu File –> Import –> Maven –> Existing Maven Projects
  • Browse to your source code location

Click Finish button to finish the importing

2.3. Library dependencies

The only dependency we need is spring-kafka

However, this example is built on Spring Boot, and we also need to run JUnit tests, the dependencies are more as following:

2.4. Classes Descriptions

2.4.1. SpringKafkaExampleApplication.java

Entry point for Spring Boot application.

2.4.2. ProducerConfig.java

This class contains configuration for KafkaTemplate. We need to specify the properties for the Kafka producer.

2.4.3. KafkaConsumerConfig

Defines the configuration to consume messages from Kafka broker.

We need to define the KafkaListenerContainerFactory which will create KafkaListenerContainer. In this example, we will use ConcurrentKafkaListenerContainer which can consume messages in multi-threaded style.

Note that the broker address is: localhost:9092

We also define a Listener which has a method annotated with @KafkaListener annotation to “listen” and process messages.

This listener will have id = foo and group = group1 and will consume the message on the topic “topic1“. It also has a countDownLatch1 variable used for unit test purpose below.

2.4.4. SpringKafkaExampleApplicationTests

Unit test class to test our example.

To send the messages to the Kafka topic, we inject the kafkaTemplate bean(@autowire). We also inject the listener (@autowire) to verify the result.

We register a ListenableFutureCallback with the kafkaTemplate to verify whether the messages are sent to the topic “Topic1” successfully or not.

2.4.5 Run the application

Step 1. Make sure the Kafka broker is running on localhost:9092

Step 2. Open the SpringKafkaExampleApplicationTests.java, Right click –> Run As –> JUnit Test or use the shortcut: Alt+Shift+x, t to start the test.

3. Summary

We have just gotten through a Spring Kafka tutorial. Spring Kafka supports us in integrating Kafka with our Spring application easily and a simple example as well. In future posts, I’s like to provide more examples on using Spring Kafka such as: multi-threaded consumers, multiple KafkaListenerContainerFactory, etc. Recently, I have some more article on Apache Kafka. If you’re interested in, you can refer to the following links:

Apache Kafka Tutorial

Getting started with Apache Kafka 0.9

Using Apache Kafka Docker

Create Multi-threaded Apache Kafka Consumer

Apache Kafka Command Line Interface

Write An Apache Kafka Custom Partitioner

How To Write A Custom Serializer in Apache Kafka

Apache Kafka Connect Example

Apache Kafka Connect MQTT Source Tutorial

Apache Flume Kafka Source And HDFS Sink Tutorial

Spring Kafka – Multi-threaded Message Consumption


0 0 vote
Article Rating