Here are some commands often be used when we work with Apache Kafka command line interface (CLI).
1. Start the Kafka server
We needs 2 steps:
1.1 Start the ZooKeeper
|
1 |
bin/zookeeper-server-start.sh config/zookeeper.properties |
1.2. Start the Kafka server
|
1 |
bin/kafka-server-start.sh config/server.properties |
2. List all topics
|
1 |
bin/kafka-topics.sh --zookeeper localhost:2181 --list |
|
1 2 |
hello-topic my-replicated-topic |
3. Create a topic
|
1 |
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-topic |
|
1 |
Created topic "hello-topic". |
4. Describe a topic
|
1 |
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello-topic |
|
1 2 |
Topic:hello-topic PartitionCount:1 ReplicationFactor:1 Configs: Topic: hello-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0 |
5. Publish messages to a topic
One of the most interesting command.
|
1 |
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic |
After this command, you can add any messages to the console, line by line. For ex:
|
1 2 3 4 |
topic Hello Halo Salut |
You can stop the console consumer via Ctrl-C
6. Consume messages
Below command will consume messages from the topic: hello-topic
|
1 |
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic |
The output on my console is:
|
1 2 3 |
Hello Halo Salut |
You can stop the console consumer via Ctrl-C
7. Alter Apache Kafka Topics
7.1. Add more partitions to the topic
Below command will add 10 more partitions to the hello-topic topic. Note that before, the topic has only 1 partition.
|
1 |
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --partitions 11 --topic hello-topic |
The output in my console:
|
1 2 |
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected Adding partitions succeeded! |
7.2. Delete a topic
|
1 |
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic hello-topic |
7.3. Add configurations to the Kafka topic
The general syntax is:
|
1 |
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic kafkatopic --config <key>=<value> |
For example, below command will set the max message size = 128000 bytes for the hello-topic topic.
|
1 |
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --config max.message.bytes=128000 |
Here is an example output:
|
1 2 3 |
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. Going forward, please use kafka-configs.sh for this functionality Updated config for topic "hello-topic". |
7.4. Add configurations to the Kafka topic
To remove above overridden configuration, we can use command:
|
1 |
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --delete-config max.message.bytes |
For the list of configurations, please reference Apache Kafka page
8. Conclusion
We have seen some popular commands that provided by Apache Kafka command line interface. They are very essential when we work with Apache Kafka. I hope this post will bring you a list for easy copying and pasting.
Below are the articles related to Apache Kafka. If you’re interested in them, you can refer to the following links:
Getting started with Apache Kafka 0.9
Apache Kafka 0.9 Java Client API Example
Create Multi-threaded Apache Kafka Consumer
How To Write A Custom Serializer in Apache Kafka
Write An Apache Kafka Custom Partitioner
Apache Kafka Connect MQTT Source Tutorial
Apache Flume Kafka Source And HDFS Sink Tutorial