Continue the tutorial series about Apache Kafka, today, I’d like to share how to get Apache Kafka installed by using Apache Kafka Docker.
1. Preparation
You first should have latest versions of Docker installed on your machine. If not, you can use virtual machine with VirtualBox or VMware player, and you can try with Vagrant which is very easy to get thing done for you. For any way, you can refer to my related posts.
Install Docker on Ubuntu 16.04, 15.10, 14.04
To me, I’m using Vagrant with an Ubuntu 14.04 box and I got Docker installed on this VM.
2. Search for Apache Kafka Docker
After getting Docker installed, we will try to search and pull Apache Kafka Docker from the Docker hub.
2.1. Search for Kafka Docker
We will issue below command to search for any Kafka Docker
|
1 |
docker search kafka |
The output is similar to below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
NAME DESCRIPTION STARS wurstmeister/kafka Multi-Broker kafka 0.8.x image 175 spotify/kafka A simple docker image with both Kafka and ... 137 sheepkiller/kafka-manager kafka-manager 40 ches/kafka Apache Kafka. Tagged versions. JMX. Cluste... 37 digitalwonderland/kafka Latest Kafka - clusterable 7 cgswong/confluent-kafka Confluent Platform Kafka 4 elevy/kafka Kafka configured to execute in a cluster w... 3 qnib/kafka 2 dockerkafka/kafka Kafka; for further details: http://dockerk... 2 blacktop/kafka Alpine Linux based Kafka Docker Image 1 debezium/kafka Kafka image required when running the Debe... 1 kousha/kafka 1 tianon/kafka Apache Kafka is publish-subscribe messagin... 0 aarongdocker/kafka An Apache Kafka Broker 0 benchflow/kafka BenchFlow Kafka with topics 0 harisekhon/kafka Apache Kafka (tags 0.8 - 0.10 with scala 2... 0 coco/kafka-proxy kafka rest proxy 0 qnib/kafka-monitor Docker image holding KafkaOffsetMonitor 0 qnib/kafka-manager Docker Image hooking into ZK and manages K... 0 itrust/kafka Apache Kafka 0 frontporch/kafka Kafka 0 eatcode/kafka Docker build for Apache Kafka 0 colisweb/docker-kafka Simple Apache Kafka on Docker 0 voidbridge/kafka Kafka 0 appcelerator/kafka Kafka image builder using only environment... 0 |
We can see a list of Kafka Docker are available on the Docker hub. The ones with highest rating stars are on the top. The highest one is wurstmeister/kafka with 175 stars. However, in this tutorial, we will use the ches/kafka Docker which has 37 stars
Another way to look for Kafka Docker is to go to the Docker hub website and search for Kafka keyword. It will show the same results as we have seen above.
Apache Kafka Docker on Docker hub
If we click on the DETAILS button, we will see more information about this Kafka Docker such as: Dockerfile, build detail, guidelines, etc.
We also see the source of this Kafka Docker on the Ches Github
3. Start Apache Kafka Docker
Just to remind you that to install Apache Kafka, we need the Zookeeper. We can use the Zookeeper bundled with Kafka or use a separated Zookeeper which can be installed on another node. In this tutorial, we need another Zookeeper Docker run on a separated container. Then we link the Kafka container with Zookeeper container together.
Firstly, we will start Zookeper Docker. We will use the Zookeeper Docker: jplock/zookeeper, give the container a name: zookeeper, bind the container port 2181 to our host OS port so that we can access that port from the our host OS if needed.
|
1 |
docker run -d -p 2181:2181 --name zookeeper jplock/zookeeper |
The output on the terminal will be similar to:
|
1 2 3 4 5 6 7 8 9 10 11 |
vagrant@geeksgn:~$ docker run -d --name zookeeper jplock/zookeeper Unable to find image 'jplock/zookeeper:latest' locally latest: Pulling from jplock/zookeeper d0ca440e8637: Pull complete a3ed95caeb02: Pull complete 05bee9feaa04: Pull complete 1ac73445c6b1: Pull complete a5d98b9fadfc: Pull complete Digest: sha256:e416e0a98ccb8e06d2d712f25bdf85627734c9c56f65b4974ef352d6b6ce3894 Status: Downloaded newer image for jplock/zookeeper:latest 71e5e96cc52755185603a12168c5b33fca89880ac9411d0d505b270e39545763 |
The next step, we will start the Kafka Docker, name it: kafka, link it to the above Zookeeper container.
|
1 |
docker run -d --name kafka --link zookeeper:zookeeper ches/kafka |
The output is similar to below. Note that for the first time, the Docker daemon will need to pull the image from the Docker hub to local machine, for the next time, it will use the existing image.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Unable to find image 'ches/kafka:latest' locally latest: Pulling from ches/kafka 012a7829fd3f: Downloading [==========================> ] 34.97 MB/65.79 MB 41158247dd50: Download complete 916b974d99af: Download complete a3ed95caeb02: Download complete 8ec2accd3368: Downloading [==========> ] 37.29 MB/186.2 MB 253fd32218f3: Download complete d7de66b60976: Downloading [============================> ] 13.51 MB/23.74 MB 4a31badbf864: Waiting f0b87da80939: Waiting d2dc03daf63a: Waiting 99e10bf6620f: Waiting a1576e01119d: Waiting c4fffdc533e8: Waiting a1ff90d481bf: Waiting |
When the start completed, we may check whether all the containers were started successfully or not. To do that, we can issue below command:
|
1 |
docker ps |
The output of the command will be
|
1 2 3 |
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9b25c096dabb ches/kafka "/start.sh" About a minute ago Up About a minute 7203/tcp, 9092/tcp kafka 717cfa653ced jplock/zookeeper "/opt/zookeeper/bin/z" About a minute ago Up About a minute 2181/tcp, 2888/tcp, 3888/tcp zookeeper |
We can see there are 2 containers: zookeeper and kafka, and their status are up.
4. Working with Kafka Docker
This section will include manipulating with Kafka broker by using Kafka CLI (command line interface). You can refer to my previous post for more detail: Apache Kafka Command Line Interface
4.1. Getting the IP addresses and ports of Zookeeper and Kafka Dockers.
After starting the Kafka broker, we may want to try some commands on it. To do that, we need to know the host and port of the Zookeeper or the cluster.
So, we will try to get these by below commands which are described on the Ches Github. First, we will get the IP addresses of the Zookeeper and the Kafka broker first. Note that, these IP addresses are assigned for Docker container automatically when we started them.
|
1 2 |
ZK_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' zookeeper) KAFKA_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' kafka) |
So, let try to print out them:
|
1 2 |
echo $ZK_IP, $KAFKA_IP 172.17.0.2, 172.17.0.3 |
4.2. Create a topic
We will issue the below command to create a topic “test” with 1 partition and 1 replicated.
|
1 2 |
docker run --rm ches/kafka \ > kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper $ZK_IP:2181 |
The output should be similar to:
|
1 |
Created topic "test". |
4.3. Produce messages
Issue below command to produce message to the “test” topic:
|
1 2 |
docker run --rm --interactive ches/kafka \ > kafka-console-producer.sh --topic test --broker-list $KAFKA_IP:9092 |
The terminal will wait for our input. Let’s input several lines to the terminal, for ex:
|
1 2 3 |
hello halo salut |
4.4. Consume messages
Open another terminal to consume the messages. Note that when we open the new terminal, the variables: ZK_IP, KAFKA_IP will be lost. We will create them again for this terminal.
|
1 2 |
ZK_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' zookeeper) KAFKA_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' kafka) |
The next, we will consume messages on the “test” topic:
|
1 2 |
docker run --rm ches/kafka kafka-console-consumer.sh \ > --topic test --from-beginning --zookeeper $ZK_IP:2181 |
The output on the terminal should be similar to:
|
1 2 3 |
hello halo salut |
5. Conclusion
We have learned how to setup an Kafka broker by using Apache Kafka Docker. In this tutorial, we just setup for 1 broker. In near future, I’d like to share how to setup a cluster of Kafka brokers by using Kakfa Docker. Note that on above section, we could see some more Apache Kafka Docker with higher stars, but we didn’t use them, the reason is they are a little bit complex or use the older version of Apache Kafka (Kafka Docker of Spotify use Kafka 0.8 currently).
Below are the articles related to Apache Kafka topic. If you’re interested in them, you can refer to the following links:
Getting started with Apache Kafka 0.9
Apache Kafka 0.9 Java Client API Example
How To Write A Custom Serializer in Apache Kafka
Write An Apache Kafka Custom Partitioner
Apache Kafka Command Line Interface
Apache Kafka Connect MQTT Source Tutorial
Apache Flume Kafka Source And HDFS Sink Tutorial
Its great and easy to follow. Thanks for writing this. Also wanted to know did you explored on kafka cluster setup using containers. Please share the link if possible. Thank you again.
Hi,
I’ve tested this solution it works 😀 great tutorial btw.
I have question. Is there an easy way to send messages from Java with this current setup?
I’ve tried with https://howtoprogram.xyz/2016/05/02/apache-kafka-0-9-java-client-api-example/ code but consumer not showing messages :/.
Tnx
Milos
Hey, thanks for useful tutorial. I also want to setup the cluster using containers. If you have tried that, plz share the link for same.