Continue the tutorial series about Apache Kafka, today, I’d like to share how to get Apache Kafka installed by using Apache Kafka Docker.

1. Preparation

You first should have latest versions of Docker installed on your machine. If not, you can use virtual machine with VirtualBox or VMware player, and you can try with Vagrant which is very easy to get thing done for you. For any way, you can refer to my related posts.

Install Docker on CentOS 7.X

Install Docker on Ubuntu 16.04, 15.10, 14.04

To me, I’m using Vagrant with an Ubuntu 14.04 box and I got Docker  installed on this VM.

2. Search for Apache Kafka Docker

After getting Docker  installed, we will try to search and pull Apache Kafka Docker from the Docker hub.

2.1. Search for Kafka Docker

We will issue below command to search for any Kafka Docker

The output is similar to below:

We can see a list of Kafka Docker are available on the Docker hub. The ones with highest rating stars are on the top. The highest one is wurstmeister/kafka  with 175 stars.  However, in this tutorial, we will use the ches/kafka Docker which has 37 stars

Another way to look for Kafka Docker is to go to the Docker hub website and search for Kafka keyword. It will show the same results as we have seen above.

Apache Kafka Docker on Docker hub

Apache Kafka Docker on Docker hub

If we click on the DETAILS button, we will see more information about this Kafka Docker such as: Dockerfile, build detail, guidelines, etc.

We also see the source of this Kafka Docker on the Ches Github

3. Start Apache Kafka Docker

Just to remind you that to install Apache Kafka, we need the Zookeeper. We can use the Zookeeper bundled with Kafka or use a separated Zookeeper which can be installed on another node. In this tutorial, we need another Zookeeper Docker run on a separated container. Then we link the Kafka container with Zookeeper container together.

Firstly, we will start Zookeper Docker. We will use the Zookeeper Docker:  jplock/zookeeper, give the container a name: zookeeper, bind the container port 2181 to our host OS port so that we can access that port from the our host OS if needed.

The output on the terminal will be similar to:

The next step, we will start the Kafka Docker, name it: kafka, link it to the above Zookeeper container.

The output is similar to below. Note that for the first time, the Docker daemon will need to pull the image from the Docker hub to local machine, for the next time, it will use the existing image.

When the start completed, we may check whether all the containers were started successfully or not. To do that, we can issue below command:

The output of the command will be

We can see there are 2 containers: zookeeper and kafka, and their status are up.

4. Working with Kafka Docker

This section will include manipulating with Kafka broker by using Kafka CLI (command line interface). You can refer to my previous post for more detail: Apache Kafka Command Line Interface

4.1. Getting the IP addresses and ports of Zookeeper and Kafka Dockers.

After starting the Kafka broker, we may want to try some commands on it. To do that, we need to know the host and port of the Zookeeper or the cluster.

So, we will try to get these by below commands which are described on the Ches Github. First, we will get the IP addresses of the Zookeeper and the Kafka broker first. Note that, these IP addresses are assigned for Docker container automatically when we started them.

So, let try to print out them:

4.2. Create a topic

We will issue the below command to create a topic “test” with 1 partition and 1 replicated.

The output should be similar to:

4.3. Produce messages

Issue below command to produce message to the “test” topic:

The terminal will wait for our input. Let’s input several lines to the terminal, for ex:

4.4. Consume messages

Open another terminal to consume the messages. Note that when we open the new terminal, the variables: ZK_IP, KAFKA_IP will be lost. We will create them again for this terminal.

The next, we will consume messages on the “test” topic:

The output on the terminal should be similar to:

5. Conclusion

We have learned how to setup an Kafka broker by using Apache Kafka Docker. In this tutorial, we just setup for 1 broker. In near future, I’d like to share how to setup a cluster of Kafka brokers by using Kakfa Docker. Note that on above section, we could see some more Apache Kafka Docker with higher stars, but we didn’t use them, the reason is they are a little bit complex or use the older version of Apache Kafka (Kafka Docker of Spotify use Kafka 0.8 currently).

Below are the articles related to Apache Kafka topic. If you’re interested in them, you can refer to the following links:

Apache Kafka Tutorial

Getting started with Apache Kafka 0.9

Apache Kafka 0.9 Java Client API Example

How To Write A Custom Serializer in Apache Kafka

Write An Apache Kafka Custom Partitioner

Apache Kafka Command Line Interface

Apache Kafka Connect Example

Apache Kafka Connect MQTT Source Tutorial

Apache Flume Kafka Source And HDFS Sink Tutorial







0 0 vote
Article Rating