Multi Node Kafka Cluster Setup Using Docker

This is a brief guide on a Multi Node Kafka Cluster Setup using Windows Docker:

Setting Up a Three Node Kafka Cluster

1- Start ZooKeeper and Kafka using Docker Compose up command

docker-compose up

2- In another terminal window, go to the same directory (kafka-cluster). Before we move on, let’s make sure the services are up and running

docker ps

3- Check the ZooKeeper logs to verify that ZooKeeper is healthy. For example, for service zookeeper-1

docker logs <zookeeper-1_containerId>

** repeat this step to varify the rest of the zookeeper containers

4- Verify that ZK ensemble is ready

docker run –net=host –rm confluentinc/cp-zookeeper:3.3.1 bash -c “echo stat | nc localhost <ZOOKEEPER_CLIENT_PORT> | grep Mode”

** repeat this step to varify the rest of the Zookeeper containers

Output: You should see one leader and two follower:

Mode: follower
Mode: leader
Mode: follower

5- Check the logs to see the broker has booted up successfully.

docker logs <kafka-1_containerId>
docker logs <kafka-2_containerId>
docker logs <kafka-3_containerId>

6- Test that the broker is working as expected.
Now that the brokers are up, we’ll test that they’re working as expected by creating a topic.

docker run –net=host –rm confluentinc/cp-kafka:latest kafka-topics –create –topic bar –partitions 3 –replication-factor 3 –if-not-exists –zookeeper localhost:32181

You should see the following output:
Created topic “bar”

7- Now verify that the topic is created successfully by describing the topic.

docker run –net=host –rm confluentinc/cp-kafka:latest kafka-topics –describe –topic bar –zookeeper localhost:32181

8- Next, we’ll try generating some data to the bar topic we just created.

docker run –net=host –rm confluentinc/cp-kafka:latest bash -c “seq 42 | kafka-console-producer –broker-list localhost:29092 –topic bar && echo ‘Produced 42 messages.'”

The command above will pass 42 integers using the Console Producer that is shipped with Kafka.
As a result, you should see something like this in your terminal:

Produced 42 messages.

9- It looked like things were successfully written, but let’s try reading the messages back using the Console Consumer and make sure they’re all accounted for.

docker run –net=host –rm confluentinc/cp-kafka:latest kafka-console-consumer –bootstrap-server localhost:29092 –topic bar –new-consumer –from-beginning –max-messages 42

**it might take some time for this command to return data. Kafka has to create the __consumers_offset topic behind the scenes when you consume data for the first time and this may take some time.

reference: https://docs.confluent.io/current/installation/docker/docs/tutorials/clustered-deployment.html

Download your Free Big Data Project Template Here


Hassan Askari

Comment (1)

  1. online timer extension
    December 2, 2017

    Wonderful goods from you, man. I’ve understand your stuff previous to
    and you are just extremely great. I actually like what you’ve acquired here, certainly like what you are stating and the way in which you
    say it. You make it enjoyable and you still take care
    of to keep it wise. I can not wait to read far more from you.

    This is really a tremendous site.

Comments are closed.