This is a brief guide on a Multi Node Kafka Cluster Setup using Windows Docker:

—————————————–
Setting Up a Three Node Kafka Cluster
—————————————–

1- Start ZooKeeper and Kafka using Docker Compose up command

docker-compose up

2- In another terminal window, go to the same directory (kafka-cluster). Before we move on, let’s make sure the services are up and running

docker ps

3- Check the ZooKeeper logs to verify that ZooKeeper is healthy. For example, for service zookeeper-1

docker logs <zookeeper-1_containerId>

** repeat this step to varify the rest of the zookeeper containers

4- Verify that ZK ensemble is ready

docker run –net=host –rm confluentinc/cp-zookeeper:3.3.1 bash -c “echo stat | nc localhost <ZOOKEEPER_CLIENT_PORT> | grep Mode”

** repeat this step to varify the rest of the Zookeeper containers

Output: You should see one leader and two follower:

Mode: follower
Mode: leader
Mode: follower

5- Check the logs to see the broker has booted up successfully.

docker logs <kafka-1_containerId>
docker logs <kafka-2_containerId>
docker logs <kafka-3_containerId>

6- Test that the broker is working as expected.
Now that the brokers are up, we’ll test that they’re working as expected by creating a topic.

docker run –net=host –rm confluentinc/cp-kafka:latest kafka-topics –create –topic bar –partitions 3 –replication-factor 3 –if-not-exists –zookeeper localhost:32181

You should see the following output:
Created topic “bar”

7- Now verify that the topic is created successfully by describing the topic.

docker run –net=host –rm confluentinc/cp-kafka:latest kafka-topics –describe –topic bar –zookeeper localhost:32181

8- Next, we’ll try generating some data to the bar topic we just created.

docker run –net=host –rm confluentinc/cp-kafka:latest bash -c “seq 42 | kafka-console-producer –broker-list localhost:29092 –topic bar && echo ‘Produced 42 messages.'”

The command above will pass 42 integers using the Console Producer that is shipped with Kafka.
As a result, you should see something like this in your terminal:

Produced 42 messages.

9- It looked like things were successfully written, but let’s try reading the messages back using the Console Consumer and make sure they’re all accounted for.

docker run –net=host –rm confluentinc/cp-kafka:latest kafka-console-consumer –bootstrap-server localhost:29092 –topic bar –new-consumer –from-beginning –max-messages 42

**it might take some time for this command to return data. Kafka has to create the __consumers_offset topic behind the scenes when you consume data for the first time and this may take some time.

__________________________________________________________________________________________________________
reference: https://docs.confluent.io/current/installation/docker/docs/tutorials/clustered-deployment.html
__________________________________________________________________________________________________________

Download your Free Big Data Project Template Here