Running Kafka Without Zookeeper (production ready) PART 1

In this article, i’ll share my experience on how to run Kafka without zookeeper for production use (Cluster mode).

Kafka users can now use Kafka without Zookeeper dependency, which was released with Apache Kafka 2.8.0, and was made production-ready with Apache Kafka 3.3.0. More details here

“Apache Kafka Raft (KRaft) is the consensus protocol that was introduced to remove Apache Kafka’s dependency on ZooKeeper for metadata management. This greatly simplifies Kafka’s architecture by consolidating responsibility for metadata into Kafka itself, rather than splitting it between two different systems: ZooKeeper and Kafka. KRaft mode makes use of a new quorum controller service in Kafka which replaces the previous controller and makes use of an event-based variant of the Raft consensus protocol.”

However, in this article, I will not delve deeper into Kafka’s Kraft architecture. Instead, I would like to share my experience of deploying Kafka in Kraft mode on Kubernetes using Helm.

I’m using GKE 1.24.9-gke.3200 with 3 nodes (e2-medium) and Cloud Shell to run the command (for testing purpose)

Step 1 prepare helm repo

helm repo add bitnami https://charts.bitnami.com/bitnami

Step 2 prepare values.yaml

#Configure pod-0 as controller
extraEnvVars:
  - name: KAFKA_CFG_BROKER_ID #
    value: "1"
#set replica count to 3 for HA
replicaCount: 3
minBrokerId: 1

#enable kraft mode
kraft:
  enabled: true
  processRoles: broker,controller
  controllerListenerNames: CONTROLLER
  clusterId: "k2yipv1sRue7z2_Y3o976A"
  #Configure pod-1 and pod-2 as Quorum Voters
  controllerQuorumVoters: "1@kafka-kraft-1.kafka-kraft-headless.default.svc.cluster.local:9095,2@kafka--kraft2.kafka-kraft-headless.default.svc.cluster.local:9095"

#disable zookeeper
zookeeper:
  enabled: false

#enable promtheus Kafka exporter
metrics:
  kafka:
    enabled: true

Step 3 Deploy Kafka

#Deploy Kafka using this command
helm install kafka-kraft bitnami/kafka --values values.yaml

#Make sure all pods are running
kubectl get pod
NAME                                    READY   STATUS    RESTARTS      AGE
kafka-kraft-0                           1/1     Running   0             63s
kafka-kraft-1                           1/1     Running   0             63s
kafka-kraft-2                           1/1     Running   0             63s
kafka-kraft-exporter-6647b8556d-d4c9v   1/1     Running   2 (53s ago)   64s

Step 4 Test

#Create pod to used as kafka client
kubectl run kafka-kraft-client --restart='Never' \
  --image docker.io/bitnami/kafka:3.4.0-debian-11-r6 \
  --namespace default --command -- sleep infinity

#Exec into the pod
kubectl exec --tty -i kafka-kraft-client --namespace default -- bash

#Create Kafka topic
kafka-topics.sh --bootstrap-server \
  kafka-kraft.default.svc.cluster.local:9092 \
  --create --topic book-order --partitions 3 --replication-factor 3

#Test Publishing and Consuming Some Data
kafka-console-producer.sh --broker-list \
  kafka-kraft-0.kafka-kraft-headless.default.svc.cluster.local:9092,kafka-kraft-1.kafka-kraft-headless.default.svc.cluster.local:9092,kafka-kraft-2.kafka-kraft-headless.default.svc.cluster.local:9092 \
  --topic book-order
##Type in new line for test
>products: book, title: Atomic Habbit

#Consume the message
kafka-console-consumer.sh \
  --bootstrap-server kafka-kraft.default.svc.cluster.local:9092 \
  --topic book-order \
  --from-beginning
##We should see the test message
products: book, title: Atomic Habbit

#You can try to delete pod-1 or pod-2 and try publish & consume message to test HA

Stay tune for PART 2 !

Reference KRaft: Apache Kafka Without ZooKeeper