Kafka basics: topics, partitions, offsets and consumer groups explained with CLI examples

Kafka is a distributed log. Producers append messages to it, consumers read them in order and remember where they stopped. Almost every Kafka question comes back to four words: topic, partition, offset, consumer group.

#The four words

Messages are not deleted when read. They stay until the retention time or size is reached, so several groups can read the same topic independently.

#Start a local broker (for learning)

docker run -d --name kafka -p 9092:9092 apache/kafka:latest
docker exec -it kafka bash
cd /opt/kafka/bin

The examples below run inside that container. On other installs, the scripts are in Kafka's bin/ folder.

#Create and describe a topic

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
./kafka-topics.sh --bootstrap-server localhost:9092 --list

On a real cluster use --replication-factor 3 so each partition has copies on three brokers.

#Produce messages, with and without keys

./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
  --property parse.key=true --property key.separator=:
> cust-1:{"order":1001}
> cust-2:{"order":1002}
> cust-1:{"order":1003}

Messages with the same key always go to the same partition, so they stay in order relative to each other. Messages with no key are spread across partitions.

Tip

Ordering is guaranteed only within a partition. If you need all events for one customer in order, use the customer ID as the key.

#Consume

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning \
  --property print.key=true --property print.partition=true --property print.offset=true

You will see each message with its partition and offset. cust-1 messages share a partition.

#Consumer groups

Start a consumer in a group:

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --group billing

Open a second terminal and start another with the same --group billing. Kafka splits the three partitions between the two consumers. Start a fourth consumer in a three-partition topic and it sits idle: a group cannot use more consumers than there are partitions.

A different group (--group analytics) gets its own copy of every message.

#Lag: how far behind is a group?

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing

Growing lag means consumers are slower than producers, or stopped.

#Retention

./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders \
  --alter --add-config retention.ms=604800000      # 7 days
./kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders --describe

If a consumer is down for longer than retention, the messages it missed are gone.

#Choosing partition count

• More partitions = more parallel consumers, but more files, more leader elections and more overhead.
• You can add partitions later but not remove them, and adding them changes which partition a key maps to (breaking per-key ordering for a while).
• Start with what your peak consumer parallelism needs, plus some headroom.

#Beginner mistakes

#Prevent it next time

• Decide keys and partition count when designing the topic, not after.
• Monitor consumer lag per group and alert on growth.
• Use replication factor 3 and min.insync.replicas=2 for production topics.