Kafka Partitions and Consumer Groups

Text Views: 45 • Nov 13, 2026 • Never expires
# Kafka Partitions and Consumer Groups for System Design Interviews

Kafka shows up in many designs: activity feeds, metrics pipelines, order processing. Knowing how partitions and consumer groups work lets you answer the follow-up questions with confidence.

## Topics and partitions

A topic is split into partitions. Each partition is an append-only, ordered log stored on a leader broker and replicated to followers. Ordering is guaranteed only within a partition, never across the whole topic.

## Choosing the partition key

Messages with the same key go to the same partition, so choose the key that needs ordering: order_id for order events, user_id for a user's activity. A skewed key (one celebrity user, one huge tenant) creates a hot partition; mitigate by salting the key or giving the big tenant its own topic.

## Consumer groups

Within a consumer group, each partition is read by exactly one consumer. That means:
- Maximum parallelism of a group = number of partitions. Ten partitions and twenty consumers leaves ten idle.
- Different groups read the same topic independently, each with its own offsets: one group feeds search indexing, another feeds analytics.

Pick the partition count for future peak throughput, because increasing it later remaps keys to different partitions and breaks per-key ordering for a while.

## Offsets and delivery

Consumers commit offsets. Commit after processing gives at-least-once delivery (duplicates on crash, so make processing idempotent). Commit before processing gives at-most-once.

## Rebalancing

When consumers join or leave, partitions are reassigned. Processing pauses briefly; cooperative rebalancing and static membership reduce the pause. Long processing that exceeds max.poll.interval causes the consumer to be kicked out, so keep per-message work short or hand it off.

## Retention and replay

Kafka keeps messages for a retention period whether or not they were consumed, so a new service can replay history, and a bug fix can be followed by reprocessing from an older offset.

## Interview tip

Say the partition key out loud and why. Most Kafka follow-ups (ordering, hot spots, scaling consumers) come back to that choice.

Disclosure: I'm with Linkjob AI (https://www.linkjob.ai/), an AI interview copilot.
🛡️

Ad Blocker Detected

PasteBin.site is free to use thanks to our sponsors. Please consider disabling your ad blocker or whitelisting our site to support our servers.

Learn More