Kafka Partitions and Consumer Groups
# Kafka Partitions and Consumer Groups for System Design Interviews
Kafka shows up in many designs: activity feeds, metrics pipelines, order processing. Knowing how partitions and consumer groups work lets you answer the follow-up questions with confidence.
## Topics and partitions
A topic is split into partitions. Each partition is an append-only, ordered log stored on a leader broker and replicated to followers. Ordering is guaranteed only within a partition, never across the whole topic.
## Choosing the partition key
Messages with the same key go to the same partition, so choose the key that needs ordering: order_id for order events, user_id for a user's activity. A skewed key (one celebrity user, one huge tenant) creates a hot partition; mitigate by salting the key or giving the big tenant its own topic.
## Consumer groups
Within a consumer group, each partition is read by exactly one consumer. That means:
- Maximum parallelism of a group = number of partitions. Ten partitions and twenty consumers leaves ten idle.
- Different groups read the same topic independently, each with its own offsets: one group feeds search indexing, another feeds analytics.
Pick the partition count for future peak throughput, because increasing it later remaps keys to different partitions and breaks per-key ordering for a while.
## Offsets and delivery
Consumers commit offsets. Commit after processing gives at-least-once delivery (duplicates on crash, so make processing idempotent). Commit before processing gives at-most-once.
## Rebalancing
When consumers join or leave, partitions are reassigned. Processing pauses briefly; cooperative rebalancing and static membership reduce the pause. Long processing that exceeds max.poll.interval causes the consumer to be kicked out, so keep per-message work short or hand it off.
## Retention and replay
Kafka keeps messages for a retention period whether or not they were consumed, so a new service can replay history, and a bug fix can be followed by reprocessing from an older offset.
## Interview tip
Say the partition key out loud and why. Most Kafka follow-ups (ordering, hot spots, scaling consumers) come back to that choice.
Disclosure: I'm with Linkjob AI (https://www.linkjob.ai/), an AI interview copilot.
Featured
Advertisement
Category
Action