In Kafka is each message replicated across all partitions of a topic?

Partitioning and replication are two different things.

Partitioning is for scalability. A topic is partitioned in one or more partitions distributed on different brokers so that more consumers can connect to these brokers in order to receive messages sent to the same topic but from different partitions. Increasing partitions increases scalability and the possibility to have more consumers to get messages from the same topic. Answering your question, each message sent to a topic comes into only one partition (of the topic itself).

Replication is for fault-tolerance. You can specify a replication factor on topic creation and it means that every partition for that topic is replicated more times on different brokers. One replica is the “leader” where producer sends and consumer gets messages; other replicas are “follower” which have copies of messages from the “leader” replica. If the broker which handles the “leader” replica goes down, one of the “follower” becomes leader.

Leave a Comment