How does Kafka store offsets for each topic? Kafka has moved the offset storage from zookeeper to kafka brokers. The reason is below: Zookeeper is not a good way to service a high-write load such as offset updates because zookeeper routes each write though every node and hence has no ability to partition or otherwise … Read more
I came across this question when trying to also consume from the __consumer_offsets topic. I managed to figure it out for different Kafka versions and thought I’d share what I’d found For Kafka 0.8.2.x Note: This uses Zookeeper connection #Create consumer config echo “exclude.internal.topics=false” > /tmp/consumer.config #Consume all offsets ./kafka-console-consumer.sh –consumer.config /tmp/consumer.config \ –formatter “kafka.server.OffsetManager\$OffsetsMessageFormatter” … Read more
Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines. If all the consumer instances have the same consumer group, then the records will effectively be load balanced … Read more
The broker tells the client which hostname should be used to produce/consume messages. By default Kafka uses the hostname of the system it runs on. If this hostname can not be resolved by the client side you get this exception. You can try setting advertised.host.name in the Kafka configuration to an hostname/address which the clients … Read more
if I don’t set any group id in the Consumer Properties, what group id will the Kafka Consumer be given? The kafka consumer will not have any consumer group. Instead you will get this error : The configured groupId is invalid Is there a single default value? Yes, you can see the consumer.properties file of … Read more
The high watermark indicates the offset of messages that are fully replicated, while the end-of-log offset might be larger if there are newly appended records to the leader partition which are not replicated yet. Consumers can only consume messages up to the high watermark. See this blog post for more details: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
When you are starting your Kafka broker you can define set of properties in conf/server.properties file. This file is just key value property file. One of the properties is auto.create.topics.enable, if it’s set to true (by default) Kafka will create topics automatically when you send messages to non-existing topics. All config options you can find … Read more
We can subscribe for multiple topic using following API : consumer.subscribe(Arrays.asList(topic1,topic2), ConsumerRebalanceListener obj) Consumer has the topic info and we can commit using consumer.commitAsync or consumer.commitSync() by creating OffsetAndMetadata object as follows. ConsumerRecords<String, String> records = consumer.poll(long value); for (TopicPartition partition : records.partitions()) { List<ConsumerRecord<String, String>> partitionRecords = records.records(partition); for (ConsumerRecord<String, String> record : partitionRecords) … Read more
Depends on what you mean by “blocked”. If you mean “are existing connections closed when rebalance is triggered” then the answer is yes. The current Kafka’s rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) … Read more
I my MAC box I was facing the same issue of console-consumer not consuming any messages when used the command kafka-console-consumer –bootstrap-server localhost:9095 –from-beginning –topic my-replicated-topic But when I tried with kafka-console-consumer –bootstrap-server localhost:9095 –from-beginning –topic my-replicated-topic –partition 0 It happily lists the messages sent. Is this a bug in Kafka 1.10.11?