How can I get the last/end offset of a kafka topic partition?
The new consumer is also complicated. //assign the topic consumer.assign(); //seek to end of the topic consumer.seekToEnd(); //the position is the latest offset consumer.position();
kafka-consumer-api
What is the difference in Kafka between a Consumer Group Coordinator and a Consumer Group Leader?
1. What is the difference? The consumer group coordinator is one of the brokers while the group leader is one of the consumer in a consumer group. The group coordinator is nothing but one of the brokers which receives heartbeats (or polling for messages) from all consumers of a consumer group. Every consumer group has … Read more
Kafka Consumer get key value pair
Using out of the box console consumer (I am using Kafka 0.9.0.1) you can only print the key and the value of messages using different formats. To print the key, set the property print.key=true. There is another property key.separator that by default is “\t” (a tab) that you can also change to anything you want. … Read more
How to write a file to Kafka Producer
You can pipe it in: kafka-console-producer.sh –broker-list localhost:9092 –topic my_topic –new-producer < my_file.txt Found here. From 0.9.0: kafka-console-producer.sh –broker-list localhost:9092 –topic my_topic < my_file.txt
What is the difference between kafka earliest and latest offset values
When a consumer joins a consumer group it will fetch the last committed offset so it will restart to read from 5, 6, 7 if before crashing it committed the latest offset (so 4). The earliest and latest values for the auto.offset.reset property is used when a consumer starts but there is no committed offset … Read more
How to send key, value messages with the kafka console producer
I found out the solution after some research and the solution is here. kafka-console-producer command kafka-console-producer.sh –broker-list localhost:9092 –topic topic-name –property “parse.key=true” –property “key.separator=:” After running this command you will enter in producer console and from there you can send key, value messages. For example key1:value1 key2:value2 key3:value3 For more clarity, I am providing sample … Read more
Topics, partitions and keys
Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition? Let’s see the following properties of kafka: each partition is consumed by exactly one consumer in the group one consumer in the group can consume more than … Read more
Delete message after consuming it in KAFKA
In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability. Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option … Read more
difference between groupid and consumerid in Kafka consumer
Consumers groups is a Kafka abstraction that enables supporting both point-to-point and publish/subscribe messaging. A consumer can join a consumer group (let us say group_1) by setting its group.id to group_1. Consumer groups is also a way of supporting parallel consumption of the data i.e. different consumers of the same consumer group consume data in … Read more
How to read data using Kafka Consumer API from beginning?
This works with the 0.9.x consumer. Basically when you create a consumer, you need to assign a consumer group id to this consumer using the property ConsumerConfig.GROUP_ID_CONFIG. Generate the consumer group id randomly every time you start the consumer doing something like this properties.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString()); (properties is an instance of java.util.Properties that you will pass … Read more