The high watermark indicates the offset of messages that are fully replicated, while the end-of-log offset might be larger if there are newly appended records to the leader partition which are not replicated yet. Consumers can only consume messages up to the high watermark. See this blog post for more details: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
When you are starting your Kafka broker you can define set of properties in conf/server.properties file. This file is just key value property file. One of the properties is auto.create.topics.enable, if it’s set to true (by default) Kafka will create topics automatically when you send messages to non-existing topics. All config options you can find … Read more
You can pipe it in: kafka-console-producer.sh –broker-list localhost:9092 –topic my_topic –new-producer < my_file.txt Found here. From 0.9.0: kafka-console-producer.sh –broker-list localhost:9092 –topic my_topic < my_file.txt
I found out the solution after some research and the solution is here. kafka-console-producer command kafka-console-producer.sh –broker-list localhost:9092 –topic topic-name –property “parse.key=true” –property “key.separator=:” After running this command you will enter in producer console and from there you can send key, value messages. For example key1:value1 key2:value2 key3:value3 For more clarity, I am providing sample … Read more
Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition? Let’s see the following properties of kafka: each partition is consumed by exactly one consumer in the group one consumer in the group can consume more than … Read more
All Kafka brokers must be assigned a broker.id. On startup a broker will create an ephemeral node in Zookeeper with a path of /broker/ids/$id. As the node is ephemeral it will be removed as soon as the broker disconnects, e.g. by shutting down. You can view the list of the ephemeral broker nodes like so: … Read more
Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go … Read more
This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let’s look at an overview of producer components: 1. When a producer is producing a message – It will specify the topic it wants to send the message to, is that right? … Read more