It is the url of one of the Kafka brokers which you give to fetch the initial metadata about your Kafka cluster. The metadata consists of the topics, their partitions, the leader brokers for those partitions etc. Depending upon this metadata your producer or consumer produces or consumes the data. You can have multiple bootstrap-servers … Read more
If you have altered a topic and want to view the topic configuration the following command will be helpful kafka-topics.sh –zookeeper localhost:2181 –describe –topics-with-overrides This will describe only the topics along with configurations that have configurations set that differ from the cluster defaults. If you want to view the configurations for all topic Either you … Read more
1. What is the difference? The consumer group coordinator is one of the brokers while the group leader is one of the consumer in a consumer group. The group coordinator is nothing but one of the brokers which receives heartbeats (or polling for messages) from all consumers of a consumer group. Every consumer group has … Read more
Using out of the box console consumer (I am using Kafka 0.9.0.1) you can only print the key and the value of messages using different formats. To print the key, set the property print.key=true. There is another property key.separator that by default is “\t” (a tab) that you can also change to anything you want. … Read more
By default, –reset-offsets just prints the result of the operation. To actually perform the operation you need to add –execute to your command: kafka-consumer-groups.bat –bootstrap-server kafka-host:9092 –group my-group –reset-offsets –to-earliest –all-topics –execute
“Which one is your favorite” is probably not a question appropriate to this forum since it is very subjective. I can, however, describe the differences. Confluent Platform includes Apache Kafka, so you will get that in any case. It also includes few things that can make Apache Kafka easier to use: Clients in Python, C, … Read more
When a consumer joins a consumer group it will fetch the last committed offset so it will restart to read from 5, 6, 7 if before crashing it committed the latest offset (so 4). The earliest and latest values for the auto.offset.reset property is used when a consumer starts but there is no committed offset … Read more
__consumer_offsets is used to store information about committed offsets for each topic:partition per group of consumers (groupID). It is compacted topic, so data will be periodically compressed and only latest offsets information available. _schema – is not a default kafka topic (at least at kafka 8,9). It is added by Confluent. See more: Confluent Schema … Read more
I found out the solution after some research and the solution is here. kafka-console-producer command kafka-console-producer.sh –broker-list localhost:9092 –topic topic-name –property “parse.key=true” –property “key.separator=:” After running this command you will enter in producer console and from there you can send key, value messages. For example key1:value1 key2:value2 key3:value3 For more clarity, I am providing sample … Read more
Newer versions(2.2+) of Kafka no longer requires ZooKeeper connection string –zookeeper localhost:2181 It throws the following exception while creating a topic Exception in thread “main” joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option Instead, add Kafka Broker –bootstrap-server localhost:9092 connection string. ./kafka-topics.sh –create –topic test-topic –bootstrap-server localhost:9092 –replication-factor 1 –partitions 4