apache-kafka – Page 6

How to change the number of replicas of a Kafka topic?

February 2, 2023 by Tarik

To increase the number of replicas for a given topic you have to: 1. Specify the extra replicas in a custom reassignment json file For example, you could create increase-replication-factor.json and put this content in it: {“version”:1, “partitions”:[ {“topic”:”signals”,”partition”:0,”replicas”:[0,1,2]}, {“topic”:”signals”,”partition”:1,”replicas”:[0,1,2]}, {“topic”:”signals”,”partition”:2,”replicas”:[0,1,2]} ]} 2. Use the file with the –execute option of the kafka-reassign-partitions tool [or … Read more

Kafka consumer list

November 29, 2022 by Tarik

Use kafka-consumer-groups.sh For example bin/kafka-consumer-groups.sh –list –bootstrap-server localhost:9092 bin/kafka-consumer-groups.sh –describe –group mygroup –bootstrap-server localhost:9092

How to view kafka message

November 26, 2022 by Tarik

You can use console consumer to view messages produced on some topic: bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning

When to use Apache kafka instead of ActiveMQ [closed]

November 24, 2022 by Tarik

Kafka and ActiveMQ may have some overlaps but they were originally designed for different purposes. So comparing them is just like comparing an Apple and an Orange. Kafka Kafka is a distributed streaming platform with very good horizontal scaling capability. It allows applications to process and re-process streamed data on disk. Due to it’s high … Read more

Can multiple Kafka consumers read same message from the partition

November 24, 2022 by Tarik

It depends on Group ID. Suppose you have a topic with 12 partitions. If you have 2 Kafka consumers with the same Group Id, they will both read 6 partitions, meaning they will read different set of partitions = different set of messages. If you have 4 Kafka cosnumers with the same Group Id, each … Read more

How to list all available Kafka brokers in a cluster?

November 23, 2022 by Tarik

This command will give you the list of the active brokers between brackets: ./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids

List all Kafka 0.10 topics using – -zookeeper flag without access to Zookeeper

November 22, 2022 by Tarik

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don’t already have one. If you do not want to install and have a separate zookeeper server, you can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance. Starting the single-node Zookeeper instance: bin/zookeeper-server-start.sh config/zookeeper.properties Starting … Read more

Is there a way to delete all the data from a topic or delete the topic before every run?

November 20, 2022 by Tarik

As I mentioned here Purge Kafka Queue: Tested in Kafka 0.8.2, for the quick-start example: First, Add one line to server.properties file under config folder: delete.topic.enable=true then, you can run this command: bin/kafka-topics.sh –zookeeper localhost:2181 –delete –topic test

Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1

November 14, 2022 by Tarik

Before KIP-62, there is only session.timeout.ms (ie, Kafka 0.10.0 and earlier). max.poll.interval.ms is introduced via KIP-62 (part of Kafka 0.10.1). KIP-62, decouples heartbeats from calls to poll() via a background heartbeat thread, allowing for a longer processing time (ie, time between two consecutive poll()) than heartbeat interval. Assume processing a message takes 1 minute. If … Read more

Delete topic in Kafka 0.8.1.1

November 13, 2022 by Tarik

Deleting topic isn’t always working in 0.8.1.1 Deletion should be working in the next release, 0.8.2 kafka-topics.sh –delete –zookeeper localhost:2181 –topic your_topic_name Topic your_topic_name is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. You may also pass in a bootstrap server instead of zookeeper: kafka-topics.sh –bootstrap-server kafka:9092 … Read more