apache-kafka
What’s the purpose of Kafka’s key/value pair-based messaging?
Kafka uses the abstraction of a distributed log that consists of partitions. Splitting a log into partitions allows to scale-out the system. Keys are used to determine the partition within a log to which a message get’s appended to. While the value is the actual payload of the message. The examples are actually not very … Read more
Kafka: Consumer API vs Streams API
Update January 2021: I wrote a four-part blog series on Kafka fundamentals that I’d recommend to read for questions like these. For this question in particular, take a look at part 3 on processing fundamentals. Update April 2018: Nowadays you can also use ksqlDB, the event streaming database for Kafka, to process your data in … Read more
Is key required as part of sending messages to Kafka?
Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go … Read more
How to find the kafka version in linux
Not sure if there’s a convenient way, but you can just inspect your kafka/libs folder. You should see files like kafka_2.10-0.8.2-beta.jar, where 2.10 is Scala version and 0.8.2-beta is Kafka version.
Data Modeling with Kafka? Topics and Partitions
When structuring your data for Kafka it really depends on how it´s meant to be consumed. In my mind, a topic is a grouping of messages of a similar type that will be consumed by the same type of consumer so in the example above, I would just have a single topic and if you´ll … Read more
Leader Not Available Kafka in Console Producer
It could be related to advertised.host.name setting in your server.properties. What could happen is that your producer is trying to find out who is the leader for a given partition, figures out its advertised.host.name and advertised.port and tries to connect. If these settings are not configured correctly it then may think that the leader is … Read more
Is there a way to purge the topic in Kafka?
Temporarily update the retention time on the topic to one second: kafka-topics.sh \ –zookeeper <zkhost>:2181 \ –alter \ –topic <topic name> \ –config retention.ms=1000 And in newer Kafka releases, you can also do it with kafka-configs –entity-type topics kafka-configs.sh \ –zookeeper <zkhost>:2181 \ –entity-type topics \ –alter \ –entity-name <topic name> \ –add-config retention.ms=1000 then … Read more
Understanding Kafka Topics and Partitions
This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let’s look at an overview of producer components: 1. When a producer is producing a message – It will specify the topic it wants to send the message to, is that right? … Read more
When to use RabbitMQ over Kafka? [closed]
RabbitMQ is a solid, general-purpose message broker that supports several protocols such as AMQP, MQTT, STOMP, etc. It can handle high throughput. A common use case for RabbitMQ is to handle background jobs or long-running task, such as file scanning, image scaling or PDF conversion. RabbitMQ is also used between microservices, where it serves as … Read more