Messaging Confusion: Pub/Sub vs Multicast vs Fan Out

I’m confused by your choice of three terms to compare. Within RabbitMQ, Fanout and Direct are exchange types. Pub-Sub is a generic messaging pattern but not an exchange type. And you didn’t even mention the 3rd and most important Exchange type, namely Topic. In fact, you can implement Fanout behavior on a Topic exchange just … Read more

Publish/Subscribe vs Producer/Consumer?

There is a difference between the publish/subscribe and producer/consumer models. Publish/Subscriber: Subscribers subscribe to the publisher. Each message the Publisher publishes is sent to all the subscribers. That is, all subscribers receive the same message. (Think of a newspaper or magazine subscription. All subscribers receive the same magazine or newspaper) Producer/Consumer: Each message the producer … Read more

In which domains are message oriented middleware like AMQP useful?

This is a great question. The main uses of messaging are: scaling, offloading work, integration, monitoring, event handling, routing, networking, push, mobility, buffering, queueing, task sharing, alerts, management, logging, batch, data delivery, pubsub, multicast, audit, scheduling, … and more. Basically: anything where you need data but don’t want to make a database request. (Caching is … Read more

Check RabbitMQ queue size from client

You can actually retrieve this via the client. When you perform a queue_declare operation, RabbitMQ returns a tuple with three values: (<queue name>, <message count>, <consumer count>). The passive argument to queue_declare allows you to check whether a queue exists without modifying the server state, so you can use queue_declare with the passive option to … Read more

Is there a performance difference between pooling connections or channels in rabbitmq?

I have found this on the rabbitmq website it is near the bottom so I have quoted the relevant part below. The tl;dr version is that you should have 1 connection per application and 1 channel per thread. Connections AMQP connections are typically long-lived. AMQP is an application level protocol that uses TCP for reliable … Read more

Comparison between RabbitMQ and MSMQ

I wrote a blog post a while back comparing MSMQ and RabbitMQ (among others): http://mikehadlow.blogspot.co.uk/2011/04/message-queue-shootout.html RabbitMQ gave slightly better performance than MSMQ, but both were comprehensively out performed by ZeroMQ. If performance is your main criteria, you should definitely look at ZeroMQ. It’s worth noting that RabbitMQ and MSMQ are very different beasts. MSMQ is … Read more

Effective strategy to avoid duplicate messages in apache kafka consumer

The short answer is, no. What you’re looking for is exactly-once processing. While it may often seem feasible, it should never be relied upon because there are always caveats. Even in order to attempt to prevent duplicates you would need to use the simple consumer. How this approach works is for each consumer, when a … Read more