Depends on what you mean by “blocked”. If you mean “are existing connections closed when rebalance is triggered” then the answer is yes. The current Kafka’s rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) …
I use postgres for a FIFO queue as well. I originally used ACCESS EXCLUSIVE, which yields correct results in high concurrency, but has the unfortunate effect of being mutually exclusive with pg_dump, which acquires a ACCESS SHARE lock during its execution. This causes my next() function to lock for a very long time (the duration …
You can loop over a copy of the underlying data store: for elem in list(q.queue) Eventhough this bypasses the locks for Queue objects, the list copy is an atomic operation and it should work out fine. If you want to keep the locks, why not pull all the tasks out of the queue, make your …
Java 5+ has all the tools you need for this kind of thing. You will want to: Put all your Producers in one ExecutorService; Put all your Consumers in another ExecutorService; If necessary, communicate between the two using a BlockingQueue. I say “if necessary” for (3) because from my experience it’s an unnecessary step. All …
You should continue to take() from the queue. You can use a poison pill to tell the worker to stop. For example: private final Object POISON_PILL = new Object(); @Override public void run() { //worker loop keeps taking en element from the queue as long as the producer is still running or as //long as …
Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation: Step 2: Start the server Kafka uses zookeeper so you need to first start a zookeeper server if you don’t already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node zookeeper instance. As to why, …