Using Kafka for a task queue is a bad idea.
Use RabbitMQ instead, it does it much better and more elegantly.
Although you can use Kafka for a task queue – you will get some issues:
Kafka is not allowing to consume a single partition by many consumers (by design), so if for example a single partition gets filled with many tasks and the consumer who owns the partition is busy, the tasks in that partition will get “starvation”.
This also means that the order of consumption of tasks in the topic will not be identical to the order which the tasks were produced which might cause serious problems if the tasks needs to be consumed in a specific order (in Kafka to fully achieve that you must have only one consumer and one partition – which means serial consumption by just one node. If you have multiple consumers and multiple partitions the order of tasks consumption will not be guaranteed in the topic level).
In fact – Kafka topics are not queues in the computer science manner. Queue means First in First out – this is not what you get in Kafka in the topic level.
Another issue is that it is difficult to change the number of partitions dynamically. Adding or removing new workers should be dynamic. If you want to ensure that the new workers will get tasks in Kakfa you will have to set the partition number to the maximum possible workers. This is not elegant enough.
So the bottom line – use RabbitMQ or other queues instead.
Having said all of that – Samza (by linkedin) is using kafka as some sort of streaming based task queue:
Samza
Edit:
scale considerations: I forgot to mention that Kakfa is a big data/big scale tool. If your job rate is huge then Kafka might be good option for you despite the things I wrote earlier, since dealing with huge scale is very challenging and Kafka is very good in doing that. If we are talking about smaller scales (say, up to few dosens/hundreds of jobs per second) then again Kafka is a poor choice compared to RabbitMQ.