Redis pub-sub is mostly like a fire and forget system where all the messages you produced will be delivered to all the consumers at once and the data is kept nowhere. You have limitation in memory with respect to Redis. Also, the number of producers and consumers can affect the performance in Redis.
Kafka, on the other hand, is a high throughput, distributed log that can be used as a queue. Here any number of users can produce and consumers can consume at any time they want. It also provides persistence for the messages sent through the queue.
Final Take:
Use Redis:
- If you want a fire and forget kind of system, where all the messages that you produce are delivered instantly to consumers.
- If speed is most concerned.
- If you can live up with data loss.
- If you don’t want your system to hold the message that has been sent.
- The amount of data that is gonna be dealt with is not huge.
Use kafka:
- If you want reliability.
- If you want your system to have a copy of messages that has been sent even after consumption.
- If you can’t live up with data loss.
- If Speed is not a big concern.
- data size is huge