Comparisons

Redis vs Kafka: in-memory data store vs distributed commit log, and when to use each

Redis vs Kafka: an in-memory store with Pub/Sub and Streams vs a disk-based commit log. Delivery guarantees, retention, running cost, licences and Valkey.

Published ~8 min read
On this page

Short answer: Redis is an in-memory data store: a cache, session store and counter server that also does messaging, with Pub/Sub that delivers at most once and Streams that keep a capped log in RAM. Apache Kafka is a distributed commit log on disk, built to store event streams for days or indefinitely, replay them and feed many independent consumers at high throughput. Use Redis for fast state and light queues; use Kafka when events are the record and must survive, be replayed and fan out.

the same order event in both
# Redis: fire-and-forget Pub/Sub, then a Stream with a consumer group
redis-cli PUBLISH orders "order 42 paid"
redis-cli XGROUP CREATE orders billing '$' MKSTREAM
redis-cli XADD orders MAXLEN '~' 100000 '*' id 42 status paid
redis-cli XREADGROUP GROUP billing worker-1 COUNT 10 STREAMS orders '>'
redis-cli XACK orders billing <entry-id>

# Kafka 4.4: create a topic, write events, read them from the start
bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092
bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic orders --from-beginning --bootstrap-server localhost:9092

The Redis Pub/Sub message reaches whoever is subscribed at that instant and is then gone. The Stream entry and the Kafka record both stay, and a consumer that was offline can catch up. Facts below are from redis.io, kafka.apache.org, valkey.io and the Linux Foundation, read on 13 November 2026; we did not run either system for this page.

Redis vs Kafka at a glance

RedisApache Kafka
What it isIn-memory data structure serverDistributed, partitioned commit log
Where data livesRAM; optional RDB snapshots and AOF log on diskDisk, in segment files per partition, read through the OS page cache
MessagingPub/Sub (no storage) and Streams (append-only, consumer groups)Topics split into partitions; consumer groups track offsets
DeliveryPub/Sub at most once; Streams at least once with XACKAt least once by default; idempotent producer on by default; transactions
RetentionUntil you trim the stream or memory runs outBy time or size; 168 hours (7 days) by default
ReplayXRANGE over entries still keptRewind a consumer's offset
Runs asOne small process; replicas, Sentinel or Cluster for HAJVM brokers (Java 17+) with KRaft controllers; three or more nodes for HA
LicenceRedis 8+: RSALv2, SSPLv1 or AGPLv3; Valkey: BSD 3-ClauseApache License 2.0
Newest releaseRedis 8.10.3 (8.12 in RC); Valkey 9.1.3Kafka 4.4.0, 22 October 2026

Data store vs commit log

Redis keeps keys holding strings, hashes, lists, sets, sorted sets and streams in memory, so reads and writes run at memory speed and the whole dataset has to fit in RAM. Messaging is one feature among many. A Stream is an append-only data structure of field-value entries with time-based IDs; consumer groups hand each entry to one consumer, keep it in a pending entries list until XACK, and let another consumer claim it after a crash. Because it lives in RAM, you cap it with MAXLEN or MINID.

Kafka is built around the log. A topic is split into partitions, each an ordered file on disk that producers append to. Its design documentation says Kafka “relies heavily on the filesystem for storing and caching messages”, and messages stay for the configured retention whether or not anyone has read them. Each consumer group just remembers an offset per partition, so ten applications can read the same events at their own pace, and any of them can rewind.

Delivery guarantees

  • Redis Pub/Sub: at most once. Per redis.io, if a subscriber cannot handle a message because of an error or a disconnect, “the message is forever lost”.
  • Redis Streams: at least once with consumer groups: unacknowledged entries stay pending and can be claimed with XAUTOCLAIM, so handlers must be idempotent. Redis 8.6 added idempotent production on the producer side.
  • Kafka: at least once by default. Producers default to acks=all and enable.idempotence=true, so retries do not write duplicates; transactions write to several partitions atomically, and Kafka Streams offers exactly-once processing by committing offsets in the same transaction.

Persistence and retention

Redis

Redis can run with no persistence, RDB snapshots (by default a dump.rdb file written at intervals, so a crash loses the minutes since the last one), the append-only file (AOF), or both. With AOF on, the default appendfsync everysec loses at most about one second of writes. Replication to replicas is asynchronous. On 64-bit systems maxmemory is 0, meaning no limit, so set it and an eviction policy, or a growing stream ends in an error or the kernel’s OOM killer.

Kafka

Kafka deletes old log segments by age (log.retention.hours, 168 by default) or size (log.retention.bytes, unlimited by default), per topic if you like. Durability comes from replication: a topic’s replication factor (broker default 1) and min.insync.replicas (default 1) decide how many brokers must hold a write before acks=all succeeds. Production clusters usually raise both, which is why Kafka wants at least three brokers.

Throughput and operational cost

Kafka batches records and appends them sequentially, which is what lets it move very large event volumes; the price is a heavier footprint. Brokers, Connect and the tools need Java 17 (clients need Java 11), each broker needs disk sized for the full retention window, and a fault-tolerant setup is several machines. Kafka 4.0, released on 18 March 2025, was “the first major release to operate entirely without Apache ZooKeeper”: metadata now lives in KRaft controllers inside Kafka, so there is no separate ZooKeeper ensemble to run. A single-node quickstart is three commands:

Kafka 4.4 quickstart, per kafka.apache.org
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
bin/kafka-server-start.sh config/server.properties

Redis is the opposite: one lightweight process that needs RAM for the data, little else, and a few minutes to set up. Its cost grows with dataset size, because memory is the storage. Kafka’s cost grows with retention and replication, because disk is.

Licences: Redis, Valkey and Kafka

Redis up to 7.2 was BSD-licensed. In March 2024 Redis moved new versions to a dual RSALv2 or SSPLv1 licence (7.4), and with Redis 8, released in May 2025, it added AGPLv3 as a third option; redis.io’s licence page calls that “not a license change, but rather the inclusion of an option”. In response to the 2024 change, the Linux Foundation announced Valkey on 28 March 2024, a fork of Redis 7.2.4 under the BSD 3-Clause licence; valkey.io lists 9.1.3, released 26 October 2026, and existing Redis clients connect to it. Kafka is an Apache Software Foundation project under the Apache License 2.0, with no licence change.

Redis or Kafka: which one should you pick?

  • Caching, sessions, rate limits, leaderboards: Redis (or Valkey). Kafka is not a key-value store.
  • Live notifications where a missed message does not matter: Redis Pub/Sub.
  • A job queue or small event stream that fits in RAM: Redis Streams, with consumer groups and XACK, trimmed with MAXLEN.
  • Events that must be kept for days, replayed, or read by many services: Kafka.
  • Very high, sustained event volume, analytics pipelines, change data capture: Kafka.
  • Only one small server and a small team: start with Redis Streams; move to Kafka when retention or fan-out outgrows memory.
  • A licence your legal team accepts without review: Valkey (BSD) or Kafka (Apache 2.0). Redis 8 adds the OSI-approved AGPLv3 next to RSALv2 and SSPLv1.

Running them on a VPS or dedicated server

Redis or Valkey runs well on a small VPS: bind it to 127.0.0.1 on port 6379 (see the common ports list), set maxmemory, and size RAM to the dataset. A single Kafka broker, for development or a modest pipeline, fits on a VPS with enough RAM for the JVM heap and the page cache, and disk for the retention you configure; the official apache/kafka:4.4.0 image runs it with Docker. A production cluster means three or more servers, and dedicated machines make sense when the disks fill with retained events. If your next question is which database sits behind the cache, see DynamoDB vs MongoDB.

Frequently asked

Can Redis replace Kafka?

For small workloads, often yes: Redis Streams give an append-only log with consumer groups and acknowledgements. But a stream lives in RAM and has to be trimmed, while Kafka keeps events on disk for days or longer, replicates them across brokers and lets any number of consumer groups replay them. Once retention or fan-out outgrows memory, Kafka is the better fit.

Is Redis Pub/Sub reliable?

Not for messages you cannot lose. Redis documents Pub/Sub as at-most-once: a message goes to the clients subscribed at that moment, and if one is disconnected or fails, its copy is lost. Use Redis Streams with consumer groups and XACK, or Kafka, for at-least-once delivery.

Does Kafka still need ZooKeeper?

No. Kafka 4.0, released on 18 March 2025, was the first major release to run entirely without Apache ZooKeeper; cluster metadata is handled by Kafka's own KRaft controllers. Kafka 4.4.0, released on 22 October 2026, is the newest release as of November 2026.

Is Redis open source again?

Since Redis 8, released in May 2025, Redis Open Source can be used under the AGPLv3, an OSI-approved licence, as well as RSALv2 or SSPLv1. Redis 7.4 was RSALv2 or SSPLv1 only, and 7.2 and earlier were BSD. Valkey, the Linux Foundation fork of Redis 7.2.4, stays under the BSD 3-Clause licence.

Related articles

All comparisons