Streams

Introduction

Redis Streams are an append-only log data type introduced in Redis 5.0. They model an event-sourcing or messaging backbone directly inside Redis: producers append events, consumers read them, and the data persists until you explicitly delete it. If you have outgrown simple List-based queues but do not yet need a dedicated message broker such as Kafka or RabbitMQ, Streams are the pragmatic next step.

Prerequisites

Completion of Basic Data Types and familiarity with Lists and Sorted Sets
Understanding of the terms "producer" and "consumer" in a messaging context
Redis 5.0 or newer (INFO server to check)

Stream Fundamentals

A Stream is a sequence of entries. Each entry has a unique ID and a set of field-value pairs. Entries are immutable and ordered by ID, which makes Streams ideal for event logs, activity feeds, and audit trails.

plaintext

Stream: events:orders
├─ 1704067200000-0  {user: "42", item: "7", action: "purchase"}
├─ 1704067200001-0  {user: "13", item: "2", action: "refund"}
└─ 1704067200002-0  {user: "42", item: "9", action: "cart_add"}

Appending Events with XADD

The XADD command appends an entry to a Stream. The ID is normally auto-generated by Redis using millisecond timestamps and a sequence number.

bash

# Append an event to the stream (Redis generates the ID)
XADD events:orders * user 42 item 7 action purchase
# "1704067200000-0"
 
# The asterisk tells Redis to auto-generate the ID
# You can also supply your own ID, but the auto format is recommended

If you want to cap the Stream length automatically, use the MAXLEN option:

bash

# Keep only the last 1000 entries, approximating the trim for speed
XADD events:orders MAXLEN ~ 1000 * user 13 item 2 action refund
 
# Exact trim (slightly slower but precise)
XADD events:orders MAXLEN 1000 * user 42 item 9 action cart_add

Tip

Approximate Trimming

The ~ modifier tells Redis to trim roughly to the target length by removing whole nodes in the internal radix tree. It is faster than exact trimming and usually good enough for log retention.

Reading Entries with XRANGE and XREVRANGE

XRANGE returns entries between two IDs, inclusive. Use - for the oldest entry and + for the newest.

bash

# Read all entries from the beginning
XRANGE events:orders - +
 
# Read a specific window
XRANGE events:orders 1704067200000-0 1704067200002-0
 
# Read in reverse order (newest first), limit to 10
XREVRANGE events:orders + - COUNT 10

Reading Entries with XREAD

XREAD is the blocking read command. It waits until new data arrives, making it the foundation of consumer loops.

bash

# Block for up to 30 seconds, waiting for new entries on events:orders
XREAD BLOCK 30000 STREAMS events:orders $
 
# Read from multiple streams at once
XREAD BLOCK 5000 STREAMS events:orders events:payments 0 0

The special ID $ means "only entries added after I started blocking." If you use 0 instead, XREAD returns all existing entries and then waits.

Consumer Groups

Consumer groups are the mechanism that lets multiple clients cooperate to process a Stream. They provide three guarantees:

Each message is delivered to only one consumer within the group.
Consumers remember their position; if a client restarts, it resumes where it left off.
Messages must be explicitly acknowledged, allowing retries if processing fails.

Creating a Group

bash

# Create a consumer group attached to events:orders, starting at the beginning
XGROUP CREATE events:orders order_processors 0
 
# Create a group that only sees future messages (not existing history)
XGROUP CREATE events:orders recent_processors $

Reading as a Group Consumer

bash

# Consumer "worker_1" in group "order_processors" reads one new message
XREADGROUP GROUP order_processors worker_1 COUNT 1 STREAMS events:orders >

The special ID > means "deliver only messages never delivered to any consumer in this group."

Acknowledging Messages

After your application successfully processes an event, acknowledge it with XACK. This removes the message from the pending entries list (PEL).

bash

# Acknowledge a specific message by its ID
XACK events:orders order_processors 1704067200000-0

If you do not acknowledge, the message stays in the PEL. You can inspect stalled messages and reassign them:

bash

# List pending messages for the group
XPENDING events:orders order_processors
 
# Claim a stalled message after 60 seconds of inactivity
XCLAIM events:orders order_processors worker_2 60000 1704067200000-0

Warning

Message Loss Is Possible

If a consumer crashes after reading a message but before acknowledging it, the message remains pending. Another consumer can claim it with XCLAIM, but there is a window where no one is actively working on it. Design your consumers to be idempotent so that double-processing is safe.

Stream vs List vs Pub/Sub

Redis offers three overlapping messaging primitives. Choosing the right one saves you from reinventing reliability later.

Feature	List (`LPUSH`/`BRPOP`)	Pub/Sub	Stream
Persistence	Yes	No	Yes
Consumer groups	No	No	Yes
Acknowledgment	No	No	Yes
Replay history	Limited	No	Yes
Complexity	Low	Lowest	Medium

Use Pub/Sub for fire-and-forget broadcasts where every subscriber must be online. Use Lists for simple worker queues with a single consumer. Use Streams when you need persistence, multiple cooperating consumers, and the ability to re-read or audit the event log.

A Minimal Producer-Consumer Example

Here is a complete pattern you can run in two terminal windows.

Terminal 1: Producer

bash

redis-cli
 
# Add three events
XADD tasks * type email to user@example.com
XADD tasks * type report generate daily
XADD tasks * type cleanup temp_files

Terminal 2: Consumer in a Group

bash

redis-cli
 
# Create the group once
XGROUP CREATE tasks task_workers 0 MKSTREAM
 
# Worker loop (in a real app this loops indefinitely)
XREADGROUP GROUP task_workers worker_1 COUNT 1 BLOCK 5000 STREAMS tasks >
# Returns the first task
 
# Acknowledge after processing
XACK tasks task_workers <returned-message-id>

In a production application, you wrap the XREADGROUP / process / XACK sequence in a loop. If processing throws an exception, skip the acknowledgment so the message can be retried or inspected.

Deleting and Trimming Streams

Streams grow until you manage them. Redis gives you two cleanup tools.

bash

# Delete a specific entry by ID
XDEL events:orders 1704067200000-0
 
# Trim the stream to a maximum length
XTRIM events:orders MAXLEN 5000
 
# Trim by minimum ID (remove everything older than a timestamp)
XTRIM events:orders MINID 1703980800000-0

For time-based retention, many teams run a periodic job or a Redis function that calls XTRIM ... MINID with a cutoff from 24 or 48 hours ago.

FAQ

Can I change an entry after it is added to a Stream?

No. Streams are append-only logs. If you need to correct data, append a compensating event or manage corrections in your application layer.

What is the maximum length of a Stream?

Limited only by memory and maxmemory settings. In practice, use MAXLEN or MINID trimming to prevent unbounded growth.

Do I have to use consumer groups?

No. You can use XREAD directly for simple fan-out or single-consumer patterns. Consumer groups add complexity only when you need competing consumers and delivery tracking.

What happens if all consumers in a group are offline?

Messages accumulate in the Stream. When a consumer comes back online, XREADGROUP with > delivers all undelivered messages. If the consumer uses $ or a specific ID instead of >, it may skip messages.

Can one consumer belong to multiple groups?

A consumer name is scoped to a single group. The same application instance can open separate connections as worker_1 in group_a and worker_1 in group_b; Redis treats them independently.

How does Stream ordering work across multiple producers?

Stream IDs are composed of a millisecond timestamp and a sequence number. If two producers call XADD in the same millisecond, Redis increments the sequence number to maintain total order. For most distributed systems, this level of ordering is sufficient.