Redis is More Than a Cache - Delaying Jobs

At Luma Health, our backend ran on RabbitMQ, a message broker, and jobs that ran asynchronously (later, without making the caller wait). Delayed execution was not an edge case. It was a core feature.

A job might need to wait because an external system was down. Or it might just need a retry in five seconds. Either way, we needed a place to hold work until it was time to publish it again.

Redis seemed perfect. Our first design was simple. It also fell apart under load.

The lesson of this post is that the access pattern matters as much as the data structure choice.

The original design

When a handler wanted to postpone a job, we serialized it as JavaScript Object Notation (JSON) and stored it in a Redis sorted set. The score was the retry timestamp.

Every five seconds, a scheduled process would:

1. read the set of delayed-job queues
2. fetch ready jobs with ZRANGEBYSCORE
3. publish them back to RabbitMQ
4. remove them from the sorted set

It was easy to explain. It was also easy to break.

Where it failed

Three problems kept getting worse:

1. It could not catch up with large backlogs. We fetched jobs in chunks, so a fast-growing queue could stay behind indefinitely and limit throughput.

2. It used too much memory. Large batches were loaded into memory before being removed.

3. It was awkward to use horizontal scaling. With multiple consumers, two workers could read the same item before either deleted it, which made coordination harder than it needed to be.

At its worst, the system pushed Redis memory usage toward tens of gigabytes and still struggled to drain delayed work.

All three problems had the same root cause: we were reading work in large batches and deleting it separately.

How to know if you have the same problem

Check whether your service:

• fetches batches but sometimes fails to delete them atomically
• grows in memory as the backlog grows
• cannot safely run multiple instances
• hits a throughput ceiling regardless of resources

If any of those sound familiar, the consumption pattern is likely the bottleneck, not Redis itself.

The change that mattered

We kept the same Redis data structure and changed only how we consumed it.

Instead of fetching large batches into memory, we moved to a streaming-style approach:

1. estimate ready jobs with ZCOUNT
2. create an asynchronous iterator that requests one item at a time
3. consume jobs one by one with ZPOPMIN
4. publish each job as it is popped

The key change was switching from fetch-then-delete to pop-as-you-go.

ZPOPMIN worked well because it removes the lowest-score item as part of the read.

So instead of:

• read a large batch
• keep it in memory
• publish it
• delete it later

we now had:

• pop one ready job
• publish it
• move on

Why this was better

Lower memory pressure

We stopped pulling large delayed-job batches into memory, so the process became much more stable.

Better backlog behavior

Throughput was still limited by Redis and publish speed, but the processor no longer paid the same large batch cost up front.

Easier scaling

Using reads that remove each item as part of the same Redis operation made coordination simpler than the previous model.

It did not solve every distributed-systems problem, but it gave us a much better base.

Results

The results were better than the size of the code change suggested.

• we drained a backlog that had grown to roughly 40 GB
• process memory dropped from multi-gigabyte levels to a few hundred megabytes
• Redis sizing pressure fell enough to affect infrastructure cost

This was one of those cases where the win came from using the same data structure in a better way.

The broader point

This is why I like saying Redis is more than a cache.

A sorted set is not just a place to store temporary values. With the right access pattern, it can act as a lightweight time-ordered work queue.

The hard part is usually not picking Redis. It is picking a consumption pattern that still works once the backlog stops being small.

Takeaways

• Redis sorted sets are a practical building block for delayed jobs.
• Consumption strategy matters as much as data structure choice.
• Streaming-style consumption ages better than loading big batches into memory.
• A small algorithmic change can remove both cost and reliability problems.

What to check in your own system

If you already use Redis for delayed jobs:

1. Are you fetching and then deleting, or removing as you read?
2. Does your processor memory grow with backlog size?
3. Can you safely run more than one consumer?
4. Do you have visibility into queue depth, processing lag, and consumer health?

If the answer to any of those questions concerns you, the fix may be simpler than replacing Redis.

Final thought

There are many ways to implement delayed jobs. RabbitMQ plugins, dedicated schedulers, and other persistence layers are all fair options.

But in day-to-day engineering, the best improvement is often the one that solves the current problem with the least operational drama.

In our case, that meant keeping Redis and fixing how we consumed it.