Redis is More Than a Cache - Data Reference
A few weeks after fixing our delayed-job processor, one queue started behaving badly over the weekend.
Monitoring caught it. At first it looked recoverable. By Tuesday it was clear it was not. A newly onboarded large customer was affected, which made the issue impossible to ignore.
I got pulled into the investigation early because I had worked on that part of the backend before.
The incident pointed to a deeper design flaw: we were using Redis to store each full payload, the application data inside a job, not as a coordination layer. This post is about why that distinction matters and how we fixed it.
The incident
After a few hours, we found the root cause.
Our polling mechanism had entered a hot loop on one queue. Jobs were being retried and delayed over and over, so that queue never drained. Because the draining process was still too coupled to the rest of the system, one pathological queue slowed down others too.
We shipped a short-term fix quickly: stop retrying after N attempts and break the loop.
That solved the incident.
It did not solve the design.
The real problem
The original delayed-job storage put the full serialized job directly into a Redis sorted set, which keeps entries ordered by a numeric score such as a scheduled time.
That was convenient, but it created two serious problems:
-
Redis memory usage scaled with payload size. Large jobs meant large sorted sets.
-
A single backlog could get very expensive. We were using a large ElastiCache instance mainly to absorb spikes and pathological cases.
At the time, our delayed-job footprint had grown into the hundreds of gigabytes. That is a very expensive way to keep postponed work around.
The root cause was the same in both cases: Redis was holding full application data when it should have been holding only enough information to coordinate re-enqueueing.
How to know if this applies to you
Look at your Redis memory usage. If your delayed-job sorted sets consume a large share of total memory, or if your Redis instance size is driven by job payload size rather than queue count, you are likely storing too much application data in Redis.
The redesign
The principle was simple: Redis should store ordering and coordination data, not full delayed-job payloads.
So instead of treating Redis as the system of record for every postponed message, we introduced a storage abstraction with multiple backends. Redis stayed in the hot path, but it no longer carried the full job body in every case.
New storage strategy
The storage decision depends on a few factors:
- how long the delay will be
- how large the payload is
- whether a storage backend is enabled for that queue or customer account (tenant)
- whether the preferred backend is available
For short delays or small payloads, Redis can still hold the full object.
For longer delays or larger payloads, the job body goes somewhere cheaper, such as S3, and Redis stores only a compact reference.
A delayed-job member might look like this:
s3::some-queue::384192393192::<generated-uuid>
The prefix tells the processor which backend to use later. In this case, it fetches the payload through the s3 storage implementation before re-enqueueing the job.
Practical migration path
You do not need to move everything at once.
- Start by adding the storage abstraction and keeping Redis as the default.
- Route one queue or one tenant to the new backend.
- Measure latency impact and failure modes.
- Expand gradually.
The key is making the reference format self-describing so the processor does not need external configuration to know where to look.
Supporting changes
We also changed the delayed-job processing model itself. Each change supported the same principle: Redis coordinates, other systems store.
Decoupled from the scheduler
The delayed-job service now drives itself on an interval instead of depending on another scheduler component. That removes an unnecessary coupling that could block draining.
Circuit breaking
We added a circuit breaker: a retry-attempt limit that exits the loop instead of letting one failure repeat forever. This prevents one queue from monopolizing the coordination layer.
Concurrent queue processing
Instead of serializing too much work, we use bounded concurrency to process multiple target queues at overlapping times. That improves draining speed while keeping resource usage controlled.
Better visibility
We added metrics for:
- re-enqueue latency
- success and failure counts
- number of delayed jobs processed
- queue-level behavior over time
Delayed-job systems often fail quietly until they fail loudly. Visibility is part of making the coordination layer operable.
Horizontal scaling
By changing how we pop work from Redis, we could use horizontal scaling and run multiple delayed-job processors with a cleaner coordination model than before.
That made the system more resilient under abnormal load.
Expected impact
The biggest expected win was infrastructure size.
If Redis stores compact references instead of large payloads for long-lived delayed jobs, memory usage drops dramatically. In our case, the difference was large enough to justify the redesign by itself.
The broader goals were:
- much smaller Redis memory usage
- faster draining under stress
- less coupling between queues and backend components
- better support for horizontal scaling
- clearer operational visibility
Remaining concerns
No redesign removes all trade-offs. The important questions after the change were:
- how many delayed jobs per second can we sustain?
- what do p75, p90, and p99 latency percentiles look like?
- how much will S3 fetch latency affect re-enqueue speed?
- how low can we safely size Redis now?
- what happens if Redis itself becomes unavailable?
Those are much healthier questions than "why is this one queue consuming hundreds of gigabytes of memory?"
Takeaway
Redis is extremely useful for coordination and ordering, but that does not mean it should hold every byte of every object forever.
The best Redis designs often store less application data and more routing and scheduling data.
In our case, moving from full payloads to references turned Redis from an expensive payload warehouse into a fast coordination layer.
When to consider this pattern
Consider moving payloads out of Redis when:
- your Redis memory is dominated by job bodies, not queue metadata
- you have jobs delayed for minutes or hours with large payloads
- your Redis instance size is driven by peak backlog, not steady-state load
- you already have cheap object storage available