The Tale of How We Paid Back Years of Tech Debt in a Month

Back in 2022 - while I was at Luma Health - we went through a major outage that cost hours of downtime, a lot of engineering effort, and a fair amount of customer frustration.

One painful discovery from that period was this: we had modeled an internal identity and access management (IAM) session object very poorly.

That mistake stayed quiet for a long time. As the company grew, it stopped being quiet.

This post is about how a single data-modeling decision created years of infrastructure debt, and why fixing it took a month but paid for itself immediately.

Why this mattered

In healthcare software, access-control state is core system data. Systems that handle protected health information (PHI) need to enforce those boundaries consistently across backend flows.

Our session object carried the data needed for that enforcement: token, settings, groups, facilities, and other metadata required to authorize work.

The problem was not the existence of the session. The problem was its size.

The modeling mistake

At some point, we chose the convenient path: load full groups and facilities data into the session JavaScript Object Notation (JSON) payload and cache it.

That worked until larger customers arrived with more complex account structures.

Eventually we had session payloads reaching 2.6 MB.

That is not a small piece of authorization context anymore.

In several job-processing paths, services fetched the cached session repeatedly as work moved through the platform. Sometimes that happened dozens of times for the same logical flow. Our system averaged around 2,000 jobs per second and could spike much higher.

At that point, the issue was no longer just wasteful serialization. It had become steady network pressure between services and the cache layer on a frequently used hot path.

What the impact looked like

That oversized payload showed up everywhere:

• unstable and slower Hypertext Transfer Protocol (HTTP) application programming interfaces (APIs)

• oversized cache infrastructure
• higher service-to-cache bandwidth consumption
• pressure that forced architectural band-aids into the system

We even introduced supporting components and patterns just to reduce the blast radius:

• a publish/subscribe service for delivery paths that were sensitive to event-loop delays
• a direct publishing path where only job identifiers were published and the body was fetched elsewhere

Those were symptoms of the same underlying problem: too much data moving through the hot path.

How to spot this in your own system

If you suspect a similar problem:

1. Check the size of your most frequently cached objects.
2. Multiply that size by how many times they are fetched per request.
3. Multiply again by your request rate.
4. If the result is in the gigabits per second range, your object model is likely too heavy for the hot path.
5. Add Redis bandwidth alarms. As a starting point, alert when a network interface exceeds 70% of its configured capacity.

The fix

The actual code change was small compared to the impact.

Instead of loading full group documents while building the session, we introduced a path controlled by a feature flag that selected only the fields needed in the hot path. In simplified form, the idea looked like this:

Query.select('_id')

That snippet is only illustrative. The important part is that we stopped materializing large relationship payloads into the session when the hot path only needed compact identifiers.

That was enough to build a much smaller session object without changing the higher-level authorization model.

Of course, the hard part was not writing the line. The hard part was rolling it out safely.

Why rollout took time

Session shape was deeply embedded across many services and internal libraries, so we had to treat this as a compatibility project rather than a simple refactor.

That meant:

• auditing service and library versions
• searching the codebase for assumptions about session shape
• using feature flags per tenant
• monitoring product and platform metrics during rollout

The questions were practical and unsettling:

• would messages stop being sent?
• would notifications silently disappear?
• would some customer workflows degrade in ways we would only discover later?

So we rolled it carefully.

Outcome

The rollout took nearly a month across around 90 backend services, followed by tenant-by-tenant enablement.

The results justified the caution:

• roughly 60% reduction in the relevant cache-related network traffic
• much more stable API latencies
• clear room to downsize parts of the representational state transfer (REST) API layer
• near-elimination of the network pressure created by our temporary band-aids

Just as important, we were finally able to start removing workarounds that should never have become architecture in the first place.

The deeper lesson

The expensive part of technical debt is often not the original bug. It is the ecosystem of compensating decisions that grows around it.

One oversized payload led to:

• bigger infrastructure
• extra services
• more complicated publish paths
• more operational anxiety

All of that because a hot-path object carried far more data than it needed.

When to consider this kind of optimization

Not every large cached object needs fixing. Consider trimming hot-path payloads when:

• the object is fetched multiple times per workflow
• the fetch rate scales with traffic volume
• the object contains data that is only needed in a small fraction of use cases
• you have already scaled infrastructure and the problem persists

Takeaway

There is a lot of justified attention on product features, growth, and customer-facing delivery.

But backend modeling decisions matter just as much when the system starts to scale.

Sometimes the best optimization is not a fancy algorithm. It is revisiting a data model that quietly became too expensive for the path it lives on.

And sometimes one month of careful cleanup pays back years of architectural interest.