Load Shedding and Backpressure

Definition

Load shedding is the intentional rejection of a portion of incoming requests when a system is beyond its safe capacity. Backpressure is the related mechanism by which a saturated component signals upstream to slow down — bounded queues, blocking writes, rate limits, or explicit "retry later" responses. Together they replace an uncontrolled failure mode, where everything times out and nothing completes, with a controlled one where a defined subset of work succeeds reliably.

Why It Matters

Systems without shedding do not fail gracefully; they fail totally. As load exceeds capacity, queues grow, latency climbs past client timeouts, clients retry, and the retries add load to a system already saturated — the classic congestion collapse. Throughput does not plateau at maximum; it falls toward zero while the infrastructure runs flat out. Shedding breaks that loop by making the refusal explicit and cheap, so the capacity that exists is spent on requests that can still be answered within the deadline.

How It Is Implemented

1. Bound every queue. Unbounded queues convert a throughput problem into a latency problem and then into an out-of-memory crash.
2. Set deadlines and propagate them. A request already past its client deadline should be dropped, not processed — work on doomed requests is pure waste.
3. Prioritise by class. Shed batch, analytics and speculative prefetch before interactive traffic; shed anonymous browsing before checkout.
4. Signal properly. Return 429 or 503 with Retry-After so well-behaved clients back off rather than hammering.
5. Require jittered exponential backoff in clients. Synchronised retries are their own denial-of-service.
6. Add circuit breakers on dependencies. Fail fast on a sick downstream instead of holding threads open waiting for it.

Real-World Example

A ticketing platform routinely melted at on-sale time. Every release, the queue depth on the reservation service grew, p99 latency crossed the mobile client's fifteen-second timeout, and clients retried automatically — tripling effective load within a minute. Nothing completed; the whole sale was lost. The fix was not more capacity. The team bounded the reservation queue at a depth derived from the deadline (queue depth times service time under the client timeout), returned 429 with a Retry-After beyond that, and added a lightweight waiting-room page for shed users. At the next on-sale, roughly 40% of initial requests were shed within the first ninety seconds — and 100% of admitted requests completed in under two seconds. Total tickets sold in the first five minutes went up, not down, because the system stopped wasting capacity on work that would time out anyway.

Practical Lessons Learned

• Shedding increases goodput. Counter-intuitive to stakeholders, obvious in the graphs — show them the graphs.
• The queue is where latency hides. Adding workers behind an unbounded queue often makes the incident worse by increasing contention downstream.
• Retries need a budget. Cap total retries per request chain; nested retries at three layers multiply, not add.
• Shed at the edge where it is cheapest. Rejecting a request after it has consumed a database connection has already cost you the scarce resource.
• Users tolerate an honest refusal. A clear "we are busy, try in thirty seconds" beats a spinner that ends in a blank screen.

Expert Tips

• Derive queue bounds from Little's Law and your deadline rather than picking a round number: bound roughly equals acceptable latency divided by service time, times concurrency.
• Tag traffic by priority at the edge and make shedding priority-aware from day one; retrofitting classification during an incident is impossible.
• Expose shed rate as a first-class metric next to error rate, and exclude deliberate shedding from your error budget burn if it protected the SLO.
• Load test the shedding path itself. Untested shedding logic frequently becomes the bottleneck it was meant to prevent.
• Give health checks their own lightweight path so shedding never causes the orchestrator to kill healthy instances mid-incident.

Common Mistakes

• Unbounded queues chosen because "we don't want to drop anything".
• Returning 500 instead of 429, so clients treat shedding as a bug and retry aggressively.
• Shedding uniformly, so a payment confirmation is discarded at the same rate as an image thumbnail.
• Autoscaling as the only response — new instances arrive minutes after the collapse has already happened.
• Retry logic without jitter, producing synchronised waves that recreate the peak every backoff interval.

Key Takeaways

• Beyond capacity, the choice is controlled refusal or total collapse.
• Bound queues, propagate deadlines and drop work that can no longer be delivered on time.
• Prioritise shedding by business value, not uniformly.
• Signal with 429 and Retry-After, and require jittered backoff from clients.
• Test the shedding path under load before you need it.

Related Concepts

Pairs with Engineering Capacity Planning, Golden Signals Monitoring, Service Level Objective, and Chaos Engineering Practice.