Engineering Capacity Planning

Definition

Engineering capacity planning is the disciplined forecasting of load against infrastructure: translating business projections (orders per second, concurrent users, queue joins) into resource demand (compute, database writes, connections, third-party quotas), setting headroom targets, and validating the plan with load tests before the demand arrives. It sits between finance, product and engineering — the discipline that answers "can we take Black Friday?" with data instead of optimism.

Why It Matters

Capacity failures are binary and public: the platform holds or it falls over, in front of every customer, at the exact moment revenue peaks. The opposite failure is quieter but just as real — permanent overprovisioning, where last year's emergency scale-up becomes this year's standing cost because nobody scaled it back. Good planning avoids both: demand is modelled in business units, bottlenecks are identified by measurement rather than intuition, headroom policy is explicit, and the peak is rehearsed under load before it arrives uninvited.

How It Is Done Properly

1. Model demand in business metrics — checkouts per minute, ticket queue joins per second — then map each to its resource cost per unit.
2. Forecast the peak — growth trend plus seasonality plus events, with pessimistic and expected cases.
3. Find the binding constraint — load test until something breaks; the first bottleneck is usually database writes, connection pools or a third-party quota, not CPU.
4. Set a headroom policy — for example, run at no more than 40–50% of proven capacity at expected peak, so a forecast miss is an alert, not an outage.
5. Validate with realistic load tests — production-shaped data, production-shaped traffic mix, at two times predicted peak.
6. Rehearse the response — scale-up runbooks, failover drills and the comms plan tested before the event, not during it.

Real-World Example

A ticketing platform faced on-sale events where queue joins hit thirty times baseline within ninety seconds. The capacity model was built on the metric that mattered — completed purchases per minute — and mapped to database writes and payment-gateway calls. A load test at twice the predicted peak found the real ceiling: the connection pool exhausted at 60% of target while CPU idled at 30%. The pool was resized, the gateway's rate limit was raised by agreement, and a second test confirmed headroom. On the day, the platform absorbed a peak 12% above prediction without degradation. The previous year, planned on CPU dashboards alone, the same event had taken the site down in four minutes — the database had never appeared on anyone's graph.

Practical Lessons Learned

• CPU is the last thing to saturate and the first thing people watch. Connection pools, write throughput, lock contention and third-party quotas run out first.
• Peak shape matters as much as peak size. A ninety-second thundering herd stresses queues, pools and caches differently from a slow ramp — test the shape, not just the number.
• Third parties are capacity too. Payment gateways, email providers and SMS services have rate limits that become your ceiling; contract the burst headroom in advance.
• Load tests lie when the data does. A test against an empty database proves nothing; production-shaped data volume and distribution are what make results believable.
• Scale-down is part of the plan. Capacity added for an event needs a scheduled, owned removal date or it becomes permanent cost by default.

Expert Tips

• Express every capacity figure in business units first — "we can process 1,200 checkouts per minute with 40% headroom" lands with finance in a way "we have 60% CPU free" never will.
• Load test at 2x predicted peak and call the system ready when the breaking point is known, documented and comfortably above the policy line.
• Watch the least-elastic component — the resource that cannot scale in minutes (usually the primary database) sets the real ceiling; protect it with queues, backpressure and shed-able load.
• Keep a capacity model as a living spreadsheet that maps business growth to resource curves, reviewed monthly; five lines of arithmetic now beat a war-room later.
• Rehearse failure, not just success: kill a zone during the load test and watch whether the platform degrades gracefully or falls off a cliff.

Common Mistakes

• Linear extrapolation from average load, ignoring seasonality, marketing events and the thundering-herd shape of real peaks.
• Load testing only the read paths while the write path — the one that actually breaks — stays unmeasured.
• Discovering third-party rate limits during the event, in production, in front of customers.
• No headroom policy, so every forecast miss becomes an outage and every overreaction becomes permanent spend.
• Treating the load test as a one-off ceremony rather than a repeatable gate that runs before every major event and every architectural change.

Key Takeaways

• Plan in business metrics, mapped to resource costs — CPU percentages convince no one and predict less.
• The binding constraint is usually database writes, connection pools or a third-party quota; find it by breaking the system in test, not in production.
• Set an explicit headroom policy so forecast misses become alerts, not outages.
• Test the peak's shape at twice its size with production-shaped data.
• Schedule the scale-down when you schedule the scale-up — emergency capacity loves becoming permanent cost.

Related Concepts

Pairs with Service Level Objective, Golden Signals Monitoring, Observability, and Production Readiness Review.