Progressive Delivery

Definition

Progressive delivery extends continuous delivery by adding controlled exposure. Instead of shipping a change to 100% of users the moment it is deployable, the change is released to a small, defined slice first — a percentage of traffic, a geographic region, an internal cohort, a set of beta customers. Metrics are watched; if the slice behaves well, exposure widens in stages until full rollout. If it misbehaves, exposure is reduced or reverted. The practice combines canary releases, feature flags, and observability into a single delivery discipline.

Why It Matters

The blast radius of a bad deploy scales with the percentage of users exposed. A change deployed to 1% of traffic that causes a 5% error rate produces a 0.05% overall impact — measurable, recoverable, invisible to most users. The same change at 100% produces a 5% incident visible on the status page and in customer support. Progressive delivery is not about slowing down; it is about buying observability before commitment. Teams that adopt it typically ship more frequently, not less, because the risk of each individual change is contained.

Techniques

• Canary release — a small percentage of traffic to the new version; grow if healthy.
• Blue-green deployment — full parallel environment; instant traffic switch, instant rollback.
• Feature flags — the change is deployed but only enabled for a controlled cohort.
• Ring deployment — expanding rings of users (dogfood → beta → early access → general availability).
• Geographic rollout — region by region, useful for latency-sensitive or regulated systems.
• Shadow traffic — production requests replayed against the new version, comparing responses without exposing users.

Real-World Example

A subscription video service rolled out a new recommendation algorithm using progressive delivery. Stage one exposed the algorithm to 1% of users in a single region, comparing session length and engagement against the control group. Stage two grew to 10%. At stage three (50%) the team saw a small but statistically significant drop in engagement for the "casual viewer" cohort — users who watched fewer than three shows per week. Rather than continue to 100% and undo weeks of work, they routed only the "engaged viewer" cohort to the new algorithm and returned casuals to the previous version. The compromise increased overall engagement 3.2% versus a full-rollout counterfactual of -0.4%. The progressive rollout did not just prevent a regression — it discovered a segmentation opportunity that a big-bang launch would have missed entirely.

How to Introduce It

1. Observability first. Progressive delivery is a data-driven practice; without reliable metrics you are shipping in the dark.
2. Define automatic promotion criteria — error rate under X%, latency under Y ms, business metric within Z% of control.
3. Define automatic rollback criteria — the equivalent thresholds for reduction.
4. Choose the primary technique — canary is a good starting point for most services.
5. Automate the promotion — a human-in-the-loop for every step defeats the practice at scale.
6. Start with a specific service and generalise; do not try to convert the whole platform at once.
7. Publish the rollout dashboard to the team so promotion decisions are visible.

Practical Lessons Learned

• The metric matters more than the technique. A canary without a strong SLI comparison is just a slow deploy.
• Stateful services complicate rollout. Database migrations, cache warmup and connection state need attention that stateless services do not.
• The first percentage is the risky one. 1% of a large service is enough to expose most regressions; growing from 1% to 100% is usually the safe part.
• Feature flags and canaries stack. Deploy behind a flag, roll out with a canary, gate exposure with the flag — three layers of safety at low incremental cost.
• Long-tail issues need long-tail rollouts. Some regressions only appear in the last 10% because they affect rare user cohorts.

Expert Tips

• Score canary health against a control group of the previous version running the same traffic pattern — absolute metrics can move for reasons unrelated to the change.
• Instrument the change with a specific "which version served this request" tag in logs and traces. Post-mortem analysis becomes trivial.
• Automate the promotion decision but keep a manual pause button. Trust the automation; retain the veto.
• For customer-visible changes, coordinate rollout with support so incoming tickets can be triaged by cohort.
• Retire abandoned canaries. A canary stuck at 5% for two weeks is technical debt.

Common Mistakes

• Progressive delivery adopted without observability — the team knows something is wrong but cannot pin it to a version.
• Promotion criteria defined loosely, so every rollout becomes a subjective judgement call.
• Rollback rehearsed once at introduction, never tested again; when it is needed the tooling has bit-rotted.
• Canary traffic routed to a single availability zone or user cohort, so the sample is not representative.
• Feature flags left in the code base long after full rollout; the branch multiplication becomes its own liability.

Key Takeaways

• Progressive delivery contains blast radius without slowing overall throughput.
• Observability is a prerequisite, not a nice-to-have.
• Compare against a control, not an absolute threshold.
• Automate the promotion decisions; keep human veto.
• Retire abandoned canaries and stale flags — the practice depends on hygiene.

Related Concepts

Pairs with Canary Release, Blue-Green Deployment, Feature Flag Management, and Rollback Plan.