Schema Change Discipline

Definition

Schema change discipline is the practice of evolving a production database through small, backward-compatible steps — conventionally expand and contract. In the expand phase you add the new structure (column, table, index) alongside the old, dual-write or backfill the data, and switch reads across behind a flag. In the contract phase — one or more releases later, once nothing depends on the old shape — you remove it. The discipline exists because application deploys are rolling and reversible, while naive schema changes are neither.

Why It Matters

The failure mode is as old as continuous deployment: release renames a column, the rolling deploy starts, and the old application version — still serving half the traffic — immediately errors against a schema that no longer has its column. Rollback makes it worse, because the new schema no longer matches the old code either. Every mature platform team has this scar. Expand-and-contract removes the coupling: at every moment during the rollout, both old and new code work against the live schema, so deploys, rollbacks and partial failures are all survivable.

The Discipline, Step by Step

1. Expand — add the new column or table as nullable or with a safe default; never rename or retype in place.
2. Dual-write — application writes to both old and new shapes; reads still come from the old.
3. Backfill — copy historical rows in small, throttled batches, with a checkpoint so the job resumes safely.
4. Verify and switch reads — compare old and new values on a sample, then flip read traffic to the new shape behind a feature flag.
5. Observe a full release cycle — including at least one rollback rehearsal — with the flag ready to flip back.
6. Contract — stop writing the old shape, then drop it in a later release, once the data proves nothing reads it.

Real-World Example

An e-commerce team needed to split users.full_name into first and family names for a checkout integration. The first attempt was a single-release rename executed on a Friday deploy. The rolling update took eleven minutes; for nine of them, the old pods threw on every profile save, and the support queue lit up with checkout failures. The team reverted, regrouped, and did it properly: added the two new columns, dual-wrote for a week, backfilled 40 million rows overnight in batches of five thousand with a ten-millisecond sleep between batches, sampled-compared values, switched reads behind a flag, and dropped the old column two releases later. Total customer impact of the second attempt: zero. The migration took three weeks of calendar time and about six hours of engineering — the Friday deploy had cost them more than that in incident response alone.

Practical Lessons Learned

• Renames do not exist. There is only add-new, migrate, drop-old. Any ORM or migration tool that emits a rename in one step is generating an outage.
• Backfills are production workloads. Throttle them, checkpoint them, monitor replication lag, and give them a kill switch — an unthrottled backfill is a denial-of-service attack on your own primary.
• The contract phase is where discipline dies. Old columns linger for years because "dropping is risky" — schedule the drop when you schedule the expand, or it will never happen.
• Test the rollback, not just the rollout. A migration that cannot survive a code rollback is an incident waiting for its deploy window.
• Locks are the silent killer. Adding a column with a default, or an index without CONCURRENTLY, can lock a hot table for minutes on some databases — know your engine's locking behaviour before the migration, not during it.

Expert Tips

• Keep migrations decoupled from application deploys as separate, reviewed steps — schema first (expand), code second, schema last (contract).
• Write every backfill as a resumable job that processes key ranges and records progress; you will restart it at least once.
• Compare old and new values continuously during dual-write — a background job that samples and alerts on divergence catches write-path bugs before the read switch amplifies them.
• Feature-flag the read switch even when it feels like overkill; the flag is what turns a bad migration from an incident into a config change.
• Keep a migrations register: what expanded when, what is dual-writing, what is due to contract. Review it monthly — it is the antidote to schema archaeology.

Common Mistakes

• Renaming or retyping columns in a single migration on a live, rolling-deployed system.
• Adding a NOT NULL column without a default to a hot table and locking it for the duration of the rewrite.
• Running backfills as one giant transaction, holding locks and blowing up replication for hours.
• Letting the ORM auto-migrate in production, where a well-meaning model change becomes an unreviewed schema event.
• Never contracting, so the schema accumulates five years of deprecated columns nobody dares touch.

Key Takeaways

• Deploys roll and roll back; schema changes must be shaped so both directions are safe at all times.
• Expand, dual-write, backfill, switch reads, contract — in that order, across releases.
• Backfills are throttled, checkpointed, monitored production jobs with kill switches.
• Renames are a fiction; only add and (later) remove.
• Schedule the contract phase when you schedule the expand — deprecated shapes do not remove themselves.

Related Concepts

Pairs with Zero-Downtime Migration, Feature Flag Management, Rollback Plan, and Blue-Green Deployment.