Chaos Engineering Practice

Definition

Chaos engineering is the practice of designing and running controlled experiments that inject failure into a running system — killed processes, dropped network packets, degraded databases, exhausted CPU — in order to observe how the system responds and to strengthen it before real failures arrive. It was formalised by Netflix's Chaos Monkey in 2010 and codified by the "Principles of Chaos Engineering" that followed. It is not a marketing exercise, and it is not about breaking things for fun; it is about learning what your system actually does under stress rather than what your architecture diagram claims it does.

Why It Matters

Distributed systems fail in ways no single engineer can hold in their head. Cascading failures, retry storms, thundering herds and partial partitions rarely appear in staging environments because staging traffic is polite and thin. Real production traffic — with its long-tail latency, its burst patterns, its dependency chains — surfaces failures that never quite match the runbook. Chaos engineering closes that gap by rehearsing failure in daylight, with the team on hand, rather than at 3 a.m. with a pager going off.

The Four Principles

1. Build a hypothesis about steady-state behaviour. Define the business or operational metric that indicates the system is healthy — order rate, latency percentiles, error budget consumption.
2. Vary real-world events. Kill instances, delay dependencies, partition the network, exhaust memory, corrupt clocks — the failure modes that actually happen in production.
3. Run experiments in production. Staging is a useful warm-up; only production carries the traffic patterns that expose real behaviour.
4. Automate to run continuously. A one-off game day is a workshop, not a discipline. Continuous, low-blast-radius experiments turn resilience into muscle memory.

Real-World Example

A European payments platform ran a monthly game day in which they killed a random availability zone for their card-processing service. Six months in, the team was confident. On the seventh game day the exercise triggered an unexpected cascade: the retry logic in the merchant SDK doubled request volume onto the remaining zones, saturating the connection pool on their fraud service, whose degradation caused the acquiring bank to time out and reject transactions. Recovery took eleven minutes; the blast radius was one per cent of live traffic. The team refactored the retry logic to add jitter and exponential backoff, tuned the fraud service connection pool, and added a circuit breaker in the SDK. Two months later a real AZ outage caused zero customer impact. The game day found in eleven minutes what would have cost hours in a live incident.

How to Introduce It

1. Start with observability — you cannot learn from an experiment you cannot measure. Steady-state metrics, alerting and dashboards must exist first.
2. Begin in staging for the first few experiments to build team confidence and refine tooling.
3. Design small blast radius first — a single instance, a single non-critical service, a single tenant.
4. Announce the experiment to the on-call team and any downstream consumers. Chaos should not be a surprise.
5. Have a documented abort criterion and someone with the authority and ability to stop the experiment instantly.
6. Run a post-experiment review even for a successful outcome; the discoveries are often in what did not happen.
7. Automate the passing experiments so they run continuously and regressions are caught.

Practical Lessons Learned

• Culture goes first. A blameful team will hide the results; a learning team will publicise them.
• Steady-state metric matters more than the failure injected. If you cannot say whether the system is healthy, injecting failure teaches nothing.
• Blast radius creep is the biggest risk. Experiments that grow because "last one was fine" eventually cause the outage they were supposed to prevent.
• The insights come from the boring runs. Every "nothing happened" experiment is a confidence increment; the interesting failure will arrive when you least expect it.
• Weekend experiments are a bad idea. The value is in the learning; nobody learns when the team is off.

Expert Tips

• Pick a business metric — orders per minute, active sessions — as your steady-state signal rather than an infrastructure metric. It keeps the practice honest.
• Rotate the game day facilitator among senior engineers. It spreads the deep systems knowledge across the team.
• Instrument experiments with the same alerting the on-call uses. If the alert does not fire during a real fault injection, tune the alert.
• Publish a summary of every game day to the whole engineering org. The reading list becomes an onboarding asset.
• Do not confuse chaos engineering with fault tolerance testing in the CI pipeline. Both are useful; only one runs in production.

Common Mistakes

• Running experiments without a defined steady-state metric — the team observes chaos but learns nothing.
• Injecting failures into critical services before observability is mature; the blast radius exceeds the team's ability to reason.
• Framing chaos engineering as "breaking things" to the business, then losing air cover after the first minor impact.
• Skipping the post-experiment review; findings never make it into the backlog.
• Believing that Netflix-scale tools are prerequisite. A weekly cron that kills one non-critical pod is a valid start.

Key Takeaways

• Chaos engineering is a discipline, not a toolset.
• Steady-state metric, hypothesis, small blast radius, review — in that order.
• Production experiments are where the learning is; staging is warm-up.
• The culture around blame determines whether the practice survives.
• Continuous automation converts individual insights into organisational resilience.

Related Concepts

Pairs with Service Level Objective, Post-Incident Review, On-Call Rotation, and Production Readiness Review.