Secrets Rotation Policy

Definition

A secrets rotation policy defines which credentials — database passwords, API keys, signing keys, service-account tokens, TLS certificates — are rotated, how often, by what mechanism, and what happens in an emergency rotation after suspected compromise. A workable policy is inseparable from the machinery that implements it: a centralised secret store, short-lived dynamic credentials where possible, and applications that can pick up a new value without a redeploy.

Why It Matters

Credentials leak in mundane ways: committed to a repository, pasted into a support ticket, captured in a debug log, or carried away on a laptop by a departing contractor. Rotation limits the blast radius by bounding how long a leaked secret remains useful. The uncomfortable truth is that most organisations discover their rotation capability during an incident, when someone asks how quickly the production database password can be changed and the honest answer involves a maintenance window and four teams.

How to Make Rotation Routine

1. Inventory first. You cannot rotate what you cannot list. Scan repositories, CI configuration, container images and infrastructure code.
2. Centralise storage. One secret manager with audit logging, versioning and fine-grained access, referenced at runtime rather than baked into images.
3. Prefer short-lived dynamic credentials. A database credential minted per session with a one-hour lifetime removes most of the rotation problem entirely.
4. Support two valid secrets at once. Dual-credential rotation — create new, deploy, verify, revoke old — is what makes zero-downtime rotation possible.
5. Automate on a schedule. Ninety days for static credentials is a common baseline; certificates should renew automatically well before expiry.
6. Rehearse emergency rotation. Time a full rotation of the highest-value secret and publish the number.

Real-World Example

A platform team found an active cloud access key in a public repository's commit history from fourteen months earlier. Revoking it took thirty seconds. Understanding what depended on it took eleven hours: the key was embedded in two container images, a legacy cron host, and a partner integration nobody in the current team had built. Two of those broke on revocation, one at 02:00. The post-incident work was more valuable than the incident: every static key moved into the secret manager, images were rebuilt to read credentials at startup, and the team adopted dual-credential rotation with a ninety-day automated schedule. Six months later, a routine rotation of the same class of key took four minutes with no customer impact — and, crucially, nobody hesitated to do it.

Practical Lessons Learned

• Rotation pain is a dependency-mapping problem. The change is trivial; knowing who uses the secret is not.
• Baked-in secrets force redeploys. Read at startup or, better, refresh at runtime.
• Certificates expire at inconvenient hours. Automate renewal and alert at 30 days, not 3.
• Rotation without revocation is theatre. The old credential must actually stop working, and someone must verify it did.
• Third-party keys need lead time. Partner integrations often require coordinated changes, so schedule them rather than surprising them.

Expert Tips

• Tag every secret with an owning team and the systems that consume it. That metadata is what turns an emergency into a routine change.
• Add secret scanning to pre-commit hooks as well as CI. Blocking the commit is far cheaper than rotating after a push.
• Alert on use of the old credential after rotation — it is the fastest way to find the consumer nobody documented.
• Rotate one production secret deliberately each quarter as a drill, in business hours, with everyone watching.
• Treat signing keys differently: they need overlap periods and published key IDs, because consumers cache them.

Common Mistakes

• A policy document requiring 90-day rotation with no automation, so the policy is quietly ignored.
• Secrets in environment variables baked into images, making rotation a full release.
• Rotating without a dual-credential window, guaranteeing downtime and therefore avoidance.
• Excluding service accounts and CI tokens from the inventory, which is where the most powerful credentials usually live.
• Never revoking the old value "just in case", which defeats the entire exercise.

Key Takeaways

• Rotation capability is measured in minutes to revoke, not in policy pages.
• Inventory and ownership metadata make emergency rotation survivable.
• Dual-credential windows are what make rotation zero-downtime.
• Short-lived dynamic credentials beat rotating static ones.
• Practise a rotation in daylight so the emergency one is boring.

Related Concepts

Pairs with Production Readiness Review, Dependency Upgrade Cadence, Backup Restore Drill, and Post-Incident Review.