Secrets rotation without a 3 AM page
Rotation that requires a maintenance window is not rotation — it is deferred risk everyone avoids until credentials leak. Dual-key grace periods let production accept old and new secrets during rollover so nobody gets paged at 3 AM.
The API key was eighteen months old. Security policy said rotate quarterly. Nobody rotated because rotation meant: generate new key, update vault, redeploy every service simultaneously, pray nothing cached the old key, roll back if anything failed. Rotation became a quarterly negotiation that always lost to feature work — until a key appeared in a log export. Secrets rotation without outage is not a vault feature checkbox. It is an architecture choice: systems accept two valid credentials during a grace window, deployments roll gradually, and old keys retire only after traffic proves zero usage.
Secrets that cannot rotate without downtime do not rotate. They age. Aging secrets are predictable breach impact. The goal is rotation boring enough to run on schedule — automated, reversible, and invisible to users.
Why rotation breaks production
| Failure mode | Cause |
|---|---|
| Single-key validation | Service accepts only one active secret |
| Simultaneous deploy required | All consumers must update in same window |
| Cached credentials | SDK or connection pool holds old secret after update |
| Hardcoded env vars | Rotation needs redeploy per instance |
| No usage telemetry | Cannot confirm old key inactive before revoke |
| Shared god-key | Twelve services, one rotation coordinates twelve deploys |
Single-key systems force big-bang rotation — the same failure mode as big-bang schema migrations. Dual-key grace periods apply expand/contract logic to credentials: expand (add new key as valid), migrate (consumers pick up new key), contract (revoke old key after proof).
A rotation runbook that starts with "schedule maintenance" is a document that will not run quarterly.
Dual-key grace period pattern
During grace period, validators accept both key_current and key_next:
Phase 1 — Expand
Vault: key_v1 (active), key_v2 (issued, not yet deployed)
Validators: accept v1 only
Phase 2 — Migrate
Deploy services with key_v2 in config (v1 still works)
Validators: accept v1 OR v2
Monitor: log which key ID signed each request
Phase 3 — Contract
All traffic on v2 (metrics confirm)
Validators: accept v2 only
Revoke v1
Grace period duration: long enough for all consumers to deploy at least once — typically 24–72 hours for daily deploy cadence, longer for weekly deploy teams or external partners.
For JWT signing keys and OAuth client secrets, the same pattern applies with kid (key ID) in token headers — validators look up the correct public key or secret by ID. OAuth scopes and authorization design covers credential scoping; rotation is easier when each service holds narrowly scoped secrets instead of one shared god-key.
Rotation by secret type
| Secret type | Rotation approach | Grace period notes |
|---|---|---|
| API keys (internal) | Dual-key validation at gateway | Log key ID per request |
| Database passwords | User with two passwords or dual DB users | Connection pool drain on deploy |
| TLS certificates | Overlap validity periods | Standard practice — apply same discipline to API keys |
| JWT signing keys | JWKS with multiple kid entries | Publish new key before signing switch |
| Third-party API keys | Provider-dependent; often manual | Stagger: new key live before old revoked |
| CI/CD tokens | Dual valid tokens in pipeline config | CI/CD secrets in production patterns |
Database rotation without outage: create app_user_v2 with new password, grant identical privileges, deploy apps pointing new connections to v2 credentials, drain old connection pools, drop v1 user after connection count zero.
Automation and observability for boring rotation
Rotation should be schedulable and alertable:
- Automated issuance — vault generates
key_nexton schedule. - Deployment hooks — CD pipeline injects latest secret version; no SSH manual updates.
- Key usage metrics — count authentications by key ID; dashboard shows v1 → v2 migration progress.
- Revoke gate — automated check: v1 usage = 0 for N hours before revoke; human ack for production.
- Rollback — if v2 causes errors, validators still accept v1 during grace period.
Alert on: rotation overdue, grace period ending with v1 traffic remaining, failed deploy of new secret version.
Zero trust for internal APIs assumes credentials rotate and scope narrowly — long-lived static internal keys without rotation undermine the model.
Organizational habits that make rotation stick
Rotation owner per secret class. Not "security team" abstractly — named owner for database creds, API gateway keys, CI tokens.
Rotation in deploy path. Services fetch secrets from vault at startup and on interval — not only at build time. Rotation without redeploy is possible when apps refresh credentials.
No secrets in git. Obvious but blocks automation. Rotation of committed secrets requires history rewrite — not rotation.
Partner lead time. External integrations may need weeks notice. Grace periods extend to partner SLA, not only internal deploy cadence.
Game day. Quarterly rotation drill in staging — full dual-key cycle timed and documented. Production rotation should be rehearsed mechanics, not novel surgery.
How should teams implement outage-free secret rotation?
These decisions prevent rotation programs that die after the first failed attempt.
How long should dual-key grace periods last?
At minimum: two full deploy cycles of every consumer plus connection pool TTL. For daily deploys, 48 hours is a reasonable floor. External consumers may need 30 days — document in partner agreements.
Can all secrets use dual-key validation?
No. Some third-party APIs issue one active key. Mitigation: request second key in advance, overlap manually, accept brief risk window. Internal secrets — full control — should always support dual-key.
When should old keys revoke automatically vs manually?
Automate revoke when metrics show zero usage for defined period AND error rate stable on new key. Manual ack for tier-0 secrets (payment, auth root keys). Never auto-revoke without usage telemetry — silent revoke causes outages worse than aging secrets.
A common argument runs the other way
The opposing view holds that frequent rotation increases risk — that changing credentials often creates more deploy-induced failures than it prevents from credential theft.
Rotation frequency should match risk tier. Low-sensitivity internal keys may rotate annually with dual-key support. High-sensitivity keys rotate quarterly or on personnel change. The alternative to rotation is not stability — it is unknown exposure duration when keys leak. Dual-key rotation reduces deploy risk to near zero when grace periods are honored.
NIST and compliance frameworks increasingly expect rotation capability — not rotation theater, but proven ability to change credentials without outage.
Key takeaways
- Rotation requiring maintenance windows does not happen — secrets age until breach.
- Dual-key grace periods: validators accept old and new credentials during rollover.
- Log key ID per authentication to confirm zero old-key usage before revoke.
- Database rotation uses dual users or passwords with connection pool drain.
- Automate issuance, deploy injection, usage metrics, and revoke gates.
- Rehearse full rotation cycle in staging quarterly.
Conclusion
Secrets rotation without a 3 AM page is a design property, not a calendar reminder. Systems that accept two keys during grace windows rotate on Tuesday afternoon like any other deploy. Systems that accept one key rotate in incident postmortems.
The inventory step: list every production secret, mark whether dual-key rotation is supported today, mark last rotation date. Gaps in dual-key support are engineering work; gaps in rotation dates are risk accumulating in plain sight.