Rollback strategy: the deploy step nobody writes

Deploy pipelines get rehearsed every sprint. Rollback gets documented once in a wiki page nobody opens during incidents. The asymmetry shows up when error rates spike twelve minutes after a release: engineers debate whether to revert, whether the migration is reversible, who has authority to call it, and whether down.sql was ever tested against production-shaped data. Rollback strategy for deploy is the missing artifact — not another script, but a decision tree written before the release, with triggers, owners, safe actions, unsafe actions, and verification steps short enough to execute under pressure.

Application rollback is fast when artifacts are immutable and tagged. Data rollback is slow, sometimes impossible, and often the real constraint. A rollback strategy that only addresses code — "redeploy previous tag" — fails the first time a migration dropped a column the old binary still reads.

Rollback fails when schema and code move together without a plan

Modern deploys touch two surfaces: application binaries and database schema. They roll back at different speeds and with different safety profiles.

Surface	Rollback mechanism	Typical time	Risk
Application	Redeploy previous tag / image	2–15 minutes	Low if artifact exists
Feature flags	Toggle off	Seconds	Low for flag-gated paths
Schema (expand)	Old code still works	N/A — forward compatible	Low during expand phase
Schema (contract)	Irreversible without data loss	Hours to never	High
Data backfill	Forward fix or restore	Hours	Very high

Expand/contract migrations exist because schema rollback is hard. Expand adds new schema alongside old — nullable column, new table, dual-write path. Migrate moves traffic and data gradually. Contract removes old schema only after proof nothing reads it. Rollback during expand often means revert application code while leaving new columns in place — safe if columns are nullable and ignored by old code. Rollback during contract may be impossible without restore from backup.

A down migration nobody ran in staging is not a rollback plan. It is hope.

The feature flags as release infrastructure pattern reduces the need for code rollback for user-facing paths — toggle off in seconds. It does not reverse schema changes. Flags and rollback strategy complement each other; neither replaces the other.

A rollback strategy is a one-page decision tree

The runbook fits on one page because nobody reads prose during incidents.

Header block

• Release version and tag
• Migration identifiers applied
• Rollback owner and backup authority
• Current expand/contract phase

Triggers — when to roll back

Define numeric thresholds, not "if something looks wrong":

• Error rate on affected endpoints > 2× baseline for 5 minutes
• p99 latency > 1.5× baseline for 10 minutes
• Payment or auth failure rate any sustained increase
• Customer-facing functional test failure post-deploy
• Manual trigger: on-call engineer or release owner

Decision branches

Trigger fired?
├─ Feature-flagged path only → Toggle flag OFF (30 sec)
│   └─ Verify metrics → Done or escalate to app rollback
├─ App bug, schema forward-compatible → Redeploy previous tag
│   └─ Run smoke tests → Monitor 30 min
├─ App bug, schema NOT compatible → Forward-fix OR restore (see unsafe)
└─ Migration caused issue → STOP app rollback if unsafe
    ├─ Expand phase → Revert app; pause backfill; schema stays
    └─ Contract phase → Forward-fix migration; do NOT run down.sql

Unsafe actions — list explicitly

• Running untested down migrations against production
• Dropping columns to "match" old code after data was written to new schema
• Reverting app while background workers write new-format data
• Restoring database backup without coordinating app version

Verification after rollback

• Smoke test checklist (auth, core CRUD, payment if applicable)
• Compare error rate and latency to pre-deploy baseline
• Query confirming no orphaned data states (e.g., rows stuck in migrating status)
• Communication: who notifies stakeholders, what to post in status page

Rehearse the tree once per quarter. Untested rollback is folklore.

Application rollback requires immutable artifacts

Redeploying previous version only works if the previous version exists and is known good.

• Tag every production deploy: v1.4.2 on immutable container image or build artifact.
• Record last known-good tag in release notes — not "whatever was there before."
• CI produces the same artifact for a given tag forever — no rebuild-from-source during incidents unless artifact registry failed.
• Smoke tests run automatically against new deploy; define whether they block or warn.

Target: application rollback completes in under fifteen minutes from trigger to verified recovery. If the pipeline takes forty minutes, rollback strategy is theoretical.

Database changes use forward-fix when backward rollback is unsafe: ship a new migration that repairs state, keep app at failed version or roll back app separately depending on compatibility. Liquibase rollback and framework down files are development tools — production incidents need reviewed forward paths, not auto-generated reverse SQL against live data.

Schema rollback coordinates with app version

The compatibility matrix must be written before deploy:

App version	Schema version	Safe?
N+1	expand (new nullable col)	Yes — deploy new app
N	expand (new nullable col)	Yes — old app ignores new col
N	contract (col dropped)	No — old app breaks
N+1	N schema	Usually yes if N+1 is backward compatible

If N+1 migration is not backward compatible with N app, simultaneous rollback requires forward-fix, not tag revert alone. Coordinate CI/CD secrets and pipeline access — rollback deploys use the same privileged paths as forward deploys; protect both equally.

What belongs in a production rollback runbook?

These elements separate rehearsed recovery from improvised damage.

Who can authorize rollback?

Name a primary and backup authority — on-call engineer, release owner, or engineering lead. "Team consensus" is not authority at 2 a.m. Pre-authorized triggers (automated flag off on error threshold) reduce debate for known paths.

When is app rollback safe after a migration?

Safe when the migration is expand-phase and backward compatible: old code ignores new schema elements, no required new columns on write paths old code uses, no data exclusively in new tables old code cannot read. Unsafe when contract-phase ran, when backfill wrote data old code misinterprets, or when down migration was never tested.

What is the forward-fix path when rollback is impossible?

Document: issue hotfix branch, migration to repair data, deploy hotfix app version, verification queries, timeline estimate. Point-in-time database restore is last resort — name RPO/RTO, who invokes restore, how app version aligns with restored data snapshot.

A common argument runs the other way

The opposing view holds that rollback culture encourages reckless deploys — that teams should invest in never needing rollback through better testing, canary deploys, and feature flags.

Canary and flags reduce rollback frequency. They do not eliminate it. Infrastructure misconfiguration, third-party outages masked as code errors, and migration edge cases still happen. Rollback strategy is insurance — the goal is never using it, the requirement is that it works when needed.

Zero rollback incidents is not the metric. Mean time to recovery when rollback is the right call is the metric.

Key takeaways

• Rollback strategy is a one-page decision tree — triggers, authority, branches, unsafe actions.
• Application rollback needs immutable tagged artifacts; schema rollback needs expand/contract discipline.
• Down migrations untested in production are not rollback plans.
• Feature flags roll back exposure in seconds; they do not reverse schema.
• Define numeric triggers, not subjective "looks bad."
• Rehearse rollback quarterly; forward-fix is the safe path when contract-phase ran.

Conclusion

Deploy pipelines are practiced because they run every week. Rollback is ignored because it is uncomfortable to imagine failure. The teams that recover in minutes wrote the uncomfortable part down before the incident: what to revert, what never to revert, who decides, and how to verify recovery.

The next release should ship with a rollback section in the release notes — not after. If the section is empty because "we'll figure it out," the release is not ready.