Feature flags are release infrastructure, not A/B tests

Teams adopt feature flags for experiments and leave them as permanent `if` statements. Feature flags as release infrastructure decouple deployment from exposure — with kill switches, rollout rings, and mandatory cleanup before the flag count becomes unmaintainable.

Engineering8 min read
Feature flagsDevOpsContinuous deliveryRelease managementProgressive delivery
Share

A production incident at 2 a.m. should not require waiting for a pipeline to finish before users stop seeing broken code. Feature flags as release infrastructure exist for that moment: a configuration change that disables a code path in seconds, without redeploy, without rollback artifacts, without coordinating twelve services. The distinction matters because most teams adopt flags for the wrong primary use case — A/B tests and multivariate experiments — and underinvest in the operational patterns that make continuous delivery safe: progressive rollout, kill switches, ownership, and retirement.

Experimentation is a valid flag type. It is not the foundation. Release infrastructure treats every deploy as code shipped dark — merged, tested in production configuration, invisible to users until a deliberate exposure decision. That inversion changes how teams schedule risk. Deployment becomes routine. Release becomes gated, measured, and reversible.

Deployment and release are the same event without flags

Traditional release trains couple two decisions: when code reaches production, and when users see it. That coupling forces big-bang releases, long-lived feature branches, and deploy windows that everyone dreads. A bug discovered after deploy is already affecting every user. Recovery means revert commits, rebuild artifacts, wait for CI, pray migrations are reversible.

Feature flags split the timeline. Code merges to main and deploys continuously. The flag defaults to off. Production runs the new binary with the old behavior for all users. Validation happens against real infrastructure — connection pools, feature flag evaluation latency, memory profiles — without user exposure. When the team is ready, exposure increases through defined rings: internal staff, 1% canary, 10% beta, general availability.

A kill switch is not an experiment. It is incident response infrastructure.

The rollback for a release flag is a toggle, not a redeploy. Mean time to remediate drops from minutes-to-hours to seconds. That speed only holds if the flag was designed as release infrastructure from the start — with a default-off posture in production, evaluation paths that do not add hundreds of milliseconds to every request, and monitoring tied to the flag's exposure percentage.

Five flag types, five lifecycles

Treating all flags as interchangeable produces debt. A permission flag that gates enterprise billing for three years is not the same artifact as a release flag for a redesigned checkout flow that should die thirty days after full rollout. Categorize at creation; enforce lifecycle rules per category.

TypePurposeDefault in prodRetirement trigger
ReleaseDecouple deploy from user exposureOff until rollout30 days after 100% exposure
Ops / kill switchInstant disable during incidentsContext-dependentNever — long-lived
Permission / entitlementsPlan-tier or role gatingPer product rulesWhen product area sunsets
ExperimentMeasure variant impactSplit trafficWhen hypothesis resolves
MigrationStrangler pattern dual-writeProgressive %When old path deleted

Release flags are the primary source of flag debt. Teams celebrate reaching 100% rollout and forget the if (flags.checkout_v2) branch still runs on every request — adding evaluation latency, cognitive load in code review, and a false sense that the feature could still be "turned off." After stabilization (7–14 days at 100% with metrics clean), the flag comes out. The new path becomes the only path. Tests update. The PR that removes the flag is as mandatory as the PR that added it.

type FlagType = "release" | "ops" | "permission" | "experiment" | "migration"
 
interface FlagMetadata {
  key: string
  type: FlagType
  owner: string
  created_at: string
  expires_at: string | null  // null only for ops/permission
  default_value: boolean
}
 
// Release flag: default off, ring-based rollout
const checkoutV2: FlagMetadata = {
  key: "checkout_v2",
  type: "release",
  owner: "payments-team",
  created_at: "2026-06-01",
  expires_at: "2026-09-01",
  default_value: false,
}

Governance is not bureaucracy. Every flag without an owner becomes nobody's problem until it breaks something. Every flag without an expiration date becomes permanent conditional logic. A weekly review agenda — flags unmodified in 30 days, flags with zero traffic in 7 days, flags past expiration — takes fifteen minutes and prevents the codebase from accumulating hundreds of stale branches.

Progressive delivery rings beat percentage guesswork

Percentage rollouts without gates are hope with a dashboard. Progressive delivery defines explicit rings with entry criteria and exit criteria measured against guardrail metrics: error rate, p99 latency, conversion on the affected funnel, support ticket volume.

A practical ring structure:

  1. Internal — employees and staging identities only. Verify happy path and obvious failures.
  2. Canary — 1–5% of production traffic. Monitor for 24–48 hours minimum unless automated guardrails trip first.
  3. Beta — 10–25%. Enough volume to surface edge cases, not enough to cause company-wide incident.
  4. GA — 100%. Stabilization period before flag removal.

Automated guardrails connect rings to paging. If error rate on the flagged path exceeds baseline by a defined threshold during canary, the flag rolls back to 0% without human intervention. Humans review afterward. The goal is not to remove judgment — it is to remove judgment under sleep deprivation at 2 a.m.

Infrastructure-level canary (two deployments, traffic split at load balancer) and application-level canary (one deployment, flag-controlled code paths) solve different problems. Infrastructure canary catches misconfigured environment variables and container startup failures. Flag canary catches logic errors in the feature itself. Mature teams use both: infrastructure canary validates the deploy artifact; release flags validate the feature behavior inside a stable deploy.

The cloud cost review in pull requests pattern applies equally here — exposure changes should be visible in the same review surface as infrastructure changes. A flag moving from 0% to 10% is a production change. It deserves the same scrutiny as a Terraform diff.

Feature flag evaluation belongs on the hot path budget

Flags that add 50 ms of evaluation latency to every request are not release infrastructure — they are performance debt. Evaluation must be:

  • Local when possible — cached flag definitions, in-memory evaluation, SDK-side bucketing for boolean flags.
  • Fail-safe — default to off (for release flags) when the flag service is unreachable. A flag outage should not take production down.
  • Observable — trace flag evaluations in the same spans as database queries. Know which requests evaluated which flags.

Avoid nested flag conditions scattered through business logic. A single decision point per feature boundary — getCheckoutFlow(user) returns implementation A or B — keeps removal clean. Nested if (flagA && !flagB && flagC) across twelve files makes retirement a multi-week archaeology project.

How should teams treat feature flags as release infrastructure?

These questions separate operational maturity from checkbox adoption.

What is the difference between a release flag and an experiment flag?

A release flag controls exposure of shipped code — default off, progressive rollout, mandatory removal after GA. An experiment flag controls traffic split between variants to measure a hypothesis — default split, success metrics defined upfront, retirement when statistical significance resolves the question. Using experiment infrastructure for releases invites premature cleanup (experiment ends, flag removed, half-built feature exposed). Using release infrastructure for experiments lacks proper bucketing and metric integration.

When should a release flag be removed from code?

Remove within 30 days of reaching 100% exposure with stable guardrail metrics for 7–14 days. The removal PR deletes the old code path, the flag definition, and tests for the old path. Keeping the flag "just in case" indefinitely means every engineer forever maintains two implementations. The kill switch value was real during rollout; after stabilization, the cost of dual paths exceeds the cost of a redeploy rollback.

Do feature flags replace rollback strategy?

No. Flags complement rollback; they do not replace it. A flag disables one feature path inside a deploy. Rollback reverts the entire deploy artifact — necessary when the failure is infrastructural, affects code outside any flag boundary, or involves schema migrations both paths depend on. The code review as risk management frame applies: flags reduce blast radius of feature changes; rollback remains the escape hatch for deploy-level failures.

A common argument runs the other way

The opposing view holds that feature flags add complexity that outweighs their benefit — that disciplined trunk-based development with small batches and good tests makes flags unnecessary, and that flag debt inevitably exceeds the incidents they prevent.

That argument is correct for teams that deploy small changes daily and can revert in minutes. It underestimates the cost of user-facing exposure during the minutes revert takes, and it ignores features that cannot be shipped atomically — large UI rewrites, payment flow changes, migrations with dual-write periods. Flags are not a substitute for good engineering. They are a control surface for the exposure decision that deploy artifacts alone cannot express.

The anti-pattern is not using flags. The anti-pattern is using flags without lifecycle discipline — which is indistinguishable from the opposing view's warning, and exactly what release infrastructure governance prevents.

Key takeaways

  • Feature flags as release infrastructure decouple deploy from exposure — experiments are one flag type, not the primary purpose.
  • Release flags default off in production, roll out through rings with guardrail metrics, and retire within 30 days of 100% exposure.
  • Five flag types — release, ops, permission, experiment, migration — each with distinct lifecycle rules.
  • Kill switches are incident response infrastructure; they justify long-lived flags with explicit ownership.
  • Evaluation must be fast, fail-safe (default off), and observable on the request hot path.
  • Flags reduce feature blast radius; they do not replace deploy-level rollback.

Conclusion

Feature flags earned a reputation as experimentation tooling because vendors marketed split testing first. The deeper value is operational: shipping code without shipping risk, turning off broken paths without turning off the deploy pipeline, and making release a configuration decision that humans or guardrails control in seconds.

The maturity test is not how many flags exist. It is how many expired release flags remain in code, whether every flag has an owner and retirement date, and whether the last incident was mitigated by a toggle or by a redeploy. Teams that pass that test treat flags as infrastructure. Teams that fail it have conditional spaghetti with a SaaS bill attached.

Related articles

Command Palette

Search for a command to run...