Circuit breakers and failure isolation for production APIs

The payment provider returned HTTP 503 for forty seconds. The checkout service retried three times per request. Each retry held a connection for thirty seconds. Connection pools saturated. Unrelated read endpoints timed out because every worker was blocked waiting on a vendor nobody could fix from inside the application. The dependency did not crash the service — it occupied it. Circuit breaker failure isolation exists to convert dependency failure from a fleet-wide stall into a bounded, fast-fail surface with a defined fallback.

A circuit breaker is a state machine wrapped around outbound calls. While healthy, calls pass through (closed). After enough failures or slow responses, the breaker opens and rejects new calls immediately — no socket wait, no retry storm. After a cooldown, a trickle of half-open probes tests recovery before closing again. The pattern is decades old. Production implementations still fail because teams treat the library as the architecture instead of defining what failure means, what to return when the circuit opens, and who gets paged when half-open probes keep failing.

Failure isolation starts with blast-radius boundaries

Failure isolation draws a line around what one dependency outage may affect. Without that line, retries propagate load backward into the caller, thread pools exhaust, and health checks fail on services that never called the broken dependency directly.

Circuit breakers sit between timeouts and bulkheads. Timeouts cap how long one call may wait. Breakers cap how many calls may wait at all once the dependency is clearly unhealthy. Bulkheads ensure the analytics circuit opening does not drain the pool used for authentication.

A circuit breaker without a fallback is a faster 503. A fallback without a circuit breaker is a slower one.

Define the fallback before production traffic hits the dependency: cached response, degraded feature, queued async job, or explicit error with a user-visible message. "Return null and hope upstream handles it" is not a fallback — it is an undocumented contract that becomes a data-integrity incident during the next deploy.

Circuit breaker states and the thresholds that matter

The state machine is simple. The tuning is not.

Closed — all calls execute. Counters track failures, slow responses, and optionally consecutive errors. A single 500 from a flaky endpoint should not open the circuit; sustained failure should.

Open — calls fail fast without network I/O. The caller returns the fallback or error immediately. No threads block. No retries fire against a known-sick peer.

Half-open — after a sleep window, one or a few probe requests test the dependency. Successes close the circuit; failures re-open it and reset the sleep window.

Threshold design separates useful breakers from noisy ones:

failure_rate_threshold = failures / (failures + successes) over rolling window
slow_call_threshold    = responses slower than T ms count as failures
minimum_calls          = do not evaluate until N requests in window (avoid opening on 1/1)
wait_duration_open     = how long to stay open before half-open probes
permitted_half_open    = max concurrent probes in half-open state

Concrete starting points for HTTP dependencies (adjust with measured baselines):

Sliding windows beat fixed counts for bursty traffic. Per-dependency breakers beat one global breaker — a global breaker turns a search outage into a login outage.

Instrument state transitions: circuit_opened, circuit_half_open, circuit_closed, tagged by dependency name and caller service. Those events belong on the same dashboard as SLIs and SLOs without an SRE team — a circuit stuck open for twenty minutes is a user-visible reliability event even if error rates look acceptable because fallbacks mask the failure.

Retries and circuit breakers must be coordinated

Retries multiply load. A circuit breaker exists partly to stop retry multiplication against a dependency that cannot absorb it.

Anti-pattern: retry three times with exponential backoff, then wrap the whole retry loop in a breaker that only sees the final failure. The dependency receives three times the traffic while the breaker thinks failure rate is low because the client absorbed two successes-before-timeout.

Better pattern:

• Apply the breaker per attempt or count each failed attempt toward the failure window.
• Cap total retry budget with a deadline context shared across attempts.
• Disable retries when the circuit is open — fail fast at the breaker boundary.
• Use idempotent operations only on retry paths; non-idempotent POST without idempotency keys should not retry blindly.

async function callPaymentProvider(
  ctx: RequestContext,
  payload: ChargeRequest,
): Promise<ChargeResult> {
  if (paymentBreaker.isOpen()) {
    return fallbackQueue.enqueue(payload)
  }
 
  const deadline = Date.now() + 2_000
  let lastError: Error | undefined
 
  for (let attempt = 0; attempt < 2; attempt++) {
    if (Date.now() > deadline) break
 
    try {
      const result = await paymentClient.charge(payload, {
        signal: AbortSignal.timeout(800),
      })
      paymentBreaker.recordSuccess()
      return result
    } catch (err) {
      lastError = err as Error
      paymentBreaker.recordFailure()
      if (paymentBreaker.isOpen()) break
    }
  }
 
  return fallbackQueue.enqueue(payload)
}

When the circuit opens during checkout, the incident is not "payment API down" alone — it is "orders entering a reconciliation queue." That behavior must appear in incident response runbooks with verification steps: queue depth, age of oldest message, duplicate charge checks when the provider recovers.

What belongs in a production circuit breaker configuration?

When should a circuit open on latency instead of errors?

Open on latency when the dependency returns HTTP 200 with unusable slowness — a common failure mode for overloaded databases and rate-limited APIs that respond slowly instead of rejecting. Treat responses slower than the caller's deadline as failures for breaker purposes even if the status code is success. Pair with upstream timeouts so slow successes never hold resources longer than the user-facing SLO allows.

How does half-open probing avoid recovery flapping?

Flapping happens when one successful probe closes the circuit, traffic floods back, and the dependency fails again within seconds. Mitigations: require multiple consecutive successes in half-open before closing; ramp permitted concurrency gradually; use a longer open duration after a re-open than after the first open. Log each half-open failure — repeated failures often indicate the dependency is recovering in bursts, not fully healed.

Should fallbacks write to the database or only return cached data?

Read paths: stale cache with a stale-while-revalidate header or response field is usually safe when the product tolerates minutes of lag. Write paths: prefer durable enqueue over silent drop. A fallback that discards a payment intent is worse than a slow payment. Document whether the fallback path is lossy, at-least-once, or exactly-once and test recovery accordingly.

A common argument runs the other way

The opposing view holds that circuit breakers hide dependency problems, encourage lazy fallbacks, and add operational complexity better spent fixing root causes — that timeouts and good SLOs on upstream vendors are sufficient.

Timeouts alone do not stop retry storms from hundreds of instances. Vendor SLOs do not prevent a regional DNS blip from saturating connection pools before anyone reads the status page. Circuit breakers do not replace fixing dependencies; they bound how long the application participates in cascading failure while fixing happens. Teams that never open a circuit either set thresholds too loose or do not instrument state transitions — both are configuration gaps, not evidence the pattern failed.

Key takeaways

• Circuit breaker failure isolation converts dependency sickness from thread starvation into fast-fail with a defined fallback.
• Per-dependency breakers, sliding windows, and minimum call counts prevent opening on noise.
• Retries must count toward breaker failure rates; never retry into an open circuit.
• Half-open probes need consecutive-success rules to avoid recovery flapping.
• Instrument open, half-open, and close events alongside SLIs — masked fallbacks still consume error budget.
• Runbooks must cover fallback paths: queue depth, reconciliation, and duplicate-operation checks.

Conclusion

Dependencies fail in slow motion more often than they fail with a clean connection refused. The fleet pays for that slowness when every service retries indefinitely into the same saturated endpoint. Circuit breakers are the admission control layer for outbound trust: explicit thresholds, explicit fallbacks, explicit recovery probes.

The next dependency integration should ship with a breaker configuration reviewed in the same PR as the client wrapper — not added after the first outage fills the postmortem with "retry storm." Failure isolation is a design choice. It is cheaper before production traffic proves the default is no isolation at all.