Circuit breakers and retries with jitter

When a dependency gets slow, every caller piles up waiting on it, holding threads and connections. Your service dies of someone else's outage. That is a cascading failure, and it is how one small problem can take down a whole platform.

Retries make it worse. The dependency is struggling, so everyone retries, so it receives more traffic than before and cannot recover. The fix is to fail fast when things are bad, and to retry in a way that does not line everyone up together.

Start with this: put a timeout on every remote call. Without one, a dependency that hangs holds your resources forever, and no amount of clever breaker logic saves you. Set the timeout from the real latency distribution, a bit above p99, not a round number someone guessed.

The breaker states. Closed means calls flow while failures are counted. Once the failure rate crosses a threshold over a meaningful window, it opens, and calls fail immediately without waiting. You stop spending resources on something that will not answer. After a cooldown it goes half-open and allows one trial call. Success closes it. Failure opens it again. Use a failure rate over a window rather than a raw count, or a quiet endpoint takes an hour to trip and a busy one trips on a hiccup.

Retries, done properly. Only retry things that are safe to repeat, or that carry an idempotency key. Only retry errors that might be temporary, like a timeout or a 503, never a 400. Cap the attempts at two or three. Use growing delays between attempts, and add randomness so callers do not line up. Without that randomness, a thousand clients that all failed at the same moment all retry at the same moment, and the recovering service is knocked over by the wave. The randomness is one line of code, and it can decide whether the service recovers or suffers a second outage.

One refinement worth mentioning is a retry budget: the whole client is allowed to spend only a small percentage of its traffic on retries. That caps the amplification no matter how many individual calls decide to retry, which matters most in a deep chain of calls. Three layers that each make three attempts turn one user request into 27 at the bottom.

Bulkheads are the companion idea. Give each dependency its own limited pool of connections or threads, so a slow one uses up only its own share and the rest of your service keeps working. Without that, one slow dependency eats every thread and takes down endpoints that never called it.

Then say what happens when the breaker is open, because that is what decides the user experience. Serve old cached data, fall back to a simpler response, drop an optional feature, or return a clear error. Decide this per dependency in advance, or the default is a blank page.