This resume is also available in English.

4. Reliability1 / 4 · All columns

Columns › 4. Reliability › Part 13

Timeouts, Retries and Circuit Breakers

逾時、重試與熔斷

Downstream services will be slow or fail. Starting from deadline budgets, this article builds retries with exponential backoff and jitter, retry budgets, a three-state circuit breaker in Go and bulkheads, so the order service survives misbehaving dependencies.

  • Go
  • Microservices
In this article
  1. What this article solves
  2. Step 1: Every call needs a deadline
  3. Step 2: Telling which errors are worth retrying
  4. Step 3: Exponential backoff plus jitter
  5. Step 4: Retry budgets to avoid retry amplification
  6. Step 5: The three states of a circuit breaker
  7. Step 6: Implementing a circuit breaker in Go
  8. Step 7: Bulkhead isolation and fallbacks
  9. Common pitfalls
  10. Key takeaways
  11. Next

In the previous article, “Service Discovery and Load Balancing for Go Microservices”, we let the order service find every instance of the inventory and payment services and spread requests evenly across them; this article deals with what to do when "we found it, but it is slow, or it returns an error".

What this article solves

In example.com/shop, placing an order works like this: api-gateway forwards the request to order, order first calls inventory to reserve stock, then calls payment to charge, and finally changes the order status from pending to paid. In the monolith era these were all function calls, and a failure was just an error; once split into services, every call adds the network, serialisation, the other side's connection pool and garbage collection, and any link in that chain can make the call slow, stuck or failing.

The most dangerous thing is not "failure" but "slowness". With a downstream that returns 503 straight away, the caller knows the result quickly; a downstream that hangs for thirty seconds before replying ties up the caller's goroutines, connection pool and memory, requests pile up, and eventually the whole chain falls over together. This is what is called a cascading failure. This article uses four tools to stop it:

ToolProblem it solvesIn one sentence
Timeouts (deadlines)Hanging without replyingEvery call needs an upper bound, and the whole chain shares one deadline
RetriesTransient failuresRetry only errors that might succeed, and back off with randomness
Circuit breakingPersistent downstream failureWhen there are too many failures, stop calling for a while: give the other side room to breathe and fail fast yourself
BulkheadsOne downstream dragging down everythingEach downstream uses its own resource pool, so they don't compete

These four tools are interrelated: without timeouts, retries never get to see a "failure"; without circuit breaking, retries hammer an already sick downstream even harder; without bulkheads, one slow payment service eats up the resources needed to call the inventory service. Next we layer them on one step at a time.

Step 1: Every call needs a deadline

The first rule is simple: no call that crosses the network may be unbounded. Go's http.Client has a default Timeout of zero, which means wait forever; an unconfigured net.Dial may also wait for an operating-system-level timeout, usually tens of seconds or longer. In practice you need upper bounds in at least two places:

  • The client as a whole: http.Client{Timeout: ...} or a gRPC call deadline, so that no single request can hang indefinitely.
  • Each request: wrapped with context.WithTimeout or context.WithDeadline, so cancellation from upstream propagates all the way down.

“Go context: Cancellation, Timeouts and Request-Scoped Values” has already covered the basics of context; the point to stress here is a problem specific to microservices: timeouts should be a "deadline budget", not a separate number for each hop.

Suppose api-gateway promises to respond within two seconds. If order gives its own two-second timeout when calling inventory, and inventory gives another two seconds when calling the database, then in the worst case the user has long since received a timeout error from the gateway, and may even have hit refresh, while the services behind it are still diligently doing work nobody is waiting for. This "orphaned work" consumes resources and may also cause side effects (for example, the stock has been reserved but the order has already been treated as failed).

The right approach is to set a deadline at the entrance and pass the same deadline down. gRPC automatically puts the context's deadline into the grpc-timeout header for the downstream; HTTP has no standard header, so a common approach is a custom header (for example carrying the remaining milliseconds), which the other side's middleware turns back into a context.WithTimeout. Using the remaining time rather than an absolute time avoids problems with the two machines' clocks being out of sync.

API Gateway deadline = now + 2.0s Order service 1.9s left on arrival Inventory: reserve stock used 0.6s, 1.2s left on return Payment: charge used 0.5s, 0.7s left on return Every hop passes the same deadline down instead of setting a fresh timeout of its own Reserve stock 0.6s Charge 0.5s 0.7s left to confirm order Gateway Order 0s 2.0s deadline

Figure: the gateway sets a two-second deadline; by the time the order service receives the request only about 1.9 seconds are left, reserving stock and charging each use up part of it, and what remains at the end is the time available for confirming the order (the numbers are only illustrative).

With a deadline budget, every service can do two clever things. First, check how much time is left when a request arrives: if what's left is less than your own fastest processing time, return an error straight away rather than starting. Second, take an even shorter deadline when calling a downstream, for example "the smaller of the remaining time and 800ms", so even if upstream grants a generous deadline, a single downstream can't eat the whole budget, leaving some for later steps.

Tip: deadlines can only get shorter, never longer. The child context produced by context.WithTimeout(parent, d) has a deadline that is the earlier of "the parent's deadline" and "now plus d", so you can safely take a shorter deadline when calling downstream without accidentally extending the upstream promise.

How do you choose the timeout value? A common starting point is the downstream's latency distribution: set the timeout at a few times the p99 latency rather than a few times the average. Set it too short and the downstream being occasionally a little slow is treated as failure, the failure rate rises and retries increase the load; set it too long and it loses its protective value. There is no standard answer for the number; what matters is having a basis for it and keeping an eye on it with the metrics discussed in “Observability in Go: Logs, Metrics and Tracing”.

Step 2: Telling which errors are worth retrying

With deadlines in place, calls can finally "fail", and the next question is: should you try again after a failure? Many people's first reaction is to retry everything three times, which is usually wrong. A retry only makes sense when both of these conditions hold:

  1. The error is transient: trying again has a reasonable chance of succeeding.
  2. The operation is idempotent: even if the first attempt actually succeeded and only the response was lost, doing it again won't cause duplicate side effects.

The first condition depends on classifying errors. Taking HTTP and gRPC as examples, they can be roughly classified like this:

CategoryHTTPgRPCRetry?
Downstream temporarily unavailable503, connection refusedUnavailableYes, after backing off
Rate limited429ResourceExhaustedYes, but honour Retry-After
Single attempt timed outClient timeoutDeadlineExceededDepends on idempotency and remaining budget
Bad arguments400, 422InvalidArgumentNo, the result will always be the same
Permissions or authentication401, 403PermissionDeniedNo
Business outcome409, card declinedFailedPreconditionNo, this is not a fault

The second condition is even easier to overlook. GET /orders/{id} is naturally idempotent and retrying it is harmless; but with POST /payments, if the first request actually charged successfully and only the response was lost on the network, a retry charges twice. This situation of "not knowing whether the other side actually did it" is the norm in distributed systems, not the exception. So for calls with side effects, either don't retry, or first make them idempotent (with an idempotency key), which is exactly the subject of the next article, “Idempotency and Distributed Locks”.

In Go, error classification is best done in the client adapter layer (for example the client in internal/payment), expressed with the sentinel errors or custom types introduced in “Error Handling in Go: Wrapping, Sentinels and Custom Types”, so that upper layers check with errors.Is instead of comparing status-code strings all over the place.

Step 3: Exponential backoff plus jitter

Once you've decided to retry, the next question is "how long to wait before trying again". Retrying immediately is almost useless: if the downstream failed because it was overloaded, an immediate retry only overloads it further. The standard approach is exponential backoff: wait base the first time, 2×base the second time, 4×base the third time, with a cap so the wait doesn't grow without limit.

But exponential backoff alone isn't enough. Imagine the payment service has a one-second outage, and within that second a thousand requests fail; they all retry after 100ms, then again after 300ms… The result is that every wave of retries floods in at the same instant, creating a "thundering herd", and the downstream is knocked over again just as it recovers. The solution is to add jitter, so each client waits a different amount of time.

Fixed exponential backoff (no jitter) 0 100ms 300ms 700ms Thousands of clients retry at once (one bar = a big wave of requests) Exponential backoff + full jitter 0 Each wait is random in [0, min(cap, base×2ⁿ)), so retries spread over the whole period

Figure: the top shows fixed exponential backoff, with every client retrying at the same moment; the bottom adds full jitter, spreading retries over the whole period, so the downstream sees a smooth load.

There are several common jitter strategies. Full jitter takes a uniformly random value in [0, min(cap, base×2ⁿ)); it spreads retries best and is also the simplest. Equal jitter takes half fixed and half random, guaranteeing at least some wait. Decorrelated jitter makes the next wait depend on the previous random result. For most services, full jitter is good enough.

The program below combines the first three steps: each attempt has its own timeout, the whole has a deadline budget, only transient errors are retried, waits use full jitter, and it gives up early when the budget isn't enough for another attempt.

package main

import (
	"context"
	"errors"
	"fmt"
	"math/rand/v2"
	"time"
)

// Transient error: a retry might succeed
var ErrUnavailable = errors.New("inventory: unavailable")

// Permanent error: the result is the same however many times you retry
var ErrInvalid = errors.New("inventory: invalid sku")

type RetryPolicy struct {
	MaxAttempts int           // maximum number of attempts in total, including the first
	Base        time.Duration // upper bound of the first backoff
	Max         time.Duration // ceiling on the backoff upper bound
	PerTry      time.Duration // each attempt's own timeout
}

func retryable(err error) bool {
	// A single attempt timing out (not the overall deadline) also counts as retryable
	return errors.Is(err, ErrUnavailable) || errors.Is(err, context.DeadlineExceeded)
}

// fullJitter returns a random wait in [0, min(Max, Base*2^attempt))
func fullJitter(p RetryPolicy, attempt int) time.Duration {
	ceil := p.Base << attempt
	if ceil <= 0 || ceil > p.Max {
		ceil = p.Max
	}
	return rand.N(ceil)
}

func Do(ctx context.Context, p RetryPolicy, op func(context.Context) error) error {
	var err error
	for attempt := 0; attempt < p.MaxAttempts; attempt++ {
		tryCtx, cancel := context.WithTimeout(ctx, p.PerTry)
		err = op(tryCtx)
		cancel()
		if err == nil {
			return nil
		}
		if ctx.Err() != nil {
			return fmt.Errorf("overall deadline reached, giving up: %w", err)
		}
		if !retryable(err) || attempt == p.MaxAttempts-1 {
			return err
		}
		wait := fullJitter(p, attempt)
		// If the remaining budget isn't enough for "wait + one more attempt", don't waste it
		if dl, ok := ctx.Deadline(); ok && time.Until(dl) < wait+p.PerTry/2 {
			return fmt.Errorf("not enough budget for another attempt: %w", err)
		}
		fmt.Printf("  attempt %d failed (%v), retrying in %v\n", attempt+1, err, wait.Round(time.Millisecond))
		t := time.NewTimer(wait)
		select {
		case <-ctx.Done():
			t.Stop()
			return fmt.Errorf("deadline reached while waiting to retry: %w", err)
		case <-t.C:
		}
	}
	return err
}

func main() {
	p := RetryPolicy{MaxAttempts: 4, Base: 20 * time.Millisecond, Max: 200 * time.Millisecond, PerTry: 80 * time.Millisecond}

	// Scenario 1: the inventory service is temporarily unavailable twice, then succeeds on the third attempt
	calls := 0
	reserve := func(ctx context.Context) error {
		calls++
		if calls <= 2 {
			return ErrUnavailable
		}
		return nil
	}
	ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
	err := Do(ctx, p, reserve)
	cancel()
	fmt.Printf("scenario 1: calls=%d, err=%v\n", calls, err)

	// Scenario 2: bad arguments, should not be retried
	calls = 0
	bad := func(ctx context.Context) error { calls++; return ErrInvalid }
	ctx, cancel = context.WithTimeout(context.Background(), 500*time.Millisecond)
	err = Do(ctx, p, bad)
	cancel()
	fmt.Printf("scenario 2: calls=%d, err=%v\n", calls, err)

	// Scenario 3: every attempt hangs until the per-try timeout, and the overall budget runs out quickly
	calls = 0
	slow := func(ctx context.Context) error {
		calls++
		<-ctx.Done()
		return ctx.Err()
	}
	ctx, cancel = context.WithTimeout(context.Background(), 200*time.Millisecond)
	err = Do(ctx, p, slow)
	cancel()
	fmt.Printf("scenario 3: calls=%d, err=%v\n", calls, err)
}

Running it shows the difference between the three scenarios (the random waits differ on every run):

  attempt 1 failed (inventory: unavailable), retrying in 1ms
  attempt 2 failed (inventory: unavailable), retrying in 21ms
scenario 1: calls=3, err=<nil>
scenario 2: calls=1, err=inventory: invalid sku
  attempt 1 failed (context deadline exceeded), retrying in 15ms
scenario 3: calls=2, err=not enough budget for another attempt: context deadline exceeded

This code has a few details worth noting. First, the ctx.Err() != nil check comes before classification: if the overall deadline has passed, you mustn't try again whatever the error looks like. Second, context.DeadlineExceeded is treated as retryable because it refers to the timeout of the child context for a "single attempt", while the parent context still has time; this is only suitable for idempotent operations. Third, the wait uses select to listen on ctx.Done() at the same time, so it doesn't blindly sleep to the end when upstream cancels. Fourth, rand.N from math/rand/v2 can produce a random value of type time.Duration directly, and the global source is seeded automatically, so you don't have to handle seeding yourself.

Step 4: Retry budgets to avoid retry amplification

Even if every call backs off correctly, retries have a system-wide problem: the amplification effect. Suppose the call chain has three layers and each layer retries three times; during an outage the bottom layer receives 3×3×3 = 27 times the requests. The downstream is already struggling, yet retries make it carry dozens of times the load, which is a common pattern in many major incidents.

There are three complementary principles for handling amplification:

  • Retry at only one layer. Usually choose the layer closest to the user, or the one that best understands idempotency. Other layers pass errors upwards on failure instead of adding another round of their own.
  • Limit the number of retries. Two to three attempts in total is enough; more usually just postpones the failure.
  • Use a retry budget. Rather than only limiting the retries for a single request, limit "the proportion of total requests that are retries". For example, rule that retries may not exceed one tenth of normal requests: in normal times, with the occasional failure, retries are almost always allowed; during a large-scale downstream outage, the retry quota runs out quickly and further failures go straight back upstream instead of adding more load.

A retry budget can be implemented very simply: use a token bucket, put 0.1 tokens in for every successful normal request, spend 1 token for each retry, and stop retrying when the bucket is empty. gRPC's retry policy has a similar retryThrottling setting (described in terms of a token count and ratio); when using gRPC, prefer the mechanism the framework provides instead of wrapping another layer of retries yourself.

Tip: when you're rate-limited (429 or ResourceExhausted), if the other side sent a Retry-After, wait exactly as long as it says; that is the downstream telling you explicitly when it will be free, which is more accurate than any backoff formula.

Step 5: The three states of a circuit breaker

Retries deal with "occasional failure", but what if the payment service is completely down for five minutes? During those five minutes, every order request dutifully waits for the timeout, retries, waits for the timeout again; users wait several seconds just to receive a failure, and all the order service's goroutines and connections are stuck waiting. Worse, the moment the downstream recovers, the accumulated requests rush in together and may knock it over again.

The circuit breaker borrows the idea of an electrical fuse: when it detects that the downstream keeps failing, it "trips" straight away, and subsequent calls are no longer sent but return an error immediately; after a while it lets a small number of requests through to probe, and returns to normal only once recovery is confirmed. It has three states:

Closed allow, count failures Open return ErrOpen at once Half-open only a few probe requests failures reach threshold wait openTimeout probe fails probe succeeds success or business error: failure count reset

Figure: when Closed, requests pass normally and failures are counted; when failures reach the threshold it jumps to Open and rejects directly; after a wait it enters Half-open and lets only a few probe requests through, returning to Closed on success and to Open on failure.

  • Closed: the normal state; requests are sent as usual while failures are counted. Failures can be counted as "consecutive failures" or as "the failure rate within a recent time window"; the latter is more stable for high-traffic services, and is usually combined with a condition like "at least N requests in the window before judging", to avoid tripping on one or two failures at low traffic.
  • Open: entered once failures exceed the threshold. During this period every call immediately returns ErrOpen without touching the network. The caller can therefore fail fast and switch to fallback logic, for example replying "the payment system is busy; your order has been held, please pay later".
  • Half-open: entered after Open has lasted a while. Only a small number (commonly one) of probe requests are let through; a successful probe means the downstream has recovered, so it returns to Closed and clears the counts; a failed probe sends it back to Open and the timer restarts.

The key design question for a circuit breaker is: what kind of error counts as a "failure"? Only errors that indicate the downstream is unhealthy should count, such as timeouts, connection failures and 5xx. A declined card, insufficient balance or bad arguments are all business outcomes from a downstream working normally; if you count them, a wave of declined card payments could trip the breaker and block all the normal payments too.

Step 6: Implementing a circuit breaker in Go

The community has mature packages (such as sony/gobreaker), and in practice you can use one directly. But writing one yourself is how you really understand its behaviour, and the core is only a few dozen lines. The implementation below uses a consecutive failure count, lets only one probe request through at a time in Half-open, and lets the caller decide which errors count as failures:

package main

import (
	"context"
	"errors"
	"fmt"
	"sync"
	"time"
)

type State int

const (
	Closed State = iota
	Open
	HalfOpen
)

func (s State) String() string {
	return [...]string{"closed", "open", "half-open"}[s]
}

var ErrOpen = errors.New("circuit breaker is open")

type Breaker struct {
	mu          sync.Mutex
	state       State
	failures    int           // consecutive failures in the closed state
	threshold   int           // trip after this many consecutive failures
	openTimeout time.Duration // how long to stay open before letting a probe through
	openedAt    time.Time
	probing     bool                 // whether a probe request is already running in half-open
	isFailure   func(err error) bool // which errors mean "the downstream is unhealthy"
}

func NewBreaker(threshold int, openTimeout time.Duration, isFailure func(error) bool) *Breaker {
	return &Breaker{threshold: threshold, openTimeout: openTimeout, isFailure: isFailure}
}

func (b *Breaker) State() State {
	b.mu.Lock()
	defer b.mu.Unlock()
	return b.state
}

// allow decides whether this call may go through
func (b *Breaker) allow() error {
	b.mu.Lock()
	defer b.mu.Unlock()
	switch b.state {
	case Open:
		if time.Since(b.openedAt) < b.openTimeout {
			return ErrOpen
		}
		b.state = HalfOpen
		b.probing = false
		fallthrough
	case HalfOpen:
		if b.probing {
			return ErrOpen // only one probe request at a time
		}
		b.probing = true
	}
	return nil
}

func (b *Breaker) record(err error) {
	b.mu.Lock()
	defer b.mu.Unlock()
	failed := err != nil && b.isFailure(err)
	switch b.state {
	case HalfOpen:
		b.probing = false
		if failed {
			b.trip()
		} else {
			b.state, b.failures = Closed, 0
		}
	case Closed:
		if !failed {
			b.failures = 0
			return
		}
		b.failures++
		if b.failures >= b.threshold {
			b.trip()
		}
	case Open:
		// Slow requests let through before tripping don't affect the state
	}
}

func (b *Breaker) trip() {
	b.state, b.openedAt, b.failures = Open, time.Now(), 0
}

func (b *Breaker) Call(ctx context.Context, fn func(context.Context) error) error {
	if err := b.allow(); err != nil {
		return err
	}
	err := fn(ctx)
	b.record(err)
	return err
}

var ErrPaymentDown = errors.New("payment: 503")
var ErrCardDeclined = errors.New("payment: card declined")

func main() {
	br := NewBreaker(3, 100*time.Millisecond, func(err error) bool {
		// A declined card is a business outcome; it doesn't mean the payment service is broken
		return !errors.Is(err, ErrCardDeclined)
	})

	healthy := false
	charge := func(ctx context.Context) error {
		if !healthy {
			return ErrPaymentDown
		}
		return nil
	}
	ctx := context.Background()

	for i := 1; i <= 5; i++ {
		err := br.Call(ctx, charge)
		fmt.Printf("call %d: err=%v, state=%v\n", i, err, br.State())
	}

	fmt.Println("… waiting out openTimeout; the payment service recovers …")
	time.Sleep(120 * time.Millisecond)
	healthy = true

	err := br.Call(ctx, charge)
	fmt.Printf("probe call: err=%v, state=%v\n", err, br.State())

	err = br.Call(ctx, func(context.Context) error { return ErrCardDeclined })
	fmt.Printf("card declined: err=%v, state=%v (not counted as a failure)\n", err, br.State())
}

The output is as follows, and the state transitions are clear to see:

call 1: err=payment: 503, state=closed
call 2: err=payment: 503, state=closed
call 3: err=payment: 503, state=open
call 4: err=circuit breaker is open, state=open
call 5: err=circuit breaker is open, state=open
… waiting out openTimeout; the payment service recovers …
probe call: err=<nil>, state=closed
card declined: err=payment: card declined, state=closed (not counted as a failure)

A few implementation details are worth explaining. First, allow and record take the lock separately, and the call itself doesn't hold the lock; otherwise one slow call would make every goroutine queue for the lock. Second, the transition from Open to Half-open in allow uses fallthrough, so "the request that arrives once the time is up" becomes the probe directly. Third, record does nothing in the Open state: the results of slow requests that were let through before the breaker tripped should not change the state any more. Fourth, if a probe request panics and never reaches record, probing stays true and the breaker is stuck in Half-open forever; a production implementation should use defer to make sure the result is always recorded, or put a timeout on the probe.

Where should circuit breakers go? A caller has one circuit breaker for each downstream (or even for each key operation of each downstream), rather than one shared by the whole service. A broken payment service shouldn't also block calls to the inventory service. As for the order of retries and circuit breaking, it is usually retries outside, circuit breaker inside: every attempt goes through the breaker, and when the breaker is open, the retry loop should give up as soon as it receives ErrOpen; ErrOpen is not a retryable error.

Step 7: Bulkhead isolation and fallbacks

The last tool comes from shipbuilding: a ship's hull is divided into compartments by bulkheads, so one flooded compartment doesn't sink the whole ship. A bulkhead here means letting different downstreams use different resource pools, so when one downstream slows down it exhausts only its own compartment's resources.

In Go, the resource most often exhausted is "the number of calls in flight at once". Goroutines are cheap, but every stuck call holds a connection, a chunk of memory and a request's lifetime. If the order service's calls to the payment service are unbounded, then as soon as the payment service slows down, all of the order service's handler goroutines may get stuck waiting for payment, and even requests that don't need payment at all, such as looking up an order, are affected. Using a buffered channel as a semaphore lets you give each downstream its own independent concurrency limit:

// Snippet: one bulkhead per downstream; when full, reject immediately instead of queueing
type Bulkhead struct{ slots chan struct{} }

func NewBulkhead(n int) *Bulkhead { return &Bulkhead{slots: make(chan struct{}, n)} }

var ErrBulkheadFull = errors.New("bulkhead full")

func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error {
	select {
	case b.slots <- struct{}{}:
		defer func() { <-b.slots }()
		return fn(ctx)
	default:
		return ErrBulkheadFull // or select on ctx.Done() to wait a short while
	}
}

// When wiring things up in cmd/order/main.go, give payment and inventory one each
// payBH := NewBulkhead(50)
// invBH := NewBulkhead(100)

golang.org/x/sync/semaphore provides a weighted semaphore based on the same idea. At the HTTP level there is another often-overlooked bulkhead: http.Transport's MaxConnsPerHost. Use a different http.Client (each with its own Transport) for each downstream and the connection pools are naturally separated, which echoes the connection pool settings discussed in “database/sql, Transactions and Connection Pools in Go”.

When the circuit breaker is open or the bulkhead is full, the caller needs a fallback strategy:

SituationPossible fallback
Recommendation service failsReturn a cached list of popular products
Inventory lookup failsThe product page shows "checking stock", with a strict check at order time
Payment service failsThe order stays pending and the user is prompted to pay later
Notification service failsPut it on a message queue and send it later

The principle for fallbacks is: be conservative with core transactions and relaxed with peripheral features. A slightly stale recommendation list is fine, but a failed payment must never pretend to have succeeded. The fallback for payment is "tell the user clearly that it hasn't been paid yet", not "treat it as paid for now".

Common pitfalls

  • Setting only a client timeout without passing the deadline: each hop uses its own fixed timeout, and after upstream gives up, downstream keeps working, producing orphaned work and inconsistent side effects.
  • Retrying non-idempotent operations: blindly retrying POSTs such as charging or creating an order causes duplicate charges when a response is lost. Get idempotency keys in place first, then talk about retries.
  • Retrying at every layer: three layers retrying three times each is a 27× amplification. Choose one layer to retry, and have the other layers pass errors upwards.
  • Backoff without jitter: every client retries at the same moment, and the downstream is knocked over by a synchronised wave of retries as soon as it recovers.
  • Counting business errors towards circuit breaking: a declined card or bad arguments don't mean the downstream is broken; counting them makes the breaker trip by mistake and block normal requests.
  • One circuit breaker shared by the whole service: one failing downstream blocks every downstream. Have one per downstream and, where necessary, one per operation.
  • Too many probes in Half-open: the just-recovered service is knocked over by probe traffic, and the breaker oscillates between Open and Half-open.
  • A timeout shorter than the downstream's timeout: the caller has already given up but the downstream hasn't timed out yet and keeps holding resources; outer deadlines should be longer than inner ones.
  • Using time.Sleep in a retry loop: it can't be interrupted when upstream cancels; use select to wait on both the timer and ctx.Done().
  • Ignoring Retry-After: retrying by your own formula while rate-limited is ignoring the downstream's explicit request.

Key takeaways

  • Slowness is more dangerous than failure; every call across the network needs a timeout, and the whole call chain shares one deadline budget, which only ever gets shorter as it is passed down.
  • Retry only operations that are both "transient" and "idempotent"; classify errors in the client adapter layer and check them with errors.Is.
  • Retries use exponential backoff plus full jitter; waits must be cancellable by the context, and give up early when the budget isn't enough.
  • Retry at only one layer, and use a retry budget to limit the proportion of retries, avoiding amplification during outages.
  • A circuit breaker has three states, Closed, Open and Half-open; only errors that indicate an unhealthy downstream count as failures; have one per downstream.
  • Bulkheads give different downstreams different resource pools; combine them with fallbacks, conservative for core transactions and relaxed for peripheral features.
  • The parameters of all these mechanisms need metrics to back them up; failure rates, retry counts and circuit breaker state changes should all be monitored.

Next

Retries let transient failures recover, but they also bring a new risk: if a charge is retried when the response is lost, the user may be charged twice. The next article, “Idempotency and Distributed Locks”, uses idempotency keys, unique constraints and fencing tokens so that "doing it twice" has the same effect as "doing it once".