Skip to main content
When a request fails transiently, the question is not “should I retry?” — it is “how do I retry without making things worse?”. A naive retry loop turns a recoverable blip into a thundering herd. This page is the canonical retry recipe. For which errors are retryable, see Handling errors §Retry semantics.

The recipe

Three rules:
  1. Honor Retry-After. If the server tells you when to retry, do exactly that.
  2. Otherwise, exponential backoff with full jitter. Random in [0, min(cap, base × 2^attempt)).
  3. Bound attempts. Three is a reasonable default. After three failures, surface the error.

The formula

The canonical “decorrelated jitter” / “full jitter” formula (AWS guidance):
Sequence with base = 500ms, cap = 30s: The randomization is essential — without it, every retrying client hits the server at the same moments after a brief outage, prolonging recovery.

Code

Which errors are retryable

Per Handling errors: INTERNAL is a deliberate exception — it indicates a server-side defect the platform understands but cannot resolve for you. Retrying compounds the problem without changing the outcome. Capture the ERROR_RECORDED id and escalate instead.

Per-error-class retry recipes

Rate limit (RESOURCE_EXHAUSTED)

Transient infrastructure (*_UNAVAILABLE)

Replay (REPLAY_DETECTED)

This error happens when your request id generator collides or persists state across attempts. The retry should succeed; if it fails again, fix the request-id generator.

Auth (SIGNATURE_INVALID, TIMESTAMP_SKEW_EXCEEDED)

Do not retry. Fix the root cause (low-S, body bytes, clock drift). Retrying without fixing returns the same error.

Validation (INVALID_ARGUMENT)

Do not retry. The request shape is wrong. Surface the per-field errors from details[] to the user and let them resubmit.

Idempotency conflict (IDEMPOTENCY_KEY_IN_USE_WITH_DIFFERENT_PARAMS)

Do not retry with the same key. The key is permanently bound to the first body that used it. Either:
  • Pick a new key and retry with the new body, OR
  • Reconcile: fetch the original order via the local persistence layer (you stored the original idempotencyKey there, right?) and decide whether to keep it or create a separate operation.

Things that go wrong

Retry without backoff

A pile-up of clients all retrying with no delay turns a brief outage into a long one. Always backoff.

Retry without bounds

If the failure is persistent (not transient), an unbounded loop never surfaces it. Cap attempts.

Retry without keeping idempotencyKey

If the first attempt succeeded but the response was lost, the second attempt with a different key creates a duplicate. The idempotencyKey must be persisted before the call and reused on every retry.

Retry on RBAC_DENY

If the credential lacks permission now, it lacks permission in five seconds. Surface the failure; request the role binding from your account team.

Beyond retries — circuit breaker

For client systems with bursty traffic, consider wrapping the BlooBank client in a circuit breaker:
  • After N consecutive failures, open the circuit — fail fast for a cooldown window without even calling.
  • Periodically allow one probe through (half-open).
  • On success, close the circuit and resume normal traffic.
Libraries to consider: opossum (Node), pybreaker (Python), hystrix-go (Go), Resilience4j (Java). A circuit breaker is not a substitute for retries; the two work together. Retries handle individual blips; the breaker handles sustained outages.

Next

Handling errors

The branching pattern in code.

Idempotency

Make every retry safe.