The recipe
Three rules:- Honor
Retry-After. If the server tells you when to retry, do exactly that. - Otherwise, exponential backoff with full jitter. Random in
[0, min(cap, base × 2^attempt)). - Bound attempts. Three is a reasonable default. After three failures, surface the error.
The formula
The canonical “decorrelated jitter” / “full jitter” formula (AWS guidance):
Sequence with
base = 500ms, cap = 30s:
The randomization is essential — without it, every retrying client hits the server at the same moments after a brief outage, prolonging recovery.
Code
Which errors are retryable
Per Handling errors:INTERNAL is a deliberate exception — it indicates a server-side defect the platform understands but cannot resolve for you. Retrying compounds the problem without changing the outcome. Capture the ERROR_RECORDED id and escalate instead.
Per-error-class retry recipes
Rate limit (RESOURCE_EXHAUSTED)
Transient infrastructure (*_UNAVAILABLE)
Replay (REPLAY_DETECTED)
Auth (SIGNATURE_INVALID, TIMESTAMP_SKEW_EXCEEDED)
Do not retry. Fix the root cause (low-S, body bytes, clock drift). Retrying without fixing returns the same error.
Validation (INVALID_ARGUMENT)
Do not retry. The request shape is wrong. Surface the per-field errors from details[] to the user and let them resubmit.
Idempotency conflict (IDEMPOTENCY_KEY_IN_USE_WITH_DIFFERENT_PARAMS)
Do not retry with the same key. The key is permanently bound to the first body that used it. Either:
- Pick a new key and retry with the new body, OR
- Reconcile: fetch the original order via the local persistence layer (you stored the original
idempotencyKeythere, right?) and decide whether to keep it or create a separate operation.
Things that go wrong
Retry without backoff
Retry without bounds
Retry without keeping idempotencyKey
idempotencyKey must be persisted before the call and reused on every retry.
Retry on RBAC_DENY
Beyond retries — circuit breaker
For client systems with bursty traffic, consider wrapping the BlooBank client in a circuit breaker:- After N consecutive failures, open the circuit — fail fast for a cooldown window without even calling.
- Periodically allow one probe through (half-open).
- On success, close the circuit and resume normal traffic.
opossum (Node), pybreaker (Python), hystrix-go (Go), Resilience4j (Java).
A circuit breaker is not a substitute for retries; the two work together. Retries handle individual blips; the breaker handles sustained outages.
Next
Handling errors
The branching pattern in code.
Idempotency
Make every retry safe.