Retries: the loop that hammers a server it can never satisfy
A retry is a bet that the same request will behave differently next time. Most failures are not that kind of failure.
Train thisThe problem
An integration was flaky, so somebody wrapped it in a retry loop. The loop is three lines long, it fixed the flakiness, and it has been in the codebase ever since.
Today a caller submits a payload the server considers malformed, and the server rejects it the same way every single time.
type Attempt = () => Promise<string>;
const delay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
async function retry(attempt: Attempt): Promise<string> {
for (;;) {
try {
return await attempt();
} catch {
await delay(1000);
}
}
}PREDICT FIRST
The request fails validation — the server will reject it identically forever. What does retry do?
Commit to an answer before you read on. Being wrong here is the part that sticks.
Why it behaves that way
Three things are missing, and each one is a separate decision.
First, classification. A 400 means the request is wrong; retrying it cannot help. A 503, a timeout or a dropped connection means the request might be fine and the moment was bad. Only the second group is retryable, and the catch block is where that judgment belongs.
Second, a bound. Retrying forever converts a transient outage into an outage that never ends, because the retries themselves become the load. Bound the attempts, and bound the total time — a caller waiting on a 30-second deadline does not benefit from attempt seven at second 45.
Third, spacing. A fixed delay synchronises every client that failed at the same moment, so the recovering server gets a wall of traffic exactly one second after it fell over. Exponential backoff spreads attempts out; jitter breaks the synchronisation. When the server tells you when to come back with Retry-After, that beats any formula you invent.
One consequence worth stating plainly: retrying a request that already reached the server means it may execute twice. A safe retry policy assumes the operation is idempotent, and unsafe operations need an idempotency key before they can be retried at all.
class RetryableError extends Error {
retryAfterMs?: number;
constructor(message: string, retryAfterMs?: number) {
super(message);
this.retryAfterMs = retryAfterMs;
}
}
interface RetryPolicy { maxAttempts: number; baseMs: number; maxDelayMs: number; deadline: number }
async function retry(attempt: Attempt, policy: RetryPolicy): Promise<string> {
for (let tries = 0; ; tries++) {
try {
return await attempt();
} catch (error) {
const retryable = error instanceof RetryableError;
const lastAttempt = tries + 1 >= policy.maxAttempts;
if (!retryable || lastAttempt) throw error;
const backoff = Math.min(policy.baseMs * 2 ** tries, policy.maxDelayMs);
const wait = error.retryAfterMs ?? backoff * (0.5 + Math.random() / 2);
if (Date.now() + wait > policy.deadline) throw error;
await delay(wait);
}
}
}What the change buys you
| Behaviour | Before | After |
|---|---|---|
| A validation failure | Retried forever | Thrown immediately to the caller |
| A transient 503 | Retried forever at 1s intervals | Retried up to maxAttempts with growing, jittered delays |
| A thousand clients failing together | A synchronised request every second | Attempts spread across each backoff window |
| A caller with a 30s deadline | Waits indefinitely | Fails fast once the next delay would cross the deadline |
| Server sends Retry-After | Ignored | Honoured in place of the computed backoff |
Variations worth trying
- Add a circuit breaker in front of the policy so a sustained outage stops generating attempts at all, rather than generating bounded ones from every caller.
- Make the classification data-driven: a set of retryable HTTP statuses plus a predicate for transport errors, so the rule can be reviewed in one place.
- Thread an
AbortSignalthrough both the attempt and the delay, so a cancelled caller does not leave a pending timer behind. - Test it with a fake clock. A retry policy tested with real timers is a slow test that only checks the happy path.
TRAIN IT
A bounded retry policy
Build the policy from a specification: classify errors, cap attempts, respect a deadline, and prove the behaviour with a controlled clock.
Open the workout