LLM API calls fail sometimes — rate limits, transient network issues, temporary provider-side errors. A deliberate retry strategy is what separates a resilient application from one that surfaces avoidable errors to users.
Exponential Backoff — The Standard Pattern
import time
import random
def call_with_retry(prompt, max_retries=4):
for attempt in range(max_retries):
try:
return llm_client.generate(prompt=prompt)
except TransientError as e:
if attempt == max_retries - 1:
raise # out of retries — let the caller handle failure
wait = (2 ** attempt) + random.uniform(0, 1) # backoff + jitter
time.sleep(wait)
Each retry waits longer than the last (exponential backoff), and adding a small random jitter prevents many clients from retrying at exactly the same moment and re-triggering the same rate limit together.
Not Every Error Should Be Retried
| Error Type | Retry? |
|---|---|
| Rate limit (429) | Yes — with backoff, ideally respecting a retry-after hint |
| Transient server error (5xx) | Yes — often resolves on retry |
| Invalid request (400) — e.g. malformed input | No — retrying identical bad input produces the same error every time; fix the request instead |
| Authentication error (401/403) | No — a retry won't fix invalid credentials |
Blindly retrying every error type wastes time and can mask real bugs in your request construction.
Setting a Sensible Retry Limit
Unlimited retries risk making a failing request hang indefinitely and can compound load during an outage. A bounded number of retries (commonly 3-5) with a final clear failure/fallback path is the standard, sensible approach.
Practical Use Case
Any production feature calling an LLM API should have retry logic for transient failures — without it, a brief provider hiccup becomes a visible, avoidable user-facing error instead of a handled, invisible retry.
Common Mistakes
- Retrying non-retryable errors (like malformed requests) repeatedly, wasting time without ever succeeding
- No jitter in backoff timing, causing many clients to retry in synchronized bursts
- No maximum retry limit, risking requests that hang far longer than acceptable for the use case
Interview Relevance
"Should you retry every failed LLM API call the same way?" — no; a good answer distinguishes retryable (transient, rate-limit) from non-retryable (invalid request, auth) errors, and describes exponential backoff with jitter for the former.
Practice Question
Write pseudocode for a retry function that uses exponential backoff, respects a maximum of 4 attempts, and does NOT retry on a 400 (bad request) error.