Beyond retries for transient failures, a production LLM integration needs to handle a broader range of error and edge-case scenarios gracefully — malformed output, content policy rejections, timeouts, and partial failures.
Categories of Things That Can Go Wrong
| Category | Example | Handling Approach |
|---|---|---|
| Network/infrastructure errors | Timeout, connection failure | Retry with backoff (see Retries) |
| Rate limiting | 429 response | Backoff, queuing, or throttling (see Rate Limits) |
| Content policy rejection | Provider declines to generate for a request | Surface a clear, appropriate message — don't silently fail or retry identically |
| Malformed/unparseable output | Expected JSON, got invalid JSON or truncated output | Validate before use; retry with adjusted parameters or a repair step |
| Empty or unexpectedly short response | Model returns near-empty output for a substantive request | Treat as a failure condition, not a valid (if underwhelming) response |
A More Complete Error-Handling Pattern
def get_llm_response(prompt):
try:
response = llm_client.generate(prompt=prompt, max_tokens=500)
except RateLimitError:
return retry_with_backoff(prompt)
except AuthenticationError:
log_critical("API auth failure — check credentials")
raise # not recoverable at request time
except ContentPolicyError as e:
log_warning(f"Content policy rejection: {e}")
return fallback_response("I can't help with that request.")
if response.finish_reason == "length":
log_warning("Response truncated — consider raising max_tokens")
if not response.text or len(response.text.strip()) == 0:
return fallback_response("Something went wrong generating a response.")
return response.textAlways Have a Fallback
Every LLM-dependent feature should define what happens when the LLM call fails entirely, after retries are exhausted — a static fallback message, a degraded non-AI code path, or a clear error state — rather than letting the failure propagate as an unhandled exception to the end user. See LLM Fallbacks.
Common Mistakes
- Only handling the "happy path" and letting any API error crash or hang the calling code
- Treating a successfully-returned but empty/truncated/malformed response as valid simply because no exception was thrown
- Not logging enough detail about failures to actually diagnose recurring issues in production
Interview Relevance
"What can go wrong with an LLM API call beyond a simple network error?" — content policy rejections, malformed output, and truncated responses are exactly the kinds of "successful but not actually usable" failures that distinguish a thorough answer.
Practice Question
Design the error-handling logic for a feature that generates a JSON product description — cover network failure, malformed JSON, and an empty response.