Prompt optimization is the iterative process of improving a prompt's wording, structure, or examples based on measured output quality — not guessing, but testing changes against real evaluation criteria.
A Practical Optimization Loop
1. Define what "good output" means for this task (see
Prompt Evaluation) — specific, checkable criteria.
2. Run the current prompt against a representative set of test inputs.
3. Identify failure patterns (not just "it's sometimes wrong" —
WHICH kind of input causes WHICH kind of failure).
4. Make one targeted change addressing that specific failure.
5. Re-run against the same test set — did it fix the issue
without breaking previously-working cases?
6. Repeat.Common Optimization Moves, With Before/After
Problem: inconsistent output format
Before: "List the key points"
After: "List exactly 3 key points as a numbered list, each
under 15 words"
Problem: model ignores provided context, uses own knowledge
Before: "Here's some info: [context]. Answer: [question]"
After: "Using ONLY the information below (not your own
knowledge), answer the question. If not covered,
say so explicitly.
Info: [context]
Question: [question]"
Problem: responses too verbose
Before: "Explain this concept"
After: "Explain this concept in exactly 2 sentences, no preamble"Change One Thing at a Time
Optimizing multiple aspects of a prompt simultaneously makes it impossible to know which change actually caused an improvement (or regression) — treat prompt changes like any other controlled experiment, isolating variables where practical.
Practical Use Case
Any production prompt that's been running for a while is a candidate for optimization once you have real failure examples to work from — production usage surfaces edge cases and failure patterns that initial testing rarely catches.
Common Mistakes
- Optimizing based on a single anecdotal bad example instead of a representative test set — a fix for one case can silently break others
- Changing multiple things in a prompt at once, losing the ability to attribute an improvement (or regression) to a specific change
- Never revisiting a prompt after initial launch, missing failure patterns that only show up with real production traffic
Interview Relevance
"How would you systematically improve a prompt that's underperforming in production?" — a strong answer describes gathering real failure examples, identifying specific patterns, making targeted changes, and re-testing against a consistent evaluation set — not ad hoc tweaking.
Practice Question
A summarization prompt sometimes includes information not present in the source text. Propose one specific prompt change to address this, and explain how you'd verify it worked.