Prompting should almost always be the first thing you try — it's free (no training cost), instant to iterate on, and works with any model. Fine-tuning is a bigger investment, worth making only once prompting has demonstrably hit a real, measured limit.
A Practical Escalation Path
1. Try prompting (zero-shot, then few-shot if needed).
2. If output format/consistency is still unreliable, try more
rigorous prompt engineering — structure, examples, structured
output modes.
3. Measure the actual failure rate with a real evaluation set —
is it genuinely a problem, or "felt" inconsistent from a
handful of manual tests?
4. Only if prompting demonstrably plateaus below what's needed
→ consider fine-tuning.Skipping straight to fine-tuning without exhausting prompting-based approaches first is a common, costly overcorrection.
Side-by-Side Comparison
| Prompting | Fine-Tuning | |
|---|---|---|
| Setup cost | None — write and test immediately | Requires a curated dataset and a training run |
| Iteration speed | Instant — change the prompt, test again | Slower — retraining takes real time per iteration |
| Per-request cost | Can be higher if using many few-shot examples repeatedly | Can be lower per-request once trained — behavior is baked in, no repeated examples needed |
| Flexibility | Easy to adjust behavior instantly | Requires retraining to meaningfully change behavior |
| Best for | Most tasks, especially ones that change or evolve | Stable, well-defined tasks with a real, measured prompting ceiling |
A Concrete Example of "Prompting Hit Its Limit"
Task: classify support tickets into 12 fine-grained categories,
matching a specific internal taxonomy with subtle distinctions.
Prompting result (measured against an eval set): ~78% accuracy,
despite extensive prompt iteration and few-shot examples.
→ A real, measured plateau — a legitimate case to evaluate
fine-tuning on a labeled dataset of correctly-categorized
historical tickets.Practical Use Case
Most production LLM features never need fine-tuning at all — well-designed prompting handles the large majority of real use cases. Fine-tuning becomes worth the investment specifically when you have (a) a well-defined, stable task, (b) a real, measured accuracy/consistency ceiling from prompting, and (c) enough quality labeled data to fine-tune on.
Common Mistakes
- Fine-tuning based on a vague sense that "prompting feels unreliable" rather than a measured evaluation showing a genuine ceiling
- Underinvesting in prompt engineering before concluding fine-tuning is necessary
- Fine-tuning for a task that changes frequently — the model needs re-training every time requirements shift, unlike a prompt that can be edited instantly
Interview Relevance
"How would you decide whether a task's performance problem should be solved with better prompting or with fine-tuning?" — a measured evaluation showing prompting has genuinely plateaued, not just a subjective impression, is the expected criterion.
Practice Question
A team wants to fine-tune because "the outputs just feel inconsistent." What would you ask them to do first, before approving a fine-tuning project?