Fine-tuning quality is bounded by dataset quality — a small, carefully curated, consistent dataset generally outperforms a larger, noisier one. Building a good fine-tuning dataset is often the majority of the real effort in a fine-tuning project.
What Makes a Good Fine-Tuning Dataset
| Quality Factor | Why It Matters |
|---|---|
| Consistency | Examples demonstrating conflicting styles/formats teach the model an unclear, muddled pattern |
| Representativeness | Should cover the real range of inputs the model will actually encounter, including edge cases |
| Correctness | Every "ideal output" example should genuinely be the behavior you want — errors in training data directly teach errors |
| Sufficient volume | Enough examples for the model to generalize the pattern reliably, not just memorize the specific training examples |
Sources of Training Data
- Real historical data — past support tickets and their genuinely good resolutions, past well-written content — often the highest-quality source, since it reflects real use cases
- Manually authored examples — written specifically to demonstrate desired behavior, useful for covering gaps or edge cases historical data doesn't include
- Synthetic data — generated by another capable model, then reviewed/filtered by humans — can help scale up a dataset, but requires careful quality control since it inherits the generating model's own limitations
Formatting for Fine-Tuning
# Conceptual format — exact structure varies by provider/framework
{"messages": [
{"role": "system", "content": "You are a support assistant."},
{"role": "user", "content": "My package arrived damaged."},
{"role": "assistant", "content": "I'm sorry to hear that! ..."}
]}
# One such example per line/record, typically hundreds to
# thousands of examples totalSplitting for Honest Evaluation
all_examples = load_examples()
train_set, validation_set = split(all_examples, ratio=0.9)
# Fine-tune only on train_set.
# Evaluate on validation_set — examples the model never
# trained on — for an honest measure of generalization,
# not just memorization of training examples.Practical Use Case
A team fine-tuning a support-response model should deliberately curate for both quality (accurate, well-written responses) and diversity (covering the real range of ticket types, tones, and edge cases their support team actually handles) — not just gather the largest possible volume of historical data indiscriminately.
Common Mistakes
- Prioritizing dataset size over quality/consistency, assuming "more data" automatically produces a better result
- Not holding out a genuine validation set, making post-training evaluation unreliable (essentially grading on the training data)
- Including examples with errors or inconsistent style, directly teaching the model those same flaws
Interview Relevance
"You have 500 high-quality examples and 5,000 mediocre, inconsistent ones. Which would you fine-tune on?" — generally the smaller, higher-quality, more consistent set; dataset quality is usually the dominant factor in fine-tuning outcomes, not raw volume.
Practice Question
Design a process for building a 500-example fine-tuning dataset from a company's historical support tickets, including a quality-filtering step.