Alignment is the training stage that shapes a model toward being helpful, honest, and harmless — reducing harmful, biased, or unsafe outputs, and improving how well responses match what humans actually prefer, beyond just "technically follows the instruction."
Why Instruction Tuning Alone Isn't Enough
An instruction-tuned model will follow instructions — including ones it probably shouldn't, or answer in a technically-correct-but-unhelpful way. Alignment is specifically aimed at closing that gap:
Instruction: "How do I pick a lock?"
Un-aligned tendency: might respond with detailed, unqualified instructions
regardless of context or intent.
Aligned tendency: responds with appropriate context-awareness — e.g.
distinguishing a locksmith-training context from a request with no
stated legitimate purpose — and may decline or add safety context
depending on the situation and system-level policy.Common Alignment Techniques (Conceptual)
| Technique | Core Idea |
|---|---|
| RLHF (Reinforcement Learning from Human Feedback) | Humans rank multiple model responses; a reward model learns those preferences; the LLM is further trained to produce outputs the reward model scores highly |
| DPO (Direct Preference Optimization) and similar | Newer approaches that use preference data more directly, without training a separate reward model |
The specific technique used by any given model provider is proprietary and evolves — treat the exact method as an implementation detail; the concept (train toward human-preferred, safer behavior) is what's stable and worth understanding.
What Alignment Trades Off
Alignment isn't free — overly cautious alignment can make a model unhelpfully evasive on legitimate requests ("over-refusal"), while under-alignment risks harmful or unsafe outputs. Providers continuously tune this balance, and it's a genuine, unresolved engineering and policy challenge, not a solved problem.
Practical Use Case
When building an application, alignment is why a general-purpose model already refuses obviously harmful requests by default — but application-level guardrails (see Guardrails) are still necessary for domain-specific safety requirements the base model's alignment wasn't specifically trained for.
Common Mistakes
- Assuming alignment makes a model's outputs completely safe in every context — it reduces risk, it doesn't eliminate it; application-level safeguards are still necessary
- Treating "aligned" as a synonym for "aligned with my specific use case's requirements" — alignment targets general helpfulness/harmlessness, not your particular domain's policies
Interview Relevance
"What is RLHF, conceptually?" is a common question — the key idea (human preference rankings → reward model → further training toward preferred outputs) is the expected answer, without necessarily needing the full mathematical detail.
Practice Question
Explain, conceptually, why a purely instruction-tuned model (no alignment stage) might be riskier to deploy directly to end users than an aligned one.