Prompt tuning is prefix tuning's simpler cousin: instead of adding trainable vectors at every layer, add them only at the input embedding layer, once — the leanest, simplest PEFT method covered in this category.
The Core Idea
\(\mathbf{p}_1,\ldots,\mathbf{p}_L\) are trainable "soft prompt" embedding vectors, prepended directly to the sequence of real token embeddings \(\mathbf{e}_1,\ldots,\mathbf{e}_n\). These soft prompt vectors then flow through the entire frozen model exactly like any other token embedding would — no special mechanism is needed elsewhere in the model.
Prefix Tuning vs Prompt Tuning — Direct Comparison
| Prefix Tuning | Prompt Tuning | |
|---|---|---|
| Where trainable vectors are added | Key/value pairs at every attention layer | Only at the input embedding layer, once |
| Number of trainable parameter sets | One set per layer | Just one set, total |
| Trainable parameter count | Higher (scales with number of layers) | Lower (independent of model depth) |
| Typical performance | Often slightly stronger, especially on smaller base models | Simpler, and increasingly competitive as base model scale grows very large |
Why Prompt Tuning Becomes More Competitive at Larger Scale
A key empirical finding: as the underlying pretrained model gets larger, the performance gap between prompt tuning and full fine-tuning (or even prefix tuning) tends to shrink substantially — a very large, capable base model apparently needs comparatively little steering to adapt well, making prompt tuning's minimal parameter footprint an increasingly attractive tradeoff specifically for the largest modern LLMs.
Code
from peft import PromptTuningConfig, get_peft_model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2")
prompt_config = PromptTuningConfig(
task_type="CAUSAL_LM",
num_virtual_tokens=20 # 20 trainable soft-prompt vectors, added ONCE at the input
)
peft_model = get_peft_model(model, prompt_config)
peft_model.print_trainable_parameters()
# trainable params are even fewer than prefix tuning's, since there's only ONE set, not one per layerCommon Mistakes
- Assuming prompt tuning is always weaker than prefix tuning regardless of model scale — the performance gap narrows significantly, and sometimes closes entirely, on very large base models.
- Confusing prompt tuning's trainable embedding vectors with discrete prompt engineering (choosing better natural-language wording) — prompt tuning's vectors are continuous, learned parameters with no corresponding actual words, a fundamentally different technique from writing better text prompts.
Interview Relevance
Q: "Why does prompt tuning have far fewer trainable parameters than prefix tuning, even though both prepend trainable vectors?" Prefix tuning adds a separate set of trainable key/value vectors at every attention layer, so its parameter count scales with model depth. Prompt tuning adds trainable vectors only once, at the input embedding layer — those same vectors then pass through every subsequent frozen layer naturally, requiring only one set of parameters regardless of how many layers the model has.
Practice Question
Why might prompt tuning's advantage narrow specifically as the base model gets larger, rather than staying constant across model sizes?
