Zero-shot learning pushes even further than few-shot: perform a task or recognize a class with zero task-specific examples at all — relying entirely on general pretrained knowledge and, typically, a natural-language description of what's being asked.
Zero-Shot Text Classification
prompt = "Classify the sentiment of this review as positive or negative: 'The service was disappointing.'"
# No examples provided at all -- the model relies purely on its pretrained
# understanding of language and the concepts "positive" and "negative"Unlike the few-shot prompt from Few-Shot Learning, there are no example (review, sentiment) pairs shown — the model must infer what "positive" and "negative" sentiment mean and correctly apply that understanding, purely from its pretraining.
Zero-Shot Image Classification via CLIP
A particularly elegant zero-shot mechanism: CLIP (covered fully in Vision-Language Models) jointly embeds images and text into a shared space. To classify an image into categories it was never explicitly trained on, simply compute the image's embedding, compute text embeddings for candidate category descriptions ("a photo of a cat," "a photo of a dog," ...), and pick whichever text embedding is most similar (via cosine similarity, see Dot Product) to the image embedding.
Code — CLIP Zero-Shot Classification
import torch
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a bird"]
image = load_some_image() # e.g. a picture of an animal never labeled during CLIP's training
inputs = processor(text=candidate_labels, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1) # similarity-based "probability" per candidate label
print(dict(zip(candidate_labels, probs[0].tolist())))
# CLIP was never explicitly trained to classify "cat vs dog vs bird" --
# it works purely by comparing embeddings in its shared image-text spaceWhy Zero-Shot Works — It's Not Actually Magic
Zero-shot capability isn't learned from nothing — it emerges from a model's pretraining having already encountered a vast, implicit range of related concepts and language patterns (or, for CLIP, image-text pairs across a huge variety of concepts). The model isn't inventing understanding on the spot; it's applying general knowledge acquired during large-scale pretraining to a specific, novel framing it hasn't seen phrased exactly that way before.
Common Mistakes
- Expecting reliable zero-shot performance on tasks or concepts genuinely absent from a model's pretraining data — zero-shot capability draws on existing pretrained knowledge; it can't produce accurate results for concepts the model has no meaningful prior exposure to.
- Confusing zero-shot classification's "probability" outputs (from CLIP's cosine similarity) with a true, calibrated probability distribution the way a trained classifier's softmax output represents — it's a useful ranking signal, but its absolute values shouldn't be over-interpreted as precise confidence.
Interview Relevance
Q: "How can CLIP classify images into categories it was never explicitly trained to recognize?" CLIP is trained to embed images and their corresponding text descriptions into a shared space where matching pairs end up close together. For a new classification task, you don't retrain anything — you compute the image's embedding and the embeddings of candidate text descriptions, then pick whichever text description's embedding is closest, exploiting the general image-text alignment learned during pretraining rather than any category-specific training.
Practice Question
Why might CLIP perform poorly at zero-shot classification for a highly specialized, technical image category (e.g. a specific rare medical condition) that's unlikely to have been well-represented in its training data?
