Contrastive Loss

Every loss function covered so far compares a prediction to a fixed label. Contrastive Loss does something different: it trains a network to produce embeddings where similar inputs end up close together in embedding space, and dissimilar inputs end up far apart.

Formula

\(D = \|\mathbf{e}_1-\mathbf{e}_2\|\) is the Euclidean distance between the embeddings of a pair of inputs (see Vector Norms). \(Y=0\) if the pair is a similar ("positive") pair, \(Y=1\) if dissimilar ("negative"). \(m\) is a margin hyperparameter — the minimum distance dissimilar pairs should be pushed apart to.

Reading the Two Terms

The \(\max(0, \cdot)\) is important: once two dissimilar embeddings are already farther apart than the margin, there's no benefit (and no gradient) to pushing them even further — the loss only cares about correcting pairs that are currently too close.

Numerical Example

With margin \(m=2\): a similar pair with \(D=0.5\): \(L=\frac{1}{2}(0.5)^2=0.125\). A dissimilar pair with \(D=0.5\) (too close!): \(L=\frac{1}{2}(2-0.5)^2=\frac{1}{2}(1.5)^2=1.125\) — a large penalty, correctly. A dissimilar pair with \(D=3\) (already far enough): \(L=\frac{1}{2}\max(0,2-3)^2=\frac{1}{2}(0)^2=0\) — no penalty, since \(\max(0,-1)=0\).

Code

import torch
import torch.nn.functional as F

def contrastive_loss(emb1, emb2, label, margin=2.0):
    D = F.pairwise_distance(emb1, emb2)
    loss_similar = (1 - label) * 0.5 * D.pow(2)
    loss_dissimilar = label * 0.5 * torch.clamp(margin - D, min=0).pow(2)
    return (loss_similar + loss_dissimilar).mean()

emb1 = torch.tensor([[1.0, 2.0], [0.0, 0.0]])
emb2 = torch.tensor([[1.1, 2.1], [5.0, 5.0]])
labels = torch.tensor([0.0, 1.0])   # first pair similar, second dissimilar
print(contrastive_loss(emb1, emb2, labels))

Where It's Used Today

Siamese network architectures for face verification (is this the same person?), signature verification, and as a foundational idea behind modern self-supervised contrastive learning methods like SimCLR (covered in the Self-Supervised Learning category), which extend this pairwise idea to much larger batches of positive/negative examples at once.

Common Mistakes

• Setting the margin \(m\) too small — dissimilar pairs stop contributing gradient once they're just barely past the margin, so a too-small margin lets the embedding space stay poorly separated overall.
• Forgetting that contrastive loss operates on pairs, not single examples — constructing good positive/negative pairs (or triplets, next note) is itself a significant part of the training pipeline design.

Interview Relevance

Q: "What is contrastive loss trying to achieve that standard classification losses aren't?" It shapes an embedding space directly — pulling similar inputs' representations close together and pushing dissimilar inputs' representations apart by at least a margin — rather than predicting a fixed set of class labels. This is essential for tasks like face verification, where the set of possible identities isn't fixed in advance.

Practice Question

With margin \(m=1.5\), compute the contrastive loss for a dissimilar pair (\(Y=1\)) with distance \(D=1.8\). Why does this produce zero loss?

Related DL Notes

• Triplet Loss
• SimCLR
• Vector Norms