Multimodal learning builds models that process and combine multiple different types of data — text, images, audio, video — jointly, rather than being restricted to a single modality the way every architecture covered individually elsewhere in this hub is.
Why Multiple Modalities Matter
Real-world understanding is rarely confined to a single modality — a video combines visual frames with an audio track; a product listing combines an image with a text description; a medical diagnosis might combine an X-ray image with a patient's written history. A model that can jointly reason across modalities can capture richer, more grounded understanding than any single-modality model working in isolation.
Three Fusion Strategies
| Strategy | How Modalities Combine | Tradeoff |
|---|---|---|
| Early fusion | Combine raw or lightly-processed inputs from each modality before any deep processing | Lets the model learn cross-modal interactions from the start, but requires modalities to be aligned in a compatible representation early on |
| Late fusion | Process each modality with an entirely separate network, combine only the final outputs | Simple, modular, but misses potentially useful early cross-modal interactions |
| Joint/intermediate fusion | Process each modality somewhat independently, then combine intermediate representations (often via cross-attention, as in Cross-Attention) at one or more points | The most common modern approach — balances modality-specific processing with genuine cross-modal interaction |
Code — A Simple Late-Fusion Sketch
import torch
import torch.nn as nn
class LateFusionModel(nn.Module):
def __init__(self, image_encoder, text_encoder, fusion_dim):
super().__init__()
self.image_encoder = image_encoder # e.g. a CNN
self.text_encoder = text_encoder # e.g. a Transformer
self.classifier = nn.Linear(fusion_dim * 2, 10)
def forward(self, image, text):
image_features = self.image_encoder(image) # processed ENTIRELY separately
text_features = self.text_encoder(text) # processed ENTIRELY separately
combined = torch.cat([image_features, text_features], dim=1) # combined only at the very end
return self.classifier(combined)Common Mistakes
- Assuming naive concatenation of modality features (a simple form of late fusion) always captures meaningful cross-modal relationships — this approach can miss important interactions that only emerge from deeper, more integrated fusion, which is why intermediate fusion via cross-attention has become the more common modern default.
- Ignoring severe imbalance in how much each modality contributes to the training signal — a model can sometimes learn to rely almost entirely on the "easier" modality, effectively ignoring a genuinely useful but harder-to-learn-from second modality.
Interview Relevance
Q: "What's the tradeoff between early, late and intermediate fusion strategies for a multimodal model?" Early fusion allows learning cross-modal interactions from the very start but requires modalities to already be in a compatible representation, which can be restrictive. Late fusion keeps modality-specific processing clean and modular but can miss useful early interactions between modalities. Intermediate fusion — combining features at one or more points partway through processing, often via cross-attention — is the common modern compromise, balancing modality-specific depth against genuine cross-modal integration.
Practice Question
Why might a naively concatenation-based (late fusion) multimodal model risk relying almost entirely on just one modality during training, effectively ignoring the other?
