Multimodal AI refers to models and systems that work with more than one type of input or output — text, images, audio, video — instead of text alone. It's a category covering several genuinely different underlying technologies, not one single capability.
Two Very Different Things Both Called "Multimodal"
| Type | What It Means | Example |
|---|---|---|
| A single multimodal model | One model that natively accepts multiple input types (e.g. text + images) and reasons about them together | A vision-language model answering questions about an uploaded photo |
| A multimodal system (multiple specialized models) | Separate, specialized models chained together — one for speech-to-text, another (text-only) for reasoning, another for text-to-speech | A voice assistant pipeline: audio → STT → LLM → TTS → audio |
Both are legitimately called "multimodal," but they're architecturally very different — worth being precise about which one you mean in a given system design.
What This Section Covers
| Note | Focus |
|---|---|
| Vision-Language Models, Image Understanding, Document Vision | Models that reason about images and text together |
| Audio AI, Speech-to-Text, Text-to-Speech | Converting between speech and text |
| Image Generation, Video Generation | Generating visual media from text descriptions |
| Multimodal RAG | Retrieval-augmented generation over non-text content |
Practical Use Case
A document-processing application might use a vision-language model to read a scanned invoice directly (understanding both the layout and the text), rather than a separate OCR-then-text-LLM pipeline — an architectural choice worth evaluating case by case, since each approach has different cost, accuracy, and complexity tradeoffs.
Common Mistakes
- Assuming every "multimodal" capability comes from one unified model, when many real systems are actually pipelines of separate specialized models
- Overestimating current multimodal capabilities based on marketing rather than testing against your specific use case
Interview Relevance
Q: "What's the difference between a multimodal model and a multimodal system?" — a single model natively handling multiple input types together, versus multiple specialized models chained in a pipeline, is the expected distinction.
Practice Question
For a voice-controlled customer support bot, sketch whether you'd use a single multimodal model or a pipeline of specialized models, and why.