A tensor is simply the generalization of scalars, vectors and matrices to any number of dimensions. In deep learning frameworks, "tensor" is the universal container for data — every input, weight and activation is a tensor of some rank.
Tensor Rank (Number of Dimensions)
| Rank | Name | Shape Example | DL Example |
|---|---|---|---|
| 0 | Scalar | () | A single loss value |
| 1 | Vector | (n,) | A feature vector, a bias vector |
| 2 | Matrix | (m, n) | A weight matrix, a batch of 1-D signals |
| 3 | 3-D Tensor | (C, H, W) | A single image: channels × height × width |
| 4 | 4-D Tensor | (N, C, H, W) | A batch of \(N\) images |
| 5 | 5-D Tensor | (N, T, C, H, W) | A batch of video clips: batch × time × channels × height × width |
Visualizing an Image as a 3-D Tensor
A single RGB image is a rank-3 tensor — one 2-D grid of pixel intensities per color channel, stacked together.
Code — Building Tensors of Different Ranks
import torch
image = torch.randn(3, 224, 224) # one RGB image: (C, H, W)
batch_of_images = torch.randn(32, 3, 224, 224) # a batch of 32 images: (N, C, H, W)
print(image.ndim, image.shape) # 3 torch.Size([3, 224, 224])
print(batch_of_images.ndim, batch_of_images.shape) # 4 torch.Size([32, 3, 224, 224])
print(image.dtype, image.device) # torch.float32 cpuKey Tensor Attributes You'll Use Constantly
| Attribute | Meaning |
|---|---|
.shape | Size along each dimension |
.ndim | Number of dimensions (the tensor's rank) |
.dtype | Data type of the elements (e.g. float32, int64) |
.device | Where the tensor lives — CPU or GPU (cuda) |
Common Mistakes
- Thinking "tensor" implies something mathematically exotic — in deep learning frameworks it's simply an n-dimensional array; the deeper mathematical-physics notion of a tensor (with transformation rules under change of basis) isn't what PyTorch/TensorFlow code is invoking.
- Mixing up the (N, C, H, W) channel-first convention (PyTorch default) with the (N, H, W, C) channel-last convention (common in TensorFlow) — feeding a tensor in the wrong layout is a very common, very silent bug.
Interview Relevance
Q: "What's the shape of a batch of 64 grayscale 28×28 images?" \((64, 1, 28, 28)\) in PyTorch's channel-first convention — batch size, then 1 channel (grayscale), then height, then width.
Practice Question
What is the shape and rank of a batch of 16 audio clips, each represented as a sequence of 8,000 samples with 1 channel? Write it in (N, C, L) form.
