Every number a neural network ever touches — an input pixel, a weight, a whole layer's parameters — is stored as one of three basic mathematical objects: a scalar, a vector, or a matrix. Getting comfortable with these (and their notation) is the prerequisite for everything else in this hub.
The Three Objects
| Object | Dimensionality | Notation | Example | DL Analogy |
|---|---|---|---|---|
| Scalar | 0-D — a single number | Italic lowercase: \(a\) | \(a = 5\) | A learning rate, a single bias value |
| Vector | 1-D — an ordered list of numbers | Bold lowercase: \(\mathbf{x}\) | \(\mathbf{x} = [1, 2, 3]\) | A feature vector, a word embedding |
| Matrix | 2-D — a grid of numbers (rows × columns) | Bold uppercase: \(\mathbf{W}\) | \(\mathbf{W} = \begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}\) | A fully-connected layer's weights |
Why This Matters Immediately
A single artificial neuron computes \(y = \mathbf{w}^\top \mathbf{x} + b\) — a vector \(\mathbf{x}\) (the inputs), a vector \(\mathbf{w}\) (the weights), and a scalar \(b\) (the bias). A full layer of neurons stacks many such weight vectors into a weight matrix \(\mathbf{W}\), so the whole layer becomes one matrix multiplication, \(\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}\). Every note in the Neural Network Fundamentals category builds directly on this notation.
Shape — The Most Important Attribute
A vector of length \(n\) has shape \((n,)\). A matrix with \(m\) rows and \(n\) columns has shape \((m, n)\). In practice, the single most common bug in deep learning code is a shape mismatch — so get in the habit of writing down a tensor's shape every time you define one.
Code
import numpy as np
scalar = 5.0
vector = np.array([1.0, 2.0, 3.0])
matrix = np.array([[1.0, 2.0], [3.0, 4.0]])
print(scalar, type(scalar))
print(vector.shape) # (3,)
print(matrix.shape) # (2, 2)import torch
scalar = torch.tensor(5.0)
vector = torch.tensor([1.0, 2.0, 3.0])
matrix = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print(scalar.shape, vector.shape, matrix.shape)
# torch.Size([]) torch.Size([3]) torch.Size([2, 2])Common Mistakes
- Confusing a vector of shape \((n,)\) with a matrix (column vector) of shape \((n, 1)\) — they behave differently under broadcasting and matrix multiplication (see Broadcasting).
- Writing \(\mathbf{w}^\top \mathbf{x}\) as \(\mathbf{w} \mathbf{x}\) — the transpose is not optional notation, it's what makes the shapes align for a valid dot product.
Interview Relevance
Q: "What's the difference between a vector and a 1-D matrix?" Mathematically, a vector is typically treated as either a row or column matrix. In deep learning frameworks, a plain 1-D tensor of shape \((n,)\) is neither — it has no row/column orientation, which matters for how it broadcasts against a 2-D matrix. A shape of \((n,1)\) or \((1,n)\) is an explicit row/column matrix.
Practice Question
A dataset has 100 samples, each with 8 features. What is the shape of the matrix holding this dataset? What is the shape of a single sample's feature vector?
