Maximum Likelihood Estimation (MLE) is the principle of choosing the parameters that make the observed data as probable as possible. It is, quite directly, the theoretical justification for why neural networks are trained by minimizing cross-entropy or mean squared error loss.
The Core Idea
Given a dataset, MLE searches over candidate parameter values \(\theta\) and picks the one under which the observed data would have been most probable. For the coin example in Likelihood, the MLE estimate for \(\theta\) turns out to be exactly \(\frac{7}{10}=0.7\) — the observed proportion of heads, which matches intuition.
Why Log-Likelihood Is Used in Practice
Multiplying many small probabilities together (as you would for a dataset of \(N\) independent examples) causes numerical underflow. Taking the logarithm converts the product into a sum, which is both numerically stable and doesn't change which \(\theta\) maximizes it (since \(\log\) is monotonically increasing):
And since minimizing a negative quantity is equivalent to maximizing the positive one, this becomes the negative log-likelihood (NLL) — the exact quantity most deep learning loss functions minimize.
Deriving Cross-Entropy Loss from MLE
Assume a classifier's true labels follow a categorical distribution, and the model predicts \(P(y=c\mid x;\theta)\) for each class \(c\). The negative log-likelihood of the correct label \(y\) for one example is:
Summed (or averaged) over the whole training set, this is exactly the cross-entropy loss formula covered in Cross-Entropy and used throughout classification. Cross-entropy loss isn't an arbitrary engineering choice — it's the direct consequence of applying MLE under a categorical label distribution.
Deriving Mean Squared Error from MLE
Now assume regression targets follow a Gaussian distribution around the model's prediction: \(y \sim \mathcal{N}(f(x;\theta), \sigma^2)\). Plugging the Gaussian PDF into the negative log-likelihood and simplifying, the terms not depending on \(\theta\) drop out, leaving exactly:
This is precisely mean squared error, up to a constant. MLE under a Gaussian noise assumption is squared-error minimization — this is why MSE is the default regression loss, not an arbitrary convention.
Code — MLE for a Bernoulli Parameter
import numpy as np
from scipy.optimize import minimize_scalar
flips = np.array([1,1,1,1,1,1,1,0,0,0]) # 7 heads, 3 tails
def neg_log_likelihood(theta):
return -np.sum(flips * np.log(theta) + (1 - flips) * np.log(1 - theta))
result = minimize_scalar(neg_log_likelihood, bounds=(1e-6, 1-1e-6), method='bounded')
print(result.x) # approximately 0.7 -- matches the analytical MLE solutionCommon Mistakes
- Treating cross-entropy and MSE as unrelated, arbitrary loss formulas — they're both instances of the same underlying MLE principle, just under different assumed output distributions (categorical vs Gaussian).
- Forgetting that MLE can overfit with limited data — it has no built-in preference for "simpler" parameter values, which is exactly the gap that regularization (a Bayesian-flavored prior on parameters) is designed to fill.
Interview Relevance
Q: "Why is cross-entropy the standard loss function for classification, rather than something else?" Cross-entropy loss is exactly the negative log-likelihood of the true label under the model's predicted categorical distribution. Minimizing it is equivalent to maximum likelihood estimation — finding the model parameters that make the observed training labels as probable as possible under the model's predictions.
Practice Question
Explain, in your own words, why minimizing mean squared error is equivalent to maximum likelihood estimation under the assumption that prediction errors are normally distributed.
