Free · 490 Topics · No Signup
Deep Learning Notes
Neural networks & modern AI — written by CodingNow 2.0's mentors. Free to read, structured to actually help you learn.
Deep Learning notes by CodingNow 2.0 cover 490 topics — from what is deep learning? to deployment project — each explained with short definitions, syntax and runnable code examples. They are 100% free, need no signup, and work as quick revision for college exams, Deep Learning interviews and CodingNow 2.0's mentor-led Deep Learning course in Pitampura, Delhi.
What Is Deep Learning?
What Artificial Intelligence is, its sub-fields, and how deep learning fits inside it — the map for this whole notes hub.
AI vs ML vs Deep Learning
The exact containment relationship between AI, Machine Learning and Deep Learning, with a side-by-side comparison table.
Traditional ML vs Deep Learning
Manual feature engineering vs automatic feature learning, compared side by side with a pipeline diagram.
Evolution of Deep Learning
From the 1958 Perceptron through two AI winters to the 2012 breakthrough and the modern LLM era.
Why Deep Learning Became Successful
The three pillars — data, compute and algorithmic fixes — that made deep networks finally trainable at scale.
Applications of Deep Learning
Real-world deep learning applications across computer vision, NLP, healthcare, finance and generative AI.
Types of Learning in Deep Learning
Supervised, unsupervised, semi-supervised, self-supervised and reinforcement learning in a deep learning context.
Deep Learning Workflow
The end-to-end deep learning project workflow, from problem definition to deployment and monitoring.
Components of a Deep Learning System
Data, model, loss function, optimizer and hardware — the components every trained deep learning system depends on.
Challenges in Deep Learning
Data hunger, compute cost, overfitting, vanishing gradients and interpretability — the real engineering challenges.
Advantages & Limitations of Deep Learning
A practical decision guide for when deep learning's advantages outweigh its costs, and when they don't.
Scalars, Vectors & Matrices
The three basic mathematical objects behind every neural network — scalars, vectors and matrices — with notation and shapes explained.
Tensors
Tensors as the n-dimensional generalization of scalars, vectors and matrices, with rank, shape and PyTorch code.
Matrix Addition
Element-wise matrix addition, its formula, and its role in bias terms and ResNet residual connections.
Matrix Multiplication
The shape rule, a full worked example, and why matrix multiplication is the core operation of every linear layer.
Dot Product
The dot product formula, geometric intuition, and its role in cosine similarity and attention scores.
Matrix Transpose
What matrix transpose does, its key properties, and its role in backpropagation and attention.
Matrix Inverse
The matrix inverse, the 2x2 formula, and why deep learning uses iterative optimization instead of computing inverses.
Determinant
The determinant formula, its geometric meaning as area/volume scaling, and its link to invertibility.
Vector Norms
L1, L2 and L-infinity norms compared, with their role in regularization and gradient clipping.
Eigenvalues
What eigenvalues are, the characteristic equation, a worked numerical example, and their role in PCA.
Eigenvectors
What eigenvectors are, how to find them, and how they power Principal Component Analysis.
Vector Spaces
The formal definition of a vector space, basis, span and dimension, connected to feature and embedding spaces.
Linear Transformations
What linear transformations are, common examples, and why they motivate the need for activation functions.
Broadcasting
The broadcasting rule in NumPy and PyTorch, worked examples, and common shape bugs to avoid.
Functions and Limits
What functions and limits are, and why calculus is needed at all to train a neural network.
Derivatives
The formal definition of a derivative, common rules, geometric meaning, and PyTorch autograd.
Partial Derivatives
How partial derivatives isolate the sensitivity of a multi-variable loss to a single weight.
Chain Rule
The chain rule formula, a worked example, and why it is the mathematical basis of backpropagation.
Gradient
What the gradient vector is, why it points uphill, and how gradient descent uses it to minimize loss.
Gradient Vector
The shape and interpretation of a gradient vector for a real weight vector, with a PyTorch example.
Jacobian
The Jacobian matrix for multi-input, multi-output functions, and why every network layer needs one.
Hessian
The Hessian matrix of second derivatives, what it reveals about minima, maxima and saddle points.
Computational Graphs
How computational graphs represent a function's operations, and how PyTorch uses them for autograd.
Why Calculus for Neural Networks
A complete end-to-end summary connecting derivatives, the chain rule and gradients to how a network actually trains.
Probability Basics
Core probability definitions and axioms, and why a classifier's softmax output must satisfy them.
Random Variables
Discrete vs continuous random variables, and how dataset labels and weight initialization use them.
Probability Distributions
PMFs, PDFs, the Bernoulli, categorical and normal distributions, and their roles across deep learning.
Expected Value
The formula for expected value, and how training loss is literally an empirical expected value.
Variance & Standard Deviation
Variance and standard deviation formulas, and why they motivate feature standardization and batch normalization.
Covariance
The covariance formula, the covariance matrix, and its role in PCA and dimensionality reduction.
Conditional Probability
The conditional probability formula, a spam-filter worked example, and independence.
Bayes' Theorem
Bayes' theorem formula, prior/likelihood/posterior, and a full worked medical-test example.
Likelihood
The difference between probability and likelihood, with a coin-flip worked example.
Maximum Likelihood Estimation
MLE explained, and a full derivation of cross-entropy and MSE loss from maximum likelihood principles.
Entropy
The entropy formula, worked examples, and how it measures uncertainty in a probability distribution.
Cross-Entropy
The cross-entropy formula, a full worked example, and PyTorch's CrossEntropyLoss explained.
KL Divergence
The KL divergence formula, its relationship to cross-entropy and entropy, and its use in VAEs and RLHF.
Biological Neuron
The biological neuron structure that inspired artificial neural networks, mapped part by part.
Artificial Neuron
The full artificial neuron computation — weighted sum plus activation — with a worked numerical example.
McCulloch-Pitts Neuron
The first mathematical neuron model from 1943, and how it modeled logical AND and OR without learning.
Perceptron
The 1958 Perceptron model, its linear decision boundary, and what makes it different from earlier neuron models.
Perceptron Learning Algorithm
The Perceptron's weight update rule, a full worked example, and the Perceptron Convergence Theorem.
Limitations of Perceptron
Why a single Perceptron cannot learn XOR, proven algebraically, and how a hidden layer fixes it.
Multi-Layer Perceptron
The MLP architecture that solves XOR, with a diagram and full PyTorch implementation.
Neural Network Architecture
Input, hidden and output layers, depth vs width, and common architecture patterns.
Input, Hidden & Output Layers
What each layer type does, and how to correctly size and activate the output layer for any task.
Weights and Bias
What weights and bias represent, weight matrix shapes, and why zero-initialization fails.
Parameters vs Hyperparameters
The precise distinction between learned parameters and chosen hyperparameters, with clear examples.
Forward Propagation
The forward propagation formula, a full numerical walkthrough, and matching PyTorch code.
Loss Function Basics
What a loss function measures, MSE vs cross-entropy introduced, and loss vs evaluation metric.
Backpropagation (Intro)
A conceptual introduction to backpropagation as efficient gradient computation via the chain rule.
Gradient Descent (Intro)
The gradient descent update rule, why learning rate matters, and a complete PyTorch training step.
Weight Updates
A complete worked example of one full training step, from forward pass to weight update, verified in PyTorch.
Step Function
The original 1958 Perceptron activation function, and why its zero gradient makes it unusable for backpropagation.
Linear Activation
The linear (identity) activation, why it collapses network depth, and where it's actually used in regression outputs.
Sigmoid Function
The sigmoid formula, derivative, graph, and why it causes vanishing gradients in deep networks.
Tanh Function
The tanh formula, derivative, and how it improves on sigmoid while sharing its saturation problem.
ReLU
The ReLU formula, why it fixes vanishing gradients, and the dying ReLU problem explained.
Leaky ReLU
How Leaky ReLU's small negative slope fixes the dying ReLU problem, with formula and code.
PReLU
How PReLU makes the negative slope a learnable parameter instead of a fixed hyperparameter.
ELU
The ELU formula, its smooth exponential negative curve, and its tradeoffs versus ReLU.
SELU
The self-normalizing SELU activation, its strict requirements, and why it's used less often in practice.
GELU
The GELU formula, its probabilistic derivation, and why it's the default in Transformer architectures.
Swish
The Swish activation function, discovered via search, and its close relationship to GELU.
Softmax Function
The softmax formula, a full worked numerical example, and the numerical stability trick every framework uses.
Sigmoid vs Tanh
A direct comparison of sigmoid and tanh, and why both still appear inside LSTM and GRU gates.
ReLU vs Leaky ReLU
A direct comparison of ReLU and Leaky ReLU, and when the dying ReLU problem actually justifies switching.
ReLU vs GELU
Why CNNs favor ReLU while Transformers favor GELU, compared side by side.
Sigmoid vs Softmax
When to use sigmoid vs softmax for an output layer, based on whether classes are mutually exclusive.
Mean Absolute Error
The MAE formula, a worked example, and why it's more robust to outliers than MSE.
Mean Squared Error
The MSE formula, a worked example, and its derivation from maximum likelihood under a Gaussian assumption.
Root Mean Squared Error
The RMSE formula and why it's reported as a metric while MSE is used as the training loss.
Huber Loss
The Huber loss formula, a worked example, and how it combines MSE and MAE's strengths.
Binary Cross-Entropy
The BCE formula, a worked example, and the critical BCELoss vs BCEWithLogitsLoss distinction.
Categorical Cross-Entropy
The categorical cross-entropy formula, a worked example, and PyTorch's expected label format.
Sparse Categorical Cross-Entropy
Why sparse categorical cross-entropy is mathematically identical to categorical cross-entropy, just a label format difference.
Focal Loss
The Focal Loss formula and how it down-weights easy examples to fix severe class imbalance.
Contrastive Loss
The contrastive loss formula, a worked example, and how it shapes embedding spaces for similarity.
Triplet Loss
The triplet loss formula, a worked example, and the hard negative mining challenge in practice.
KL Divergence Loss
How KL divergence is implemented as a practical loss function, with the PyTorch input-format gotcha explained.
Reconstruction Loss
MSE vs BCE reconstruction loss for autoencoders, and its role alongside KL divergence in a VAE.
Gradient Descent
The optimization problem formalized, convex vs non-convex loss surfaces, and iterative descent visualized.
Batch Gradient Descent
How batch gradient descent uses the full dataset per update, and why it's rarely used in practice.
Stochastic Gradient Descent
How SGD uses one random example per update, its noisy path, and why that noise can help.
Mini-Batch Gradient Descent
The practical compromise between batch and stochastic gradient descent, and how it maps to PyTorch's DataLoader.
Learning Rate
A deep dive on the learning rate hyperparameter, typical ranges, and its interaction with batch size.
Momentum
The momentum formula, the physical rolling-ball analogy, and how it smooths oscillation in narrow ravines.
Nesterov Momentum
How Nesterov momentum's look-ahead gradient improves on standard momentum.
AdaGrad
The AdaGrad formula, per-parameter adaptive learning rates, and its diminishing learning rate flaw.
RMSProp
How RMSProp fixes AdaGrad's diminishing learning rate problem with a decaying average.
Adam Optimizer
The Adam optimizer's formula, bias correction explained, and why it became the default optimizer.
AdamW
How AdamW fixes Adam's broken interaction with L2 regularization, with a full optimizer comparison table.
Learning Rate Scheduling
Why a fixed learning rate is a compromise, and an overview of the major scheduling strategies.
Step Decay
The step decay formula, a worked example, and its staircase-shaped learning rate curve.
Exponential Decay
The exponential decay formula and how it compares to step decay's sudden drops.
Cosine Annealing
The cosine annealing formula, why its smooth shape is popular, and warm restarts explained.
Warmup Learning Rate
Why gradually increasing the learning rate at the start of training stabilizes early updates.
One Cycle Learning Rate
The One Cycle policy's rise-fall-anneal shape and how it enables faster "super-convergence" training.
Why Backpropagation
Why backpropagation exists — the impractical cost of finite differences versus backprop's efficiency.
Chain Rule in Backpropagation
How the chain rule applies to a multi-layer network, with local gradients reused across layers.
Forward Pass
What values must be cached during the forward pass for backpropagation to use later.
Backward Pass
The recursive formula for computing each layer's error signal during the backward pass.
Gradient Calculation
The outer product formula for computing weight and bias gradients from a layer's error signal.
Backpropagation Weight Updates
How backpropagation's computed gradients are handed off to a separate, swappable optimizer.
Backpropagation Worked Example
A complete numerical backpropagation example through a 2-layer network, verified with PyTorch autograd.
Vanishing Gradient Problem
The full mechanism behind vanishing gradients, a numerical demonstration, and every solution mapped out.
Exploding Gradient Problem
The mechanism behind exploding gradients, a numerical demonstration, and how to recognize it in practice.
Gradient Clipping
The gradient clipping formula, a worked example, and why it's standard practice for RNNs and Transformers.
Dataset Train/Val/Test Split
Why datasets are split into three parts, typical ratios, and the data leakage traps to avoid.
Epoch, Batch, Iteration
The precise definitions of epoch, batch and iteration, and how they relate mathematically.
Training Loop
The complete, annotated standard PyTorch training loop structure, piece by piece.
Validation Loop
The complete validation loop structure, model.eval(), and why it differs from the training loop.
Checkpointing
What to save in a training checkpoint, periodic vs best-model checkpointing, with full PyTorch code.
Early Stopping
The patience-based early stopping algorithm, with full code and how to choose the patience value.
Model Saving and Loading
state_dict vs whole-model saving in PyTorch, and how to correctly load partial weights for transfer learning.
Underfitting
The signature of underfitting, its common causes, and the specific fixes for each.
Overfitting
The signature of overfitting, why it happens, and every major fix explained.
Bias-Variance Tradeoff
The formal bias-variance decomposition of model error, and how it explains underfitting and overfitting.
Generalization in Deep Learning
What generalization means, the generalization gap, and why deep networks generalize despite huge parameter counts.
Why Regularization
The general idea behind regularization — trading a little bias for a lot less variance.
L1 Regularization
The L1 regularization formula, why it produces sparse weights, and when it's the right choice.
L2 Regularization
The L2 regularization formula, why it shrinks weights smoothly, and why it's the common default.
Weight Decay
The weight decay formula, its algebraic equivalence to L2 under SGD, and why it breaks under Adam.
Dropout
How dropout randomly disables neurons during training, inverted dropout scaling, and why it works.
Data Augmentation
How data augmentation expands the effective training set, common techniques, and the label-preservation rule.
Regularization Comparison
A complete side-by-side comparison of every regularization technique, and a practical decision guide.
Why Normalization
The shared formula behind every normalization technique, and the shifting-activation-distribution problem they solve.
Batch Normalization
The BatchNorm formula, the critical training-vs-inference statistics split, and its small-batch weakness.
Layer Normalization
The LayerNorm formula, how it differs from BatchNorm, and why Transformers use it.
Instance Normalization
The InstanceNorm formula and why it suits style transfer by removing per-instance contrast.
Group Normalization
The GroupNorm formula, how it generalizes InstanceNorm and LayerNorm, and its batch-size independence.
RMSNorm
The RMSNorm formula, why it drops mean-centering, and a complete comparison of all 5 normalization techniques.
Confusion Matrix
The 2x2 confusion matrix foundation — TP, TN, FP, FN — every classification metric is built from.
Accuracy
The accuracy formula and why it can be deeply misleading on imbalanced classification datasets.
Precision & Recall
The precision and recall formulas, their tradeoff, and how to choose which matters more by context.
F1 Score
The F1 score formula, why it uses a harmonic mean, and the F-beta generalization.
Specificity
The specificity (true negative rate) formula and how it complements recall.
ROC Curve
The ROC curve's TPR vs FPR plot across every threshold, with a worked code example.
ROC-AUC
The ROC-AUC metric, its probabilistic interpretation, and a worked numerical example.
Precision-Recall Curve
The Precision-Recall curve and why it's more informative than ROC for imbalanced datasets.
PR-AUC
The PR-AUC metric, its class-imbalance-dependent baseline, and when to prefer it over ROC-AUC.
Regression Metrics
The R² metric formula, a worked example, and how MAE/MSE/RMSE double as evaluation metrics.
Perplexity
The perplexity formula for language model evaluation, and its intuitive branching-factor interpretation.
BLEU Score
The BLEU score formula, n-gram precision, the brevity penalty, and its known limitations.
ROUGE Score
The ROUGE score formula, why it emphasizes recall, and how it compares to BLEU.
IoU (Intersection over Union)
The IoU formula, a worked bounding box example, and its role in object detection matching.
Dice Score
The Dice score formula, its exact relationship to IoU, and why it's popular in medical imaging.
Mean Average Precision
How mAP combines IoU matching and per-class precision-recall into the standard object detection metric.
What Is CNN?
A high-level introduction to CNNs, the convolution/pooling/FC pipeline, and a minimal PyTorch example.
Why CNN?
Why plain MLPs fail on images, and how sparse connectivity and parameter sharing fix it, with a parameter comparison.
Image Representation
How images are represented as tensors, pixel value ranges, and channel-first vs channel-last conventions.
Convolution Operation
The convolution formula, a worked numerical example, and its implementation in PyTorch.
Kernel and Filter
The precise difference between a kernel and a filter, with classic edge-detection kernel examples.
Feature Map
What a feature map represents, and how multiple filters produce multiple stacked feature maps.
Stride
The stride parameter, its output size formula, and why larger strides act as built-in downsampling.
Padding
The padding parameter, valid vs same padding, and why it prevents edge-pixel underprocessing.
Channels
How multi-channel convolution combines input channels, and input vs output channel counts.
Receptive Field
How receptive field grows across stacked layers, and why small kernels are stacked instead of large ones.
Pooling
The general pooling concept, why it helps, and its zero-parameter downsampling benefit.
Max Pooling
The max pooling formula, a worked numerical example, and why it suits feature detection.
Average Pooling
The average pooling formula, a worked example, and when it's preferred over max pooling.
Global Average Pooling
How GAP collapses entire feature maps, with a dramatic parameter-count comparison to flattening.
Flattening
How flattening bridges spatial feature maps to fully connected layers, with a worked example.
Fully Connected Layer
The role of the fully connected layer at the end of a CNN, and why it often dominates parameter count.
Manual Convolution Example
A complete manual convolution example with padding and stride, verified against PyTorch's conv2d.
LeNet
The original 1998 CNN for digit recognition, establishing the conv-pool-FC template.
AlexNet
The 2012 ImageNet breakthrough that kicked off the modern deep learning era.
VGG
The VGG architecture's uniform 3x3 kernel design philosophy, and its parameter count tradeoff.
GoogLeNet
How GoogLeNet's Inception modules and 1x1 convolutions achieved efficiency over VGG.
Inception Network
A deep dive into the Inception module's parallel multi-scale branches and concatenation.
ResNet
How residual connections solved vanishing gradients, enabling networks over 100 layers deep.
DenseNet
How DenseNet's dense, concatenated connections maximize feature reuse across layers.
MobileNet
How depthwise separable convolutions make MobileNet efficient for mobile deployment.
EfficientNet
How compound scaling balances depth, width and resolution together for efficient accuracy.
Xception
How Xception takes the Inception idea to its extreme with depthwise separable convolutions throughout.
ConvNeXt
How ConvNeXt modernized CNNs with Transformer-inspired design to match Vision Transformer performance.
Image Classification
The core image classification task, the standard CNN pipeline, and benchmark datasets.
Transfer Learning
Feature extraction vs fine-tuning for reusing pretrained CNNs, with PyTorch code for both.
Object Detection
The object detection task, one-stage vs two-stage detector families, and evaluation with mAP.
Image Segmentation
The general pixel-level segmentation task, and the three flavors: semantic, instance and panoptic.
Semantic Segmentation
Per-pixel class labeling without instance distinction, with a worked numerical example.
Instance Segmentation
How instance segmentation distinguishes individual objects, built via detection plus masking.
Panoptic Segmentation
How panoptic segmentation unifies "things" and "stuff" into one complete scene output.
Face Recognition
Embedding-based face verification and identification using triplet and contrastive loss.
OCR
The two-stage OCR pipeline of text detection and sequence-based text recognition.
Image Captioning
The CNN encoder plus RNN/Transformer decoder pipeline for generating image descriptions.
Pose Estimation
Heatmap-based keypoint detection for reconstructing body pose from an image.
R-CNN
The original region-proposal-based object detector, and why it was extremely slow.
Fast R-CNN
How Fast R-CNN shares CNN computation across regions using ROI Pooling.
Faster R-CNN
How the Region Proposal Network made object detection fully end-to-end trainable.
SSD (Object Detection)
How SSD predicts detections in a single pass using multi-scale feature maps.
YOLO
The grid-based single-pass detection approach behind YOLO's real-time speed.
FCN (Fully Convolutional Network)
How FCN replaced fully connected layers with convolutions for practical semantic segmentation.
Mask R-CNN
How Mask R-CNN adds instance segmentation to Faster R-CNN with ROI Align and a mask branch.
Sequential Data
What makes data sequential, common examples, and why standard feedforward networks struggle with it.
Why RNN
The three requirements a sequence model needs, and how the RNN architecture satisfies all three.
RNN Architecture
The three weight matrices of an RNN cell, and how RNN architecture compares to a standard feedforward layer.
Hidden State
What an RNN's hidden state represents, its fixed-size bottleneck, and the batch-size tradeoff.
Recurrent Connections
The recurrent connection formalized, and why the same weights must be shared across every time step.
Unrolling RNN
How unrolling an RNN across time steps reveals its connection to backpropagation and network depth.
RNN Forward Propagation
The complete RNN forward pass formulas, with a full 3-step numerical example verified in PyTorch.
Backpropagation Through Time
How BPTT sums gradient contributions across time steps for RNN's shared weights, and truncated BPTT.
RNN Vanishing Gradient
Why vanishing gradients hit RNNs especially hard over long sequences, with a numerical example.
RNN Exploding Gradient
Why RNNs are especially prone to exploding gradients, and why gradient clipping is nearly mandatory.
Limitations of RNN
The complete list of RNN limitations, why LSTM/GRU exist, and the parallelization problem they still don't solve.
Why LSTM
The core motivation for LSTM — a cell state pathway that protects gradients across long sequences.
LSTM Architecture
The five components of an LSTM cell and how they connect, with a diagram and PyTorch weight shapes.
LSTM Cell State
The cell state update formula and why its largely-additive structure protects gradients across time.
LSTM Hidden State
How the LSTM hidden state is derived from the cell state, and why the two states are kept separate.
Forget Gate
The LSTM forget gate formula, a worked example, and how it decides what memory to discard.
Input Gate
The LSTM input gate formula, a worked example, and how it decides how much new information to add.
Candidate State
The LSTM candidate state formula, why it uses tanh instead of sigmoid, and a worked example.
Output Gate
The LSTM output gate formula and how it controls what the cell state exposes as the hidden state.
LSTM Equations
All six LSTM equations assembled into one complete reference, with a summary table and diagram.
LSTM Forward Pass
A complete numerical LSTM forward pass through one time step, verified against PyTorch.
LSTM Advantages & Limitations
LSTM's genuine strengths and remaining weaknesses, including the parallelization problem it never solved.
LSTM Applications
Real-world LSTM applications — translation, speech recognition, forecasting — with PyTorch code examples.
GRU Architecture
GRU's simplified structure compared to LSTM — one state, two gates instead of two states, three gates.
GRU Update Gate
The GRU update gate formula, how it merges LSTM's forget and input gates, with a worked example.
GRU Reset Gate
The GRU reset gate formula and how it shapes the candidate hidden state computation.
GRU Equations
All four GRU equations assembled together, with a complete numerical worked example.
GRU vs LSTM
A complete side-by-side comparison of GRU and LSTM, with practical guidance on which to choose.
Encoder-Decoder Architecture
How splitting a model into an encoder and decoder decouples input and output sequence lengths.
Context Vector
The fixed-size context vector, why it's a bottleneck, and its role in basic Seq2Seq models.
Seq2Seq Model
The complete assembled Seq2Seq pipeline and autoregressive decoding, with a full code example.
Teacher Forcing
How teacher forcing speeds up Seq2Seq training, and the exposure bias it introduces.
Seq2Seq Limitations
The complete list of basic Seq2Seq limitations that directly motivated the attention mechanism.
Why Attention
The core motivation for attention, and how it directly fixes the Seq2Seq context-vector bottleneck.
Query, Key, Value
The Q/K/V framework explained via a search-engine analogy, with formulas and a numerical example.
Attention Score
How the dot product between query and key produces a raw relevance score, with a worked example.
Dot-Product Attention
The complete dot-product attention formula, from scores to softmax to weighted value sum.
Scaled Dot-Product Attention
Why attention scores are divided by the square root of dk, and how this keeps softmax gradients healthy.
Self-Attention
How self-attention relates every token in a sequence directly, without an RNN's distance penalty.
Cross-Attention
How cross-attention lets a decoder query an encoder's outputs, formally solving the Seq2Seq bottleneck.
Multi-Head Attention
How running multiple attention heads in parallel lets a model capture different relationship types.
Transformer Motivation
Why "Attention Is All You Need" removed recurrence entirely, and why parallelization matters so much.
Transformer Architecture
The complete Transformer architecture at a glance, plus encoder-only, decoder-only and encoder-decoder variants.
Transformer Encoder
One Transformer encoder layer's structure — self-attention and feed-forward, each with residual and norm.
Transformer Decoder
One Transformer decoder layer's structure — masked self-attention, cross-attention, and feed-forward.
Positional Encoding
The sinusoidal positional encoding formula, why sine and cosine, with a worked example.
Feed-Forward Network (Transformer)
The position-wise feed-forward sublayer's formula and why it adds non-linear capacity to attention output.
Residual Connections
How residual connections around every Transformer sublayer enable training very deep stacks.
Layer Normalization (Transformer)
Post-norm vs pre-norm placement in Transformer blocks, and why layer norm is used over batch norm.
Masked Self-Attention
How causal masking prevents the decoder from seeing future tokens, with the exact masking mechanism.
Transformer Data Flow
The complete Transformer data flow from input tokens to predicted output, with a full PyTorch implementation.
Text Preprocessing
Classical text preprocessing steps and why modern Transformers rely on much less of them.
Tokenization
Word, character and subword tokenization compared, and why BPE became the modern standard.
Vocabulary
The vocabulary size tradeoff and how it directly sizes a model's embedding and output layers.
One-Hot Encoding (NLP)
Why one-hot word encoding fails — huge dimensionality and zero notion of similarity.
Word Embeddings
How dense word embeddings capture semantic similarity, with the famous king-queen analogy.
Word2Vec
The self-supervised idea behind Word2Vec, and a preview of its CBOW and Skip-Gram architectures.
CBOW
The Continuous Bag of Words formula and how it predicts a target word from averaged context.
Skip-Gram
The Skip-Gram formula, and a direct comparison with CBOW including rare-word performance.
GloVe
How GloVe learns embeddings from global co-occurrence statistics, compared to Word2Vec.
Contextual Embeddings
Why static embeddings can't distinguish word meanings, and how contextual embeddings fix this.
BERT
BERT's bidirectional architecture, Masked Language Modeling, and Next Sentence Prediction explained.
RoBERTa
How RoBERTa improved on BERT using the identical architecture, purely through a better training recipe.
T5 Model
How T5 reframes every NLP task as text-to-text using a full encoder-decoder Transformer.
GPT Architecture
GPT's decoder-only architecture, next-token prediction, and a comparison against BERT.
What Is an LLM?
What makes a model "large," emergent capabilities, and why LLMs aren't automatically chatbots.
LLM Architecture
The modern refinements (RMSNorm, RoPE, grouped-query attention) on top of the base Transformer decoder.
Tokens and Tokenization (LLMs)
Why token counts matter for cost, context budget and latency, with tiktoken code.
Embeddings (LLMs)
The LLM input embedding layer at scale, and the weight-tying trick between input and output layers.
Positional Embeddings
Learned positional embeddings vs Rotary Positional Embeddings (RoPE), and why RoPE became dominant.
Transformer Blocks
The transformer block as the fundamental unit of LLM scale, and depth vs width tradeoffs.
LLM Pretraining
The massive self-supervised pretraining stage, and a brief look at scaling laws.
Next-Token Prediction
The full formula for next-token prediction and the parallel-supervision trick causal masking enables.
Supervised Fine-Tuning
How SFT reshapes a pretrained model using a much smaller, curated dataset of example responses.
Instruction Tuning
How training on diverse instruction-phrased tasks teaches models to generalize to unseen instructions.
LLM Alignment
Why capability and alignment are distinct concerns, and what alignment actually aims to achieve.
RLHF
The full 3-stage RLHF process — reward model training and RL fine-tuning with a KL penalty.
DPO
How Direct Preference Optimization achieves RLHF-like results without a reward model or RL.
Context Window
Why attention's quadratic cost limits context window size, with a practical token-budget example.
Temperature (Sampling)
The temperature sampling formula and how it sharpens or flattens a model's output distribution.
Top-K Sampling
The top-K sampling algorithm, a worked example, and its fixed-count limitation.
Top-P Sampling
Nucleus (top-P) sampling, how it adapts to distribution shape, compared directly to top-K.
KV Cache
Why the KV cache is valid, and how it turns quadratic generation cost into linear cost.
LLM Perplexity
The perplexity formula, a worked example, and what perplexity does and doesn't measure.
Generative vs Discriminative Models
The core distinction between learning P(y|x) and learning the data distribution P(x) itself.
Autoencoders
The encoder-bottleneck-decoder structure, why the bottleneck matters, and practical uses.
Denoising Autoencoders
How training on corrupted inputs with clean targets produces more robust learned features.
Sparse Autoencoders
How a sparsity penalty forces specialized, interpretable features even with a large latent space.
Variational Autoencoder
The VAE's probabilistic latent space, the reparameterization trick, and the full loss formula.
GAN
The GAN minimax formula, adversarial training loop, and why training is notoriously unstable.
DCGAN
The architectural guidelines that stabilized convolutional GAN training.
Conditional GAN
How conditioning both generator and discriminator on a label enables controlled generation.
StyleGAN
StyleGAN's mapping network and multi-resolution style injection for fine-grained generation control.
Why Diffusion Models
Why diffusion models offer more stable training than GANs, at the cost of generation speed.
Diffusion Forward Process
The fixed forward noising process formula, and the closed-form shortcut to any noise step.
Diffusion Reverse Process
The learned reverse denoising process and the full generation loop from pure noise to an image.
Noise Prediction
The simplified noise-prediction training objective diffusion models actually use in practice.
U-Net (Diffusion)
Why U-Net's skip connections and timestep conditioning suit it for diffusion noise prediction.
Diffusion Conditioning
How cross-attention and classifier-free guidance steer diffusion generation toward a text prompt.
Latent Diffusion
How running diffusion in a compressed VAE latent space makes generation computationally practical.
Stable Diffusion
The complete Stable Diffusion pipeline assembling text encoding, VAE and U-Net conditioning.
Text-to-Image Generation
The practical parameters shaping text-to-image output, prompt engineering, and reproducible generation.
Diffusion vs GAN
A complete side-by-side comparison of diffusion models and GANs, and when to choose each.
What Is Transfer Learning?
Why transfer learning works, based on the general-to-specific pattern in learned representations.
Pretrained Models
Where pretrained models come from, why they save so much compute and data, and how to choose one.
Feature Extraction
The frozen-backbone feature extraction approach, when it works best, and full PyTorch code.
Freezing Layers
The requires_grad mechanic behind freezing, and progressive unfreezing as a training strategy.
Fine-Tuning
How fine-tuning differs from feature extraction, and why it needs a much smaller learning rate.
Partial vs Full Fine-Tuning
A decision framework for choosing feature extraction, partial or full fine-tuning based on your data.
Domain Adaptation
Handling systematic distribution shift between source and target domains beyond standard fine-tuning.
Full Fine-Tuning
The real memory cost of full fine-tuning at LLM scale, and why it becomes prohibitive.
PEFT
The core idea behind Parameter-Efficient Fine-Tuning and why training under 1% of parameters can work.
LoRA
The LoRA formula, exact parameter savings math, and a full PyTorch implementation.
QLoRA
How QLoRA combines LoRA with 4-bit quantization to fine-tune huge models on one GPU.
Adapters
The adapter bottleneck module formula, and a direct comparison to LoRA including inference latency.
Prefix Tuning
How trainable virtual tokens at every attention layer steer a frozen model's behavior.
Prompt Tuning
The simplest PEFT method, and a direct comparison against prefix tuning.
Model Quantization
The quantization formula, a worked numerical example, and the precision/efficiency tradeoff.
4-Bit Quantization
NF4 and double quantization, the techniques that keep 4-bit quantization accurate.
8-Bit Quantization
The 8-bit quantization middle ground, and 8-bit optimizer states for training memory savings.
What Is Self-Supervised Learning?
The precise three-way distinction between supervised, unsupervised and self-supervised learning.
Pretext Tasks
What a pretext task is, common examples across vision and NLP, and why solving it well matters.
Contrastive Learning
The InfoNCE loss formula and the core positive/negative pair recipe behind contrastive learning.
SimCLR
The SimCLR pipeline, its projection head trick, and why it needs large batch sizes.
MoCo
How MoCo's momentum encoder and negative queue decouple contrastive learning from batch size.
Masked Language Modeling
MLM as a pretext task, and the 80/10/10 masking scheme that reduces train/fine-tune mismatch.
Masked Image Modeling
Why masked image modeling needs a much higher masking ratio than text, with MAE-style code.
Representation Learning
The overarching goal of self-supervised learning, and linear probing as the standard evaluation.
Few-Shot Learning
In-context learning vs prototypical networks, and how LLMs achieve few-shot learning with zero updates.
Zero-Shot Learning
How CLIP achieves zero-shot image classification, with full code and why it isn't magic.
Meta-Learning
The MAML algorithm's inner and outer loops, with formulas and a code sketch.
Knowledge Distillation
The distillation loss formula, why soft labels carry "dark knowledge," and full PyTorch code.
Continual Learning
The catastrophic forgetting problem and the EWC regularization technique to combat it.
Federated Learning
The FedAvg algorithm for training across decentralized, private data, with the aggregation formula.
Multimodal Learning
Early, late and intermediate fusion strategies for combining multiple data modalities.
Vision-Language Models
CLIP's contrastive training objective across images and text, with a full code example.
Mixture of Experts
The MoE routing formula and how sparse activation decouples capacity from compute cost.
Neural Architecture Search
The three components of NAS and the search strategies used to automate architecture design.
PyTorch Installation
Installing PyTorch with GPU support and verifying CUDA availability correctly.
PyTorch Tensors
Practical tensor creation, inspection, and the NumPy conversion memory-sharing gotcha.
Tensor Operations
Indexing, reshaping, view vs reshape, and the cat vs stack distinction, with code.
Broadcasting (PyTorch)
The classic silent shape-mismatch bug in real PyTorch code, with a worked debugging example.
Autograd
The practical autograd API — requires_grad, backward(), no_grad(), and gradient accumulation.
Computational Graphs (PyTorch)
How PyTorch builds dynamic graphs on every forward pass, and retain_graph explained.
nn.Module
The base class for all PyTorch models, parameter registration, and train/eval modes.
PyTorch Layers
A practical reference catalog of common PyTorch layer types with exact syntax.
PyTorch Activations
Module vs functional activation forms, with a full quick-reference table.
PyTorch Loss Functions
A practical reference for PyTorch loss classes and their exact input format requirements.
PyTorch Optimizers
A practical reference for PyTorch optimizers, including per-parameter-group learning rates.
PyTorch Dataset
The minimal Dataset interface — __len__ and __getitem__ — with a full working example.
PyTorch DataLoader
DataLoader parameters, batching, shuffling, and custom collate functions for variable-length data.
PyTorch Training Loop
The complete practical training loop with device management and memory-safe loss logging.
PyTorch Validation Loop
The complete practical validation loop with correctly weighted loss averaging.
PyTorch GPU/CUDA
Moving models and data to GPU correctly, device-mismatch errors, and mixed precision training.
Saving PyTorch Models
Practical syntax for saving weights, full checkpoints, and best-model tracking.
Loading PyTorch Models
Practical syntax for loading weights, checkpoints, and partial weights across devices.
PyTorch Transfer Learning
Complete, runnable PyTorch code for feature extraction and partial fine-tuning.
PyTorch Custom Datasets
A complete realistic custom Dataset loading images from disk with CSV labels.
PyTorch Custom Training Loops
When to write a custom training loop, with gradient accumulation and custom logging patterns.
TensorFlow Basics
Eager execution vs @tf.function graph compilation in modern TensorFlow.
TensorFlow Tensors
The tf.constant vs tf.Variable distinction and GradientTape for automatic differentiation.
Keras Overview
The three Keras model-building APIs previewed, and how Keras relates to TensorFlow.
Keras Sequential API
Building linear layer stacks with the Sequential API, including a full CNN example.
Keras Functional API
Building branching architectures, residual connections and multi-input models with the Functional API.
Keras Custom Models
Subclassing tf.keras.Model with call(), and the training argument for mode-dependent layers.
Keras Callbacks
Built-in callbacks for early stopping, checkpointing and learning rate scheduling.
Keras Training
The compile() and fit() high-level training API, contrasted with PyTorch's manual loop.
Keras Evaluation
The evaluate() and predict() methods, and the difference between them.
Saving/Loading Keras Models
Keras's full-model-save default compared to PyTorch's state_dict-focused approach.
PyTorch vs TensorFlow
A complete side-by-side comparison of PyTorch and TensorFlow/Keras with practical decision guidance.
Hyperparameters Overview
A complete catalog of every hyperparameter covered in this hub, and the general tuning philosophy.
Learning Rate Tuning
The learning rate range test technique and how to diagnose too-high vs too-low learning rates.
Batch Size Tuning
The linear scaling rule, memory constraints, and the generalization tradeoff of batch size.
Epochs Tuning
Why modern practice relies on early stopping rather than fixing epoch count manually.
Network Depth Tuning
A practical strategy for choosing network depth based on underfitting/overfitting symptoms.
Hidden Units Tuning
Common width patterns like the funnel shape, and the width/data-size relationship.
Dropout Tuning
Typical dropout rates, where to apply dropout, and diagnosing rate adjustments from symptoms.
Weight Decay Tuning
Typical weight decay ranges and why AdamW is essential for meaningful weight decay tuning.
Optimizer Selection
A practical decision guide for choosing between AdamW, SGD and RMSProp.
Activation Function Selection
A practical decision guide for choosing activation functions by layer type and architecture.
Grid Search
The exhaustive grid search algorithm, its combinatorial explosion problem, and code.
Random Search
Why random search often outperforms grid search for the same compute budget.
Bayesian Optimization
The surrogate model and acquisition function behind sample-efficient Bayesian optimization.
Optuna
The Optuna framework, its TPE search algorithm, and the compute savings from trial pruning.
DL Problem Definition
Defining success metrics and feasibility before any modeling begins, with a checklist.
Dataset Collection
Common data sources, rough volume guidance, and licensing/ethical considerations.
Data Exploration
The core exploration checklist for catching data issues before modeling.
Data Cleaning
Detecting corrupt files, duplicates and mislabeled examples, with practical code.
Data Preprocessing
The critical rule of fitting normalization statistics on training data only.
Data Augmentation Pipeline
Assembling a complete augmentation pipeline with correct operation order.
Train/Val/Test Split (Lifecycle)
Practical splitting code, stratification, and k-fold cross-validation for smaller datasets.
Model Selection
A practical decision framework for choosing architecture based on data size and constraints.
Model Training
The training stage checklist, including the valuable tiny-batch overfitting sanity check.
Model Evaluation
Choosing the right metrics for the task and comparing against baselines honestly.
Hyperparameter Tuning (Lifecycle)
When to tune in the project lifecycle, and how to budget compute across tuning stages.
Error Analysis
The core error analysis process for finding systematic patterns in model mistakes.
DL Model Saving (Lifecycle)
Building a complete model artifact bundle with config, preprocessing and metadata.
DL Deployment Lifecycle
A preview of the deployment stages, and batch vs real-time inference.
DL Monitoring
Why deployed models degrade silently, and a practical prediction-drift monitoring signal.
Model Serialization
What gets serialized, and why native PyTorch format is not always deployment-ready.
Pickle Models
How Python pickle works, and the real security risk of unpickling untrusted files.
TorchScript
Tracing vs scripting for converting PyTorch models to a Python-independent format.
ONNX
Exporting PyTorch models to a framework-agnostic format for portable inference.
FastAPI Model Serving
Wrapping a trained model in a real callable web API using FastAPI.
REST API Deployment
Production-grade API considerations: validation, error handling, and health checks.
Docker Deployment
Containerizing a model-serving application for reproducible deployment.
GPU Deployment
When GPU inference is worth it, and dynamic batching for efficient GPU utilization.
Cloud Deployment
Common cloud deployment patterns and the cold-start tradeoff in serverless.
AWS Deployment
Deploying models with SageMaker and other AWS services, with working code.
Batch Inference
Running inference on accumulated data on a schedule, and why it is often simpler.
Real-Time Inference
Serving individual requests immediately, and why percentile latency matters most.
Model Optimization (Deployment)
Quantization, pruning, distillation and compilation for faster production inference.
ML Pipelines
Formalizing the project lifecycle into automated, reproducible stages.
Experiment Tracking
Systematically recording every training run so results stay comparable.
MLflow
A practical open-source tool for experiment tracking, models, and the model registry.
Model Registry
Centralized management of model versions and deployment stages.
Data Versioning
Tracking exactly which dataset version was used for a given training run.
Model Versioning
Systematically tracking every trained model version and its lineage.
ML Monitoring
The layers of production ML monitoring, from data quality to system health.
Data Drift
Detecting shifts in input data distribution with a statistical test.
Concept Drift
When the relationship between inputs and correct outputs changes over time.
Model Drift
The observed performance decline that data and concept drift produce.
A/B Testing (ML)
Comparing a new candidate model against production on live traffic.
Inference Latency
Decomposing and profiling where prediction request time actually goes.
Inference Throughput
Maximizing requests handled per second, and its tradeoff with latency.
GPU Utilization
Diagnosing why a GPU sits idle and fixing common data loading bottlenecks.
Memory Optimization
Gradient checkpointing and mixed precision for fitting larger models in memory.
Distributed Training
Data and model parallelism for training across multiple GPUs, with code.
Embeddings (Modern AI)
How dense vector representations power modern search, retrieval and agent systems.
Vector Databases
Specialized storage for efficient approximate nearest neighbor search at scale.
RAG (Retrieval-Augmented Generation)
Combining retrieval with generation to ground LLM answers in real documents.
Multimodal AI
Models that jointly process and reason across text, images, audio and video.
Vision-Language Models
Connecting pretrained vision encoders and language models for image understanding.
AI Agents
LLM-based systems that plan, use tools and take multi-step actions toward a goal.
Tool Calling
The mechanism that lets a language model invoke external tools and APIs.
Function Calling
The structured JSON schema mechanism behind reliable LLM tool use.
Mixture of Experts (Modern AI)
Why sparse expert routing powers the largest modern language models.
Long-Context Models
Overcoming quadratic attention cost and the lost-in-the-middle problem.
Reasoning Models
Models trained to generate extended reasoning steps before a final answer.
Efficient Inference
Speculative decoding, continuous batching and KV-caching for large model serving.
Reading Research Papers
A practical multi-pass strategy for reading deep learning papers efficiently.
Literature Review
Systematically surveying existing research before contributing new work.
Baselines in Research
Why fair, strong baselines are essential for credible research comparisons.
SOTA Models
What state-of-the-art claims actually mean, and the benchmark saturation problem.
Benchmarking
What makes a good benchmark, and the risk of benchmark gaming.
Ablation Studies
Systematically isolating which components of a method actually matter.
Reproducibility in Research
Why deep learning has a reproducibility challenge, and practices that help.
Experimental Design
Core principles for planning trustworthy deep learning experiments.
Statistical Significance
Distinguishing genuine effects from random training noise, with code.
Model Complexity
How model expressiveness connects to the bias-variance tradeoff in research.
Model Parameters Count
How to compute parameter count, and why it is an imperfect complexity proxy.
FLOPs
Measuring actual computational cost as a complement to parameter count.
Inference Latency (Research)
How to measure and report inference latency rigorously in research papers.
Memory Requirements
Estimating training and inference memory footprint, with practical formulas.
Interview Questions
Index and prep strategy for the full Deep Learning interview question section.
DL Basic Interview Questions
Foundational Deep Learning interview questions with fully explained answers.
CNN Interview Questions
CNN interview questions covering convolution, pooling and key architectures.
RNN & LSTM Interview Questions
RNN and LSTM interview questions covering gating, BPTT and vanishing gradients.
Transformer Interview Questions
Transformer interview questions covering self-attention and architecture.
Optimization Interview Questions
Optimizer, learning rate and regularization interview questions explained.
PyTorch Interview Questions
PyTorch interview questions covering autograd and the training loop, with code.
LLM Interview Questions
LLM interview questions covering pretraining, fine-tuning, sampling and KV-cache.
Deployment Interview Questions
Deployment interview questions covering serving, security and production concerns.
Scenario-Based Interview Questions
Open-ended, situational Deep Learning interview scenarios, worked through.
Practice Questions
Index and usage guide for the Deep Learning practice problems section.
Practice: Neural Networks
Build a perceptron and MLP from scratch, with manual backpropagation.
Practice: CNN
Manual convolution, output shape math, and building a real CNN classifier.
Practice: RNN & LSTM
Implement an RNN cell manually and build an LSTM sentiment classifier.
Practice: Transformers
Implement scaled dot-product attention, masking and multi-head attention.
Practice: PyTorch
Custom Datasets, training loops, debugging exercises and custom losses.
Practice: Optimization
Implement gradient descent, momentum and Adam from scratch.
Practice: Model Evaluation
Compute metrics, ROC curves and cross-validation from scratch.
Projects
Index and approach guide for the twelve end-to-end Deep Learning projects.
Image Classification Project
Build a transfer-learning image classifier end to end, with full code.
Object Detection Project
Fine-tune a pretrained detector on a custom dataset, with full code.
Image Segmentation Project
Build a U-Net for pixel-level semantic segmentation, with full code.
Sentiment Analysis (LSTM) Project
Build an LSTM text classifier from raw text to a trained model.
Text Generation Project
Train a character-level language model and generate new text with it.
Chatbot Project
Build a document-grounded RAG chatbot with citations, with full code.
GAN Image Generation Project
Build and train a DCGAN to generate new images from noise.
Diffusion Image Generation Project
Implement a simplified DDPM diffusion model from first principles.
Transformer From Scratch Project
Build a mini-GPT decoder-only Transformer entirely from scratch.
Fine-Tuning an LLM Project
Fine-tune a pretrained LLM with LoRA on a custom instruction dataset.
Multimodal Project (Image Captioning)
Build a CNN encoder + LSTM decoder image captioning model.
Deployment Project
Deploy a trained model as a real, containerized REST API service.
Ready to go from notes to a real career?
Join CodingNow 2.0's Deep Learning course — live mentorship, hands-on projects, and 100% placement support in Delhi NCR.
Enroll Now — Free Demo AvailableDeep Learning Notes – FAQs
What students search before reading Deep Learning notes.
