FLOPs (floating-point operations) measure the actual computational cost of running a model — a complementary metric to parameter count that more directly reflects compute requirements and inference speed.
FLOPs vs Parameter Count — Why Both Are Reported
| Metric | What It Measures | Limitation |
|---|---|---|
| Parameter count | Total learnable capacity / model size | Doesn't directly indicate compute cost (especially for sparse/MoE architectures) |
| FLOPs | Actual computational operations required for one forward pass | Doesn't directly indicate memory requirements or capacity |
Together, these two metrics give a more complete picture than either alone — a model can be reported as, for example, "7B parameters, 15 GFLOPs per forward pass," giving readers a sense of both its scale/capacity and its actual runtime computational cost.
Computing FLOPs for Common Operations
| Operation | Approximate FLOPs |
|---|---|
| Fully-connected layer (matrix multiply) | \(2 \times \text{input\_dim} \times \text{output\_dim}\) (the factor of 2 accounts for both a multiply and an add per computation) |
| Convolutional layer | \(2 \times \text{kernel\_h} \times \text{kernel\_w} \times \text{in\_channels} \times \text{out\_channels} \times \text{output\_h} \times \text{output\_w}\) |
Code — Measuring a Model's FLOPs
from fvcore.nn import FlopCountAnalysis
import torch
model = MyModel()
dummy_input = torch.randn(1, 3, 224, 224)
flop_analysis = FlopCountAnalysis(model, dummy_input)
total_flops = flop_analysis.total()
print(f"Total FLOPs for one forward pass: {total_flops:,}")
print(f"That's {total_flops / 1e9:.2f} GFLOPs")Why FLOPs Matters Practically
FLOPs correlates much more directly with actual inference latency and energy cost than parameter count alone — two models with identical parameter counts but different architectures (e.g. different amounts of weight sharing or activation reuse) can have meaningfully different FLOPs, and therefore meaningfully different real-world inference speed, even at the same nominal "model size."
FLOPs Isn't the Whole Story Either
Real-world inference latency also depends on factors FLOPs doesn't capture directly — memory bandwidth (moving data can be as much of a bottleneck as computing on it), hardware-specific optimization (how well an operation maps to available hardware acceleration), and parallelization efficiency. This is exactly why the practical latency profiling covered in Inference Latency (Research) and memory requirements analysis remain necessary alongside theoretical FLOPs counting, not a full replacement for it.
Common Mistakes
- Assuming lower FLOPs always means faster real-world inference — memory bandwidth bottlenecks and hardware-specific efficiency can sometimes matter more than raw FLOPs count for actual measured latency.
- Reporting only parameter count without FLOPs (or vice versa) when comparing model efficiency — the two metrics capture genuinely different, complementary aspects of computational cost.
Interview Relevance
Q: "Why do research papers often report both parameter count and FLOPs, rather than just one or the other?" Parameter count reflects a model's total learnable capacity/size but doesn't directly indicate actual computational cost, particularly for architectures like Mixture of Experts where most parameters go unused per input. FLOPs directly measures the computational operations required per forward pass, correlating more closely with real inference speed and energy cost — but doesn't capture memory requirements or capacity on its own. Reporting both together gives a more complete picture of a model's efficiency profile than either metric alone would provide.
Practice Question
Why might two models with identical FLOPs still show meaningfully different measured inference latency on the same hardware?
