Deploying a model that benefits from GPU acceleration in production introduces considerations beyond training-time GPU usage — cost, utilization efficiency, and the decision of whether GPU inference is even necessary at all.
When GPU Inference Is Actually Worth It
| Situation | GPU Recommended? |
|---|---|
| Large models (e.g. large Transformers, big CNNs) with high request volume | Yes — GPU parallelism significantly reduces per-request latency and increases throughput |
| Small models with low request volume | Often no — CPU inference may be fast enough, and a GPU instance's cost may not be justified |
| Batch inference with relaxed latency requirements | Depends on total volume and model size — sometimes CPU with more instances is more cost-effective |
This decision should be made deliberately, based on actual measured latency/cost tradeoffs, rather than defaulting to GPU simply because training used one — a model's inference computational profile can be quite different from its training profile, particularly at batch size 1.
GPU Utilization Efficiency — Not Wasting an Expensive Resource
# Batching multiple incoming requests together before running a GPU forward
# pass amortizes the GPU's fixed per-call overhead across more work,
# substantially improving throughput compared to one-request-at-a-time GPU calls
import asyncio
class BatchedInferenceServer:
def __init__(self, model, max_batch_size=32, max_wait_ms=10):
self.model = model
self.max_batch_size = max_batch_size
self.max_wait_ms = max_wait_ms
self.pending_requests = []
async def predict(self, x):
# Requests accumulate briefly, then run together as one batched
# GPU forward pass -- see Inference Throughput for the full pattern
...This dynamic batching pattern, covered in full in Inference Throughput, is one of the most impactful techniques for using GPU resources efficiently in a serving environment with many small, concurrent requests.
Multi-GPU and Model Placement for Serving
For very large models that don't fit on a single GPU's memory, or for scaling throughput across many concurrent requests, multiple model replicas (each on its own GPU) behind a load balancer is a common serving pattern — distinct from the multi-GPU training strategies in Distributed Training, though some of the same underlying hardware/networking considerations apply.
Common Mistakes
- Provisioning GPU inference infrastructure by default without measuring whether CPU inference would actually be fast and cheap enough for the actual traffic pattern — GPU instances are typically far more expensive, and this cost should be justified by an actual measured need.
- Running GPU inference one request at a time without any batching — this leaves substantial GPU throughput capacity unused, since much of a GPU's advantage comes from parallelizing across a batch of work, not from processing single small requests quickly.
Interview Relevance
Q: "Why might dynamically batching incoming inference requests together significantly improve GPU serving efficiency, compared to processing each request individually as it arrives?" A GPU's core advantage is parallel computation across a batch — processing requests one at a time under-utilizes this parallelism and pays the GPU's fixed per-call overhead (kernel launch, memory transfer) repeatedly for small amounts of work each time. Accumulating several requests into one batch before running a single forward pass amortizes that fixed overhead across more work, substantially improving overall throughput, at the cost of a small added latency while requests accumulate.
Practice Question
A small model serving a low-traffic internal tool currently runs on an expensive GPU instance. What would you check before recommending a switch to CPU inference?
