While Inference Latency measures how long a single request takes, inference throughput measures how many requests a serving system can handle per unit time — a related but distinct concern, and the two are often in direct tension.
The Latency-Throughput Tradeoff
Batching multiple requests together before a single forward pass (as introduced in GPU Deployment) improves throughput by better utilizing hardware parallelism, but adds latency to individual requests that must wait briefly for a batch to accumulate. Maximizing throughput and minimizing latency generally pull in opposite directions — the right balance depends entirely on the application's actual latency tolerance.
Code — Dynamic Batching for Throughput
import asyncio
import torch
class DynamicBatcher:
def __init__(self, model, max_batch_size=32, max_wait_ms=10):
self.model = model
self.max_batch_size = max_batch_size
self.max_wait_ms = max_wait_ms
self.queue = asyncio.Queue()
async def predict(self, x):
future = asyncio.Future()
await self.queue.put((x, future))
return await future
async def batch_loop(self):
while True:
batch, futures = [], []
deadline = asyncio.get_event_loop().time() + self.max_wait_ms / 1000
while len(batch) < self.max_batch_size and asyncio.get_event_loop().time() < deadline:
try:
x, fut = await asyncio.wait_for(self.queue.get(), timeout=0.001)
batch.append(x); futures.append(fut)
except asyncio.TimeoutError:
continue
if batch:
inputs = torch.stack(batch)
with torch.no_grad():
outputs = self.model(inputs)
for fut, out in zip(futures, outputs):
fut.set_result(out)Incoming requests accumulate briefly (bounded by max_wait_ms) into a batch, then run together as one GPU forward pass — significantly better throughput than one-at-a-time processing, at the cost of a small, bounded added latency per request.
Measuring Throughput
import time
def measure_throughput(model, batch_generator, duration_seconds=30):
start = time.time()
total_processed = 0
while time.time() - start < duration_seconds:
batch = next(batch_generator)
with torch.no_grad():
_ = model(batch)
total_processed += batch.shape[0]
elapsed = time.time() - start
print(f"Throughput: {total_processed / elapsed:.1f} requests/second")Choosing the Right Balance for the Application
| Application Type | Priority |
|---|---|
| A live conversational assistant | Latency — users notice and are bothered by delay directly |
| High-volume batch scoring, or backend systems with looser latency tolerance | Throughput — total processing capacity and cost efficiency matter more than any single request's speed |
Common Mistakes
- Maximizing batch size for throughput without regard for the resulting added latency, in a use case where users are actually latency-sensitive — optimizing the wrong metric for the actual application.
- Measuring throughput using unrealistically small or synthetic batches that don't reflect actual production traffic patterns, producing a misleading capacity estimate.
Interview Relevance
Q: "Explain the tradeoff between inference latency and inference throughput, and how batching affects each." Batching multiple requests together before a single forward pass improves throughput by better utilizing hardware parallelism and amortizing fixed per-call overhead across more work — but it requires individual requests to wait briefly while a batch accumulates, adding latency to each one. This makes latency and throughput a direct tradeoff, tuned via parameters like maximum batch size and maximum wait time: favoring larger batches and longer wait windows increases throughput at the cost of higher per-request latency, and the right balance depends entirely on the specific application's actual latency tolerance.
Practice Question
Why would a batch scoring system that processes millions of records overnight likely prioritize throughput over minimizing individual-request latency?
