Real-Time Inference

In contrast to Batch Inference, real-time inference responds to individual requests immediately as they arrive — required whenever a live system or user is directly waiting on the result.

When Real-Time Inference Is Required

The Core Real-Time Serving Challenge: Latency

Unlike batch inference, real-time serving is judged primarily by per-request latency, often under strict requirements (milliseconds to low seconds, depending on the application). This shapes many downstream decisions covered elsewhere in this category: whether GPU inference is justified (GPU Deployment), whether model optimization techniques are needed (Model Optimization), and how aggressively requests can be batched together without violating latency requirements.

Code — Measuring Real-Time Serving Latency

import time
import torch

model.eval()

def measure_latency(model, input_tensor, num_trials=100):
    # Warm-up runs -- excluded from timing, since the first few calls
    # often include one-time initialization overhead (e.g. CUDA kernel compilation)
    with torch.no_grad():
        for _ in range(10):
            _ = model(input_tensor)

    latencies = []
    with torch.no_grad():
        for _ in range(num_trials):
            start = time.perf_counter()
            _ = model(input_tensor)
            torch.cuda.synchronize()   # ensures GPU work actually finished before timing stops
            latencies.append((time.perf_counter() - start) * 1000)   # milliseconds

    latencies.sort()
    p50 = latencies[len(latencies) // 2]
    p99 = latencies[int(len(latencies) * 0.99)]
    print(f"p50: {p50:.2f}ms, p99: {p99:.2f}ms")

Reporting percentile latencies (p50, p99), not just an average, matters — an average can look perfectly acceptable while a meaningful fraction of requests (the tail, captured by p99) experience much worse latency, which is often what actually determines whether users perceive the service as reliably fast.

Micro-Batching Under Latency Constraints

Real-time serving can still batch multiple concurrent requests together (as covered in GPU Deployment) for better throughput, but only with a small, carefully bounded wait window (e.g. a few milliseconds) — large enough to gather a useful batch, small enough that it doesn't itself violate the latency requirement.

Common Mistakes

• Reporting only average latency, hiding a problematic tail of slow requests that a percentile-based metric (p95, p99) would reveal clearly.
• Excluding GPU synchronization from latency measurement — without torch.cuda.synchronize(), timing can stop before the GPU has actually finished its asynchronous work, producing an artificially low, incorrect measured latency.

Interview Relevance

Q: "Why is p99 latency often a more important metric than average latency for a real-time model-serving system?" Average latency can look perfectly acceptable even while a meaningful fraction of requests experience much worse performance — p99 latency directly reveals how bad the slowest 1% of requests are, which is often what actually determines whether real users perceive the service as reliably fast, since even a small fraction of slow requests can significantly harm user experience or violate service-level agreements.

Practice Question

Why is it important to run several "warm-up" inference calls before starting to measure a model's serving latency?

Related DL Notes

• Batch Inference
• Inference Latency
• Model Optimization (Deployment)