With a serialized model ready, FastAPI is one of the most common ways to wrap it in a real, callable web API — a modern, fast, Python-native web framework well suited to serving model predictions.
A Minimal Model-Serving API
from fastapi import FastAPI
from pydantic import BaseModel
import torch
app = FastAPI()
model = MyModelClass()
model.load_state_dict(torch.load("model_weights.pt"))
model.eval()
class PredictionRequest(BaseModel):
features: list[float]
class PredictionResponse(BaseModel):
prediction: float
confidence: float
@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
x = torch.tensor([request.features], dtype=torch.float32)
with torch.no_grad():
output = model(x)
probs = torch.softmax(output, dim=1)
confidence, predicted_class = probs.max(dim=1)
return PredictionResponse(
prediction=predicted_class.item(),
confidence=confidence.item()
)Running this locally (uvicorn main:app) exposes a real HTTP endpoint — any client, in any programming language, can now send a POST request to /predict and receive the model's prediction back as JSON.
Why Pydantic Request/Response Models Matter
Defining explicit PredictionRequest/PredictionResponse classes (via Pydantic, which FastAPI uses natively) gives automatic input validation — a malformed request is rejected with a clear error before it ever reaches the model — plus automatic, always-accurate API documentation generated directly from these type definitions, a genuinely valuable practical benefit for any team consuming the API.
Handling Model Loading Efficiently
# Load the model ONCE at startup, not on every request
# (the code above already does this correctly, at module level)
# For heavier models, an explicit startup event is often clearer:
@app.on_event("startup")
def load_model():
global model
model = MyModelClass()
model.load_state_dict(torch.load("model_weights.pt"))
model.eval()Loading a model fresh on every incoming request would add substantial, unnecessary latency to every single call — the model should be loaded exactly once, when the service starts, and reused across all subsequent requests.
Common Mistakes
- Loading the model inside the request-handling function itself rather than once at startup — this adds significant, avoidable latency to every single prediction request.
- Forgetting
torch.no_grad()during inference in the serving endpoint — without it, PyTorch unnecessarily tracks gradients for every request, wasting memory and compute that serve no purpose at inference time.
Interview Relevance
Q: "Why should a model be loaded once at application startup rather than inside the request-handling function of a serving API?" Loading model weights from disk into memory takes real, non-trivial time — doing this on every incoming request would add that loading latency to every single prediction, dramatically slowing the API and wasting compute. Loading once at startup and keeping the model in memory for the service's lifetime means each individual request only pays the (much smaller) cost of the forward pass itself.
Practice Question
Why does explicitly defining Pydantic request and response models for a FastAPI endpoint provide value beyond just accepting raw JSON dictionaries?
