Serving a PyTorch model on a Canadian GPU VPS

I’m deploying an image classification model for a Canadian client, and the inference server has to run in Canada. PIPEDA, BC public body rules and Quebec Law 25 all mean their uploaded images can’t leave the country. Here is the plan, and I’d like to hear what others would change:

  1. Model exported with torch.export and loaded once at startup, not per request.
  2. FastAPI app behind Gunicorn with Uvicorn workers, one worker per GPU.
  3. Dynamic batching with a short wait window so small requests are grouped.
  4. torch.compile tested against eager mode, keeping whichever is faster on our GPU.
  5. Half precision with torch.autocast, after checking accuracy on a validation set.
  6. Docker image pinned to exact PyTorch and CUDA versions, built from the official base image.
  7. Images deleted after inference, with only predictions and timing kept in logs on Canadian storage.

For anyone serving PyTorch models in production: did TorchServe or Triton beat a plain FastAPI setup for you, and how do you manage GPU memory with several models on one card?