Most teams start with hosted AI APIs, and for many that's the right long-term choice. But data-residency rules, predictable high volume, or the need for custom models lead some organisations to self-host models. Kubernetes is the natural platform for that, with a few important differences from ordinary web apps.
Should you self-host at all?
Self-hosting makes sense when:
- data must not leave your infrastructure or country,
- you have steady, high volume, where GPU servers become cheaper than per-token pricing,
- you need open-weight or fine-tuned models for a specialised task,
- you need predictable latency without depending on a third party.
Otherwise, hosted APIs are simpler and usually give you access to stronger models. Many teams use a mix of both.
GPUs in Kubernetes
- Install the NVIDIA GPU Operator (or your cloud's equivalent) to provide drivers and the device plugin.
- Request GPUs explicitly:
resources.limits: nvidia.com/gpu: 1. - Put GPU nodes in a separate node pool with taints, so only AI workloads land on expensive hardware.
- Consider GPU sharing (time-slicing or MIG on supported GPUs) for small models that don't need a whole GPU.
Model serving
Don't write your own inference server. Use a purpose-built one:
- vLLM: high-throughput LLM serving with an OpenAI-compatible API, so existing client code works with minimal changes,
- Hugging Face TGI and NVIDIA Triton / NIM as alternatives,
- KServe for a Kubernetes-native layer on top, with autoscaling and canary rollouts for models.
containers:
- name: llm
image: vllm/vllm-openai:<pinned-version>
args: ["--model", "/models/my-model", "--max-model-len", "8192"]
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: model-weights
mountPath: /models
The big-file problem
Model weights can be many gigabytes. Baking them into the container image makes pulls painfully slow. Better options:
- store weights on a persistent volume or fast network storage and mount them,
- download from object storage in an init container with a node-local cache,
- pre-pull images and weights onto GPU nodes to reduce cold starts.
Autoscaling AI workloads
CPU usage is a poor signal for GPU inference. Scale on what matters, such as request queue length, concurrent requests or latency, using KEDA or custom metrics with the HPA. Remember that a new GPU node plus model load can take minutes, so keep a warm minimum for latency-sensitive services and use scale-to-zero only for batch or internal tools.
Put a gateway in front
A small LLM gateway service (often FastAPI) between your applications and the models is worth it. It handles:
- authentication and per-team quotas,
- routing between self-hosted and hosted models,
- logging of usage and cost,
- retries, timeouts and fallbacks.
Watch the costs
GPUs are the most expensive resource in most clusters. Track utilisation per GPU. An idle GPU costs the same as a busy one. Use spot GPUs for batch jobs, right-size model choices (a smaller model is often good enough), and switch off development GPU nodes outside working hours. See our Kubernetes cost optimisation guide for more.
Planning to host AI models in your own infrastructure? Our Cloud & DevOps team can design and run it with you.