Kubernetes is the default answer to running containers at scale, and AI workloads inherit that default whether or not it suits them. Often it does. Sometimes it is a great deal of machinery in front of two servers that would have been fine with neither.
This is what the orchestrator actually provides for AI, the places its model fits awkwardly, and how to tell which situation you are in.
What it gives you
Four things, none of them AI specific and all of them relevant.
- Placement. Deciding which node has capacity for a workload, which becomes hard the moment nodes are not identical, and accelerator nodes never are.
- Reconciliation. A declared desired state that the system works to maintain, so a lost instance is replaced without anybody being paged.
- A rollout mechanism. Versioned deployments with a defined way forward and back, which model serving needs more than most workloads.
- A common substrate. One identity integration, one secrets mechanism, one network policy model and one monitoring stack across everything.
The fourth is usually the strongest argument in an enterprise. If the organization already runs Kubernetes, putting AI workloads somewhere else means a second set of everything.
Accelerators are not a native resource
The scheduler understands CPU and memory. It does not understand devices until something tells it to. Kubernetes handles this through an extension point: its device plugin framework exists to advertise system hardware, and the documentation describes device plugins as letting you configure your cluster with support for devices or resources that require vendor specific setup, such as GPUs, NICs, FPGAs.
In practice an administrator installs the vendor's driver on the node and runs the vendor's plugin, usually as a daemon set. After that, as the Kubernetes documentation puts it, your cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu, and containers request it the way they request cpu or memory.
The constraint that surprises people
Device resources do not behave like CPU, and the difference is documented rather than incidental. Kubernetes states that GPUs are only supposed to be specified in the limits section, which means you may specify a limit without a request, or both if they are equal, but you cannot specify a request without a limit.
The consequence is that the request and limit gap that lets CPU be oversubscribed does not exist here. A device is allocated whole to a container. Two pods cannot be scheduled onto one device by asking for half each, and a cluster with plenty of half idle devices will still refuse to schedule a pod that needs one.
Sharing therefore has to be arranged deliberately, by time slicing, process level sharing or hardware partitioning, each with different isolation properties. Those mechanisms are covered in GPU infrastructure for enterprise AI. The point for cluster design is that utilization is a configuration decision made in advance, not something the scheduler will optimize for you.
Autoscaling helps less than expected
Kubernetes describes the mechanism clearly: a HorizontalPodAutoscaler automatically updates a workload resource, and horizontal scaling means that the response to increased load is to deploy more pods.
For a stateless web service this is close to instantaneous. For inference it is not. A new pod must be scheduled onto a node with a free device, pull a large image, load model weights into memory and warm up. Minutes, not seconds, and a demand spike is usually over before the response arrives.
Three adjustments make it useful rather than decorative. Scale on a leading signal such as queue depth rather than a lagging one such as utilization. Keep a warm floor of instances so bursts are absorbed rather than waited out. And separate the two scaling problems: adding pods is fast if a node has a free device, and adding nodes is slow, so the cluster autoscaler and the pod autoscaler have very different response times and should be tuned as such.
Scheduling policy is where the design lives
Most of the real Kubernetes work for AI is telling the scheduler about a fleet that is no longer uniform.
- Node pools. Accelerator nodes belong in their own pool, with taints so that ordinary workloads do not land on expensive hardware.
- Priority. Interactive serving should preempt batch work. Without priority classes, an overnight indexing job can hold the capacity a user is waiting on.
- Disruption budgets. A model serving pod takes minutes to replace, so a node drain that evicts several at once is an outage. Budgets make the platform drain politely.
- Topology. Where instances must be spread across failure domains, say so explicitly, because the default is to pack them.
These are unglamorous settings and they are the difference between a cluster that behaves during a node upgrade and one that does not.
What Kubernetes does not solve
It does not make the workload secure. Containers share a kernel, and NIST is explicit in SP 800-190 that they do not offer as clear and concrete of a security boundary as a VM. An orchestrator schedules those containers; it does not change what they are. Workloads of genuinely different sensitivity need separation the kernel does not share, plus deliberate network policy.
It does not size anything. The scheduler places what you declare, and a pod that declares too little memory will be killed under load while the cluster reports itself healthy.
It does not manage models. Versioning, rollout and rollback of a model are application concerns that sit on top, described in AI model serving architecture.
When not to use it
Three situations where the orchestrator is more cost than benefit.
One model, one machine, one team. A single dedicated server running one endpoint does not need a cluster, and adding one adds an entire operational discipline to maintain.
No existing Kubernetes practice. Adopting it for an AI project means learning it during that project. The failure modes are unfamiliar, and an organization debugging both a new platform and a new workload at once is debugging neither well.
A managed endpoint would do. If the workload could run on a hosted model service, an entire cluster is a large answer to a question somebody else has already answered. The shapes available are in AI model hosting options for enterprise.
The honest test: if the orchestrator is removed and the workload still runs, it was optional. That is a perfectly good outcome to discover early.
Storage and networking inside the cluster
Two cluster subsystems need attention that a web workload never demanded of them.
Storage. Model weights are large, read constantly and written rarely, and every replica wants the same bytes. A read only shared volume suits that shape well; a per pod persistent volume does not, because it multiplies the storage and the transfer by the replica count. Node local scratch space is worth providing separately for caches, where losing the data on reschedule costs time rather than correctness.
Networking. Streamed responses hold a connection open for the whole generation, which is far longer than a typical request. Ingress controllers, service meshes and load balancers configured with a sixty second timeout will cut long answers off mid sentence, and the symptom presents as a model problem rather than a proxy problem. Check the timeout at every hop, and prefer least outstanding request balancing over round robin, because inference request costs vary so widely that round robin distributes work very unevenly.
Operating it day to day
The routine events that go badly are node upgrades and image rollouts, both because instances are slow to replace.
Drain a node holding serving pods and the replacements take minutes to become ready, so the remaining capacity absorbs the load in the meantime. With a disruption budget and enough headroom this is a small degradation. Without them it is an outage, and it happens on a schedule somebody else set.
The practical discipline is to treat any operation that evicts a serving pod as a capacity event, plan it for a quiet period, and confirm the replacement is genuinely ready rather than merely running, using the real readiness probe described in AI inference infrastructure.
A workable starting configuration
For an organization that does have a Kubernetes practice and does need to self host, a defensible first environment looks like this. A dedicated accelerator node pool, tainted. The vendor device plugin as a daemon set with a recorded driver version. Model weights on a read only shared mount rather than in the image. A warm floor of serving replicas with a queue depth based autoscaler above it. Priority classes separating interactive from batch. A pod disruption budget that permits one instance down at a time. And per instance metrics carrying the model version.
LABUSA builds exactly this, and operates it afterwards, as part of its AI infrastructure engineering work.