An organization that has decided to run models itself now owns an unfamiliar class of infrastructure. Accelerators behave differently from the servers around them in ways that matter operationally, and most of the surprises are not about speed.
This is the infrastructure view: what constrains capacity, how devices are shared, what utilization means, and what owning them commits an operations team to. Whether you need them at all is a separate question, covered in do you need GPUs for enterprise AI.
Memory is the constraint, not arithmetic
The first correction most teams make is to stop thinking about accelerators in terms of speed and start thinking about them in terms of capacity.
To serve a model, the parameters must be resident in device memory. So must the working state of every request currently in flight, and that state grows with the amount of text each request carries. A device that comfortably serves four concurrent conversations may fail on the twelfth, not because it became slower but because it ran out of room.
Three practical consequences follow. The largest model you can serve is bounded by memory rather than by throughput. Concurrency is bounded by whatever memory the model leaves behind. And long context requests are disproportionately expensive, because their working state is larger for the whole time they are being served.
This is why capacity questions phrased as requests per second are usually unanswerable. The honest unit is concurrent requests at a given context length, which is what AI infrastructure capacity planning works in.
Inference and training are different infrastructure
The two workloads pull hardware design in opposite directions and it is worth being explicit, because vendor material often addresses the more expensive one.
Training is throughput bound. It runs long, benefits from many devices cooperating, and therefore depends heavily on the interconnect between them. Inference is latency bound. It runs briefly, is usually served by a single device holding one model, and depends far more on memory capacity than on device to device bandwidth.
An enterprise serving models it did not train needs an inference environment. Buying a training shaped cluster for an inference workload produces expensive interconnect that nothing uses.
Sharing a device between workloads
Assuming one workload equals one device is the fastest way to build an environment that sits idle. Several mechanisms allow a single accelerator to serve more than one process, and they differ in how strongly the tenants are isolated.
Time slicing lets processes take turns. It is simple and it means a heavy neighbor can make a light one slow, because there is no reservation.
Process level sharing keeps several clients resident at once. NVIDIA's Multi-Process Service is documented as providing reduced GPU context switching, and states that the MPS server allocates one copy of GPU storage and scheduling resources shared by all its clients, rather than each process allocating separately. The benefit is better utilization; the consequence is that clients share a fate.
Hardware partitioning divides a physical device into smaller isolated instances with their own memory. This gives the firmest boundary between tenants and the least flexibility, because the partitioning is decided in advance.
The choice is a real architectural decision. Isolation matters most when the tenants are different teams or different sensitivity classes; utilization matters most when the workload is one team's and bursty. It is worth deciding deliberately rather than accepting whatever the platform defaults to.
How the orchestrator sees a device
If the environment is containerized, accelerators are not visible to the scheduler by default. Kubernetes exposes them through an extension mechanism: its device plugin framework exists to advertise system hardware such as GPUs, NICs, FPGAs to the cluster.
Once a vendor's plugin is installed, the cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu, and pods request it the way they request CPU or memory. With one important difference documented by the project: GPUs are only supposed to be specified in the limits section, so the request and limit split that lets CPU be oversubscribed does not apply. You cannot schedule two pods onto one device by asking politely; sharing has to be arranged by one of the mechanisms above.
The operational consequence is that a node either has capacity for a workload or it does not, and a cluster full of half used devices will still refuse to schedule. That makes accurate right sizing of pod requests unusually important. The wider orchestration picture is in Kubernetes for AI workloads.
Utilization is the number that justifies the spend
Accelerator capacity is expensive enough that idle time is the dominant risk to the business case, and it is invisible unless somebody instruments it.
Measure device utilization and memory occupancy separately. They diverge constantly: a device can be almost entirely full of a resident model while doing very little arithmetic, which looks healthy on one graph and wasteful on the other. Both matter, and the second is usually what decides whether another workload can be placed there.
Attribute usage to a workload and a team. Shared capacity with no attribution becomes a queue with no owner, and the loudest team wins.
Power, cooling and the facilities conversation
Accelerator servers draw substantially more power per unit than the general purpose servers most enterprise racks were provisioned for, and they turn nearly all of it into heat. Two failure modes follow, and both are discovered late.
The first is rack level power. A rack rated for a familiar server density may support only a few accelerator nodes, which means capacity is bounded by the facility rather than by budget.
The second is thermal. Sustained inference is a sustained load rather than a bursty one, and equipment that stays within limits during a benchmark can throttle during a working week. Throttling shows up as latency variation that no application change explains.
Neither is a reason not to proceed. Both are reasons to involve facilities before the purchase order rather than after delivery, and they are among the strongest practical arguments for hosted or colocated capacity, which for public agencies can be bought through TIPS 260302 Data Center.
Lifecycle, drivers and the things that age
An accelerator estate carries a maintenance stream that ordinary servers do not.
- Drivers and firmware have their own compatibility matrix against the kernel, the container runtime and the serving framework. Upgrades are coordinated events rather than routine patches.
- Serving frameworks move quickly, and a version that supports a new model may require a newer driver.
- The hardware itself ages against a fast moving field. A three year old device is still working and may no longer be the economical way to serve the model you now want.
- Spares. Lead times for replacement are long enough that a failed device in a small estate is a capacity event, not a maintenance ticket.
Plan the refresh cycle at purchase, and size for the demand you expect at the point the next purchase would land rather than for today.
Buying, renting and colocating
Owning accelerators does not require owning a building, and the three options carry different risks rather than different amounts of the same risk.
On premises gives the shortest path between the model and the data, and makes power, cooling, physical security and hardware failure your problem. It is the right answer when the data genuinely cannot move and the organization already runs a facility.
Colocation keeps the hardware yours and makes the facility somebody else's. Power density and cooling become a contractual specification rather than a construction project, which removes the single most common physical obstacle. Remote hands and lead times become the operational questions instead.
Reserved cloud capacity keeps the devices dedicated to you without owning any, at a price that reflects the convenience. It suits an organization that needs isolation and predictable performance but not physical possession, and it converts a capital decision into a commitment term.
The comparison is not simply cost. It is about which risk the organization is better placed to carry: a capital commitment against uncertain demand, or a dependency on somebody else's supply.
What good looks like
An accelerator environment is being run well when four things are true: somebody can state current utilization and memory occupancy per device from a dashboard, capacity is attributed to workloads by name, the driver and framework versions are recorded and deliberately chosen, and there is a written answer to what happens when one device fails.
Most environments that struggle are missing the first. LABUSA designs, builds and operates this layer as part of AI infrastructure services, including the measurement that tells you whether the capacity was the right size.
Sources and further reading
- Kubernetes documentation, Device Plugins.
- Kubernetes documentation, Schedule GPUs.
- NVIDIA, Multi-Process Service documentation. Cited for that vendor's own product behavior only.