Inference is the part of AI that runs constantly. Training happens rarely, if at all, in most enterprises. Inference happens every time somebody asks a question, and it is the workload the infrastructure has to be built for.
It is also the workload most likely to be sized using assumptions borrowed from web serving, which do not transfer. The differences are specific and worth understanding before capacity is bought.
Inference against training
Training produces a model. Inference uses one. The distinction matters because the two have almost opposite infrastructure profiles.
Training is a batch job. It runs for hours or days, is measured in throughput, tolerates interruption if it checkpoints, and benefits from many devices working together with fast links between them. Nobody is waiting for it in a browser.
Inference is an interactive service. It runs in seconds, is measured in latency, cannot be interrupted without a user noticing, and is usually served by a single device holding one model. What it needs is memory and predictable response, not interconnect bandwidth.
Fine tuning sits between the two, and it is occasional. That combination makes it the classic candidate for rented capacity rather than owned, which is one of the arguments in do you need GPUs for enterprise AI.
What happens when a request arrives
The sequence explains most of the operational behavior.
- The model must already be loaded into memory. Loading is slow, so it happens at startup rather than per request.
- The request text is tokenized into the units the model works in.
- The model processes the input, which is fast because it can be done in parallel.
- The output is produced one token at a time, each depending on the last. This part is inherently sequential.
- Working state for the request is held in memory throughout, and released at the end.
Two facts in that list drive everything else. Startup is expensive, so instances are long lived. And output generation is sequential, so a long answer takes proportionally longer and cannot be parallelized away.
Why the latency numbers look strange
A single average response time is close to meaningless for an inference endpoint, because response time is dominated by how much text comes out.
Two measures are worth separating. Time to the first token is what a user experiences as responsiveness, and it is largely a function of queueing and input length. Total time is a function of output length, and a request producing a long answer will be slow no matter how much hardware is available.
This is why streaming exists. Returning tokens as they are produced changes the perceived latency without changing the total, and for interactive features it is usually the difference between usable and abandoned. It also changes the infrastructure: a streamed response holds a connection open for its whole duration, which affects load balancer timeouts and connection limits in ways a request and response service never exposed.
Batching, and the trade it makes
Serving one request at a time wastes an accelerator badly. Batching combines several concurrent requests so the device does useful work on all of them at once, and it is the single largest lever on throughput.
The trade is latency. A larger batch means better utilization and a longer wait for any individual request, because each waits for the group. Continuous batching, where new requests join a batch already in progress rather than waiting for a batch boundary, reduces that penalty considerably and is the behavior to look for in a serving framework.
The practical point for infrastructure planning is that throughput and latency are set by one dial, and the position of that dial is a product decision as much as a technical one. An internal batch summarization job and an interactive assistant should not share serving configuration.
Concurrency is the capacity unit
Memory holds the model and the working state of every in flight request. So the number of requests that can be served at once is bounded by whatever memory the model leaves behind, and it shrinks as requests carry more text.
That makes concurrent requests at a given context length the honest unit of capacity, rather than requests per second. Two deployments of the same model on the same hardware can differ several fold in what they can serve, purely because of how much context each request carries. The arithmetic is set out in AI infrastructure capacity planning, and the memory constraint itself in GPU infrastructure for enterprise AI.
Autoscaling does not work the way it does for web tiers
The reflex for a busy service is to add instances. Kubernetes describes the mechanism plainly: a HorizontalPodAutoscaler automatically updates a workload resource, and horizontal scaling means that the response to increased load is to deploy more pods.
For inference that reflex is sound and slow. A new instance must acquire a device, pull a container image that is frequently very large, load the model into memory and warm up. The interval between deciding to scale and serving traffic is minutes rather than seconds, which is longer than most demand spikes.
Three adjustments follow. Scale on a signal that leads demand, such as queue depth, rather than one that lags it, such as utilization. Keep warm headroom, accepting the idle cost as the price of absorbing bursts. And treat the queue as a real component with a policy for what happens when it is full, because at some load the honest answer to a user is to wait or to be refused, and the alternative is a system that fails for everybody at once.
Model loading is an infrastructure problem
The weights have to get onto the machine. For a large model that is a substantial transfer, and it happens on every cold start, every scale out event and every node replacement.
Where the artifact lives therefore matters. Pulling from a remote registry across a slow link turns a two minute start into a fifteen minute one, which shows up as an outage during an incident rather than as a slow deployment. Local caching, a registry close to the compute, and pre pulled images on nodes are ordinary answers.
The same applies to versioning. An inference environment should be able to state which model version each instance is serving, and to roll back. That belongs to AI model serving architecture.
What to monitor
Six signals cover most incidents.
- Queue depth and wait time. The leading indicator of saturation, and the one users feel first.
- Time to first token as a distribution, not a mean.
- Device memory occupancy, which decides how much concurrency remains.
- Device utilization, which is a different question from occupancy and often disagrees with it.
- Errors by class, separating out of memory from timeout from upstream failure, because they have different fixes.
- Cost per workload, attributed at the gateway.
Collect them before the first incident. An inference environment with only a request count and an average latency cannot answer the question that gets asked during an outage, which is whether the system is short of capacity or has a stuck instance.
Cold start is the failure mode nobody rehearses
Most inference outages are not the model failing. They are the environment losing an instance and taking several minutes to replace it, during which the remaining instances absorb the entire load and queue lengths climb until requests time out.
The pattern is worth rehearsing because it is self reinforcing. One instance fails, the survivors saturate, latency rises, users retry, the retries add load, and a second instance falls over. A web tier recovers from this in seconds because replacements start instantly. An inference tier does not.
Three mitigations, in order of usefulness. Run more instances than the minimum, so losing one is a degradation rather than a cliff. Bound the queue and shed load explicitly rather than letting every request slow down together. And make the client's retry behavior deliberate, because an aggressive automatic retry against a saturated endpoint is an attack on your own service.
Where this fits
Inference infrastructure is one layer of a larger environment, and it is rarely the layer that fails first. Identity, retrieval and the gateway sit above it, and ordinary compute and network sit below. The whole picture is in enterprise AI infrastructure architecture.
LABUSA designs and operates this layer as part of its AI infrastructure and deployment capability, including the measurement that establishes what the capacity actually needs to be.
Sources and further reading
- Kubernetes documentation, Horizontal Pod Autoscaling.
- NVIDIA, Multi-Process Service documentation. Cited for that vendor's own product behavior only.