Model serving is the layer that turns a file on disk into something an application can call. It is a small idea with a surprising amount of operational surface, and most of that surface concerns change: how a model gets deployed, how it gets replaced, and how you find out that replacing it was a mistake.
What a model server does
A model server is a long lived process that loads a model into memory and exposes it over the network. Beyond that it typically handles request batching, concurrency limits, tokenization, streaming responses, health reporting and metrics.
The reason this is a distinct component rather than library code inside an application is startup cost. Loading a model takes time and a great deal of memory, so it happens once per process rather than once per request, and the process therefore has to outlive any individual caller. Everything else about the architecture follows from that constraint.
One model per process, usually
The simplest and most common arrangement is one model per server process, with as many processes as capacity allows. It is easy to reason about, isolates failures, and makes it obvious what a restart affects.
Multi model servers, which hold several models in one process and route internally, improve utilization when individual models are small or lightly used. The cost is a shared fate: a fault or a restart touches every model in the process, and memory pressure from one affects the others.
The sensible default is separation, moving to co-location only where measurement shows capacity is genuinely being wasted. Where several models are in use for different purposes, the routing decision usually belongs a layer up rather than inside a server, which is one of the arguments for an AI gateway.
Versioning is the whole discipline
A model is a dependency that changes, and an endpoint that cannot say which version answered a request is an endpoint that cannot be debugged.
Three practices carry most of the value. Address models by explicit version rather than by a moving label, so a deployment is a deliberate act. Record the version on every response, or at minimum in the request log, so a complaint about behavior can be tied to a build. And keep the previous version deployable, because the fastest fix for a regression is almost always to go back.
The last one is a capacity requirement as well as a process one. Running two versions briefly needs headroom, and an environment sized exactly to one version cannot roll one out safely. That headroom belongs in the plan from the start, and is one of the inputs in AI infrastructure capacity planning.
Deploying a new version without an outage
Model deployments are slower and riskier than application deployments because instances take minutes to become useful and because the change is behavioral rather than functional. A new version rarely fails loudly. It answers differently.
Shadow traffic, where the new version receives a copy of real requests and its responses are recorded but not returned, is the most useful technique available. It exercises the model on genuine input with no user impact, and it surfaces the performance profile before anyone depends on it.
Canary routing follows: a small share of live traffic, watched, then widened. Both require the routing layer to be able to split traffic by version, which is a reason to have one even in a single model environment.
What none of this replaces is evaluation. Infrastructure can tell you the new version is fast and healthy. Whether its answers are better is a different question, belonging to the model owners rather than to the platform, and the distinction is worth stating explicitly so nobody assumes a green deployment dashboard means a good model.
Health checking is harder than it looks
A model server can be listening on its port, reporting itself healthy, and unable to serve anything, because the device it depends on is in a bad state or the weights failed to load fully.
A useful readiness check therefore exercises the actual path: a small fixed inference request whose result is known, run against the loaded model, rather than a plain response on a health endpoint. It costs a little capacity and it distinguishes an instance that is alive from one that is working.
Separate liveness from readiness deliberately. An instance that is loading a model is not ready and should receive no traffic, but it is very much alive and restarting it produces an endless loop of cold starts. Getting this wrong is a common cause of an environment that never stabilizes after a deployment.
The endpoint is an API, with an API's problems
Once a model is exposed over the network it inherits the entire security surface of any other API, and the OWASP API Security Top 10 reads as a checklist for it.
Two entries are especially apt. API4:2023 Unrestricted Resource Consumption describes exactly the failure mode of an unbounded model endpoint, where a single expensive request or a flood of ordinary ones exhausts the capacity everyone shares. And API9:2023 Improper Inventory Management describes the state most organizations reach at their third model, where nobody can list the endpoints that exist or who is calling them.
API2:2023 Broken Authentication is the third to plan for, because an internal model endpoint reachable without authentication is an internal service anyone on the network can spend money on.
Where limits belong
Every serving environment needs limits, and the useful question is which layer enforces which.
- The model server enforces what protects itself: maximum concurrent requests, maximum input length, and a request timeout.
- The gateway enforces what protects everyone else: per caller rate limits, quotas and priority.
- The platform enforces what protects the node: memory and device limits on the container.
Limits set only at the gateway leave the server defenseless against anything that reaches it another way. Limits set only at the server produce a first come first served environment in which one caller can consume everything.
Observability that answers the real question
During an incident the question is almost always the same: is this saturation, a bad instance, or a bad model version. The telemetry has to be able to distinguish them.
That needs per instance metrics rather than only per endpoint aggregates, because a single sick instance hides inside an average. It needs latency as a distribution, because a mean conceals the tail users complain about. It needs the model version as a dimension on every metric, so a regression that started at a deployment is visible as such. And it needs queue depth, which is the signal that separates a system that is short of capacity from one that is broken.
Where the model artifact comes from
A serving instance is useless without weights, and how those weights arrive is an infrastructure decision with security consequences.
The artifact needs a home: a registry or object store that is backed up, access controlled and close enough to the compute that a cold start is not dominated by the transfer. Pulling a large model across a slow link on every scale out event turns a routine capacity change into an incident.
It also needs provenance. A model is executable content in the sense that matters: it determines what your system says to people. Recording where each artifact came from, checking its integrity before it is loaded, and keeping the set of places a model may be pulled from short are ordinary supply chain hygiene applied to a new kind of dependency.
Two failure modes are worth naming. A model pulled directly from a public source at deploy time makes an external service part of your startup path. And an artifact store with no retention policy quietly becomes the largest thing in the environment, because nobody deletes a model that something might still be running.
A minimal serving environment
Stripped to essentials, a production serving layer that behaves well has six properties. It runs more than one instance. Each instance reports readiness by doing real work rather than by answering a health URL. Traffic is routed by an addressable version. The previous version can be restored without a rebuild. Limits exist at both the server and the layer in front of it. And every metric carries the version as a dimension.
An environment with all six can be changed on a Tuesday afternoon. An environment missing the fourth cannot be changed safely at all, which in practice means it stops being changed, and an inference environment that nobody dares to update is one that slowly stops being useful.
Buying instead of building
All of the above exists in managed form. A hosted endpoint gives you serving, scaling, versioning and availability as somebody else's responsibility, and the trade is control of the data path and of when the model changes. The available shapes are compared in AI model hosting options for enterprise, and the wider layered picture in enterprise AI infrastructure architecture.
LABUSA designs and operates serving environments as part of its model serving and deployment services, including the rollout mechanics that make a model change reversible.
Sources and further reading
- OWASP API Security Top 10, 2023 edition. Community security guidance, not regulation.
- Kubernetes documentation, Horizontal Pod Autoscaling.