A reference architecture is useful for one reason: it gives a group of people who own different parts of a system a shared picture to argue over. It is not a design. Any organization that adopts one unchanged has skipped the work that makes it correct for them.
What follows is the layered model LABUSA uses as a starting point for enterprise AI environments, what each layer is responsible for, and the points where real designs diverge from it.
The request path
Read from the top, because that is the order a request travels.
- Users and applications issue a request, from a chat interface, an existing business system, or another service.
- Identity and access establishes who is asking, against the organization's own identity provider.
- The AI gateway authenticates the call, decides which model should serve it, applies limits and policy, and records that it happened.
- Model serving holds a model in memory and produces a response.
- Retrieval finds relevant organizational content before the model answers, filtered by what the asking user is allowed to see.
- Vector and data stores hold embeddings, source documents and metadata.
- Enterprise data sources are the systems of record the content came from in the first place.
- Compute, storage and network carry all of it.
Cutting across every layer: security, logging, monitoring, backup, automation and governance. Those are not a layer at the bottom. They are properties every layer has to have, and drawing them as a box underneath is the most common way a diagram lies about the work.
Identity comes first, not last
The most consequential architectural decision is usually made early and quietly: whether the AI system authenticates users individually, or whether it holds a service credential and acts on everyone's behalf.
The second is far easier to build and it collapses your permission model. A retrieval layer holding one set of standing credentials can see everything those credentials can see, and any user who can reach the assistant inherits that view unless something else stops them. The something else is usually a filter applied after retrieval, which is exactly the wrong place, because by then the content has already been selected and summarized.
The principle is not specific to AI. NIST states it plainly in Zero Trust Architecture: there is no implicit trust granted to assets or user accounts based solely on their physical or network location. Zero trust focuses on protecting resources, not network segments. A model endpoint is a resource.
The gateway is where policy becomes real
A gateway looks optional in a small deployment and becomes indispensable at the second model. It is the single place where authentication, routing, rate limiting, logging and cost attribution can be applied consistently, rather than reimplemented in each application that calls a model.
It is also the answer to a question that arrives later than it should: which applications are calling which models, and who is paying. The layer and its trade offs are covered in what is an AI gateway.
Serving is stateful in a way web tiers are not
The serving layer is where AI architecture stops resembling web architecture. A model must be loaded into memory to answer, loading is slow, and memory is the scarce resource rather than CPU time.
Three consequences follow. Scaling out is not instant, so bursts must be absorbed by queueing or by capacity held in reserve. Instances are expensive to keep warm, so idle capacity is a real cost rather than a rounding error. And a rolling model upgrade needs enough headroom to run two versions briefly, which is a capacity decision made at design time. AI model serving architecture covers the layer in detail.
Retrieval is an infrastructure component, not a feature
Most enterprise AI value comes from answering questions about the organization's own material, which means a search happens before the model is called. Architecturally that introduces a second data path with its own latency budget, its own failure modes and its own access control problem.
The infrastructure questions are ordinary ones. Where do embeddings live, how are they backed up, what happens when the source document changes, how is the store replicated, and how does a query carry the user's identity. The design of secure retrieval itself belongs to Private & Secure Enterprise AI; where the components run and how they are operated is this layer.
Where designs legitimately diverge
Four decisions produce most of the variation between real architectures, and none has a universally right answer.
Hosted endpoint or self hosted model
A hosted endpoint removes the serving layer entirely and replaces it with a contract. Self hosting gives control of the data path and adds accelerators, capacity planning and an operations burden.
One model or several
Routing different workloads to different models is efficient and adds a routing decision, a fallback policy and a testing surface. That is a reason gateways appear.
Where retrieval sits
Retrieval can live beside the model or beside the data. Putting it beside the data usually keeps sensitive content closer to home and adds network latency to every request.
How much runs on shared infrastructure
An AI workload can live on the platform the organization already runs, or on dedicated capacity. Sharing is cheaper and couples the failure domains. Dedicated capacity is cleaner and more of it sits idle.
Cross cutting concerns that decide whether it survives
The layers above describe a system that works on a good day. These decide what happens otherwise.
- Segmentation. Which components can reach which, and what has no route to the internet at all. See AI network segmentation and security.
- Secrets. Model provider keys, database credentials and service identities, held somewhere better than an environment variable in a container image.
- Observability. Utilization, latency distribution, error rates, queue depth and cost per workload, collected before you need them.
- Resilience. What is redundant, what degrades gracefully, and what simply stops. Covered in high availability for AI systems.
- Recovery. Which components are restored from backup and which are rebuilt from configuration, addressed in AI backup and disaster recovery.
What a layer diagram never shows
Three things decide whether an architecture survives its first year, and none of them appears on any diagram.
Who operates each layer. A boundary on a diagram is often a boundary between teams, and the layers that fail first are the ones where the answer is ambiguous. Retrieval is the usual casualty: the platform team considers it application logic, the application team considers it infrastructure, and nobody owns the index when it goes stale.
What changes without your involvement. A self hosted model changes when you change it. A hosted endpoint changes when the provider decides, and behavior you tested and signed off can move underneath a system that has not been redeployed. An architecture that depends on a model behaving as it did in testing needs somewhere to pin a version, or an evaluation that runs on a schedule and tells you when something moved.
Where the money goes. Cost in an AI environment attaches to a unit of work rather than to a server, and it is invisible unless the gateway attributes it. An architecture with no cost attribution produces a bill nobody can decompose, which is how a useful pilot becomes an argument.
How to use this model
The model above is a starting point for a conversation between people who own different parts of the estate. It is deliberately not a product list, and it is deliberately not prescriptive about the four decisions in the previous section, because those are where the organization's own obligations do the deciding.
Draw your own version, with your actual systems named in it, and mark three things on it: where organizational data crosses a boundary, where a credential is used, and which components you would have to rebuild rather than restore. Most architecture problems in AI deployments are visible on that annotated diagram before any code is written.
Then check it against something external. The joint guidance in Deploying AI Systems Securely is written for organizations deploying externally developed AI, and aims at improving the confidentiality, integrity, and availability of AI systems. Those three words are a serviceable checklist for a diagram.
LABUSA produces exactly this artifact as a deliverable in LABUSA AI infrastructure architecture services, alongside the capacity and resilience numbers that make it a plan rather than a picture.
Sources and further reading
- NIST SP 800-207, Zero Trust Architecture, August 2020.
- NSA, CISA and international partners, Deploying AI Systems Securely, April 2024.
- NIST, AI Risk Management Framework, intended for voluntary use.