An AI environment introduces a set of components that talk to each other constantly, hold or reach sensitive content, and are frequently deployed by people whose main concern is getting the model to answer. The network design is usually decided by default, and defaults are permissive.
This is the infrastructure view of that problem: which paths should exist, which should not, and how to make the difference enforceable rather than documented. It is not a general cybersecurity page. The wider practice is Security, and the risk framing for AI belongs with AI Governance.
Start from what should be able to reach what
The useful exercise is to list the components and then draw only the arrows that have to exist. In most environments the list is short.
- Users reach the application. Nothing else should.
- The application reaches the gateway.
- The gateway reaches model serving, and only the gateway should.
- The retrieval service reaches the vector store, and only it should.
- The retrieval pipeline reaches the systems of record, usually on a schedule rather than per request.
- Everything reaches identity and observability.
What is missing from that list is instructive. Model serving does not need to reach the internet. The vector store does not need to reach anything at all; it is reached. The application does not need direct access to model serving if a gateway exists. Each absence is a rule worth enforcing, because each is a path an attacker or an accident would otherwise use.
Location is not trust
The traditional design places the AI components inside a trusted network and treats anything already there as authorized. NIST's Zero Trust Architecture is direct about why that fails: zero trust assumes there is no implicit trust granted to assets or user accounts based solely on their physical or network location.
The same document makes the point that matters most for this workload, that zero trust focuses on protecting resources, not network segments, as the network location is no longer seen as the prime component to the security posture of the resource.
Applied here, the model endpoint authenticates its callers even though they are internal, the vector store authenticates the retrieval service even though they share a subnet, and being inside the environment gets a component nothing by itself. Segmentation is still worth doing, as a way to reduce what an incident can reach. It is not a substitute for authentication at each hop.
Egress is the control most often missing
Most attention goes to who can reach the AI system. The more valuable control in practice is what the AI system can reach.
A model serving process that can open arbitrary outbound connections is a component with access to your data and a route off the network. So is a retrieval pipeline. Neither needs general internet access to do its job: serving needs the artifact store, retrieval needs the sources and the index, and both need identity and telemetry.
Default deny on egress, with an explicit allowlist of destinations, is the single highest value network control in an AI environment. It is also the one most likely to be resisted during a build, because it breaks the convenient habit of pulling dependencies at runtime. That habit is worth breaking anyway, for the supply chain reasons in containers for AI workloads.
Where a hosted model provider is used, the allowlist has exactly one interesting entry, and it is worth being deliberate about which service and which region it points to.
Private endpoints where they are available
Where the environment consumes a managed service, whether a model endpoint, an object store or a database, reaching it over a private path rather than a public one removes an entire class of exposure. The traffic never traverses a public network, and the service can often be configured to refuse connections that do not arrive that way.
Two things make this more than cosmetic. The refusal side matters more than the private path: a private endpoint alongside a service that still accepts public connections has changed the route, not the exposure. And name resolution has to follow, or applications will resolve the public address and quietly use it.
Administrative interfaces are the soft target
An AI environment accumulates management surfaces quickly: a serving framework's admin API, a vector database console, a notebook environment, an orchestrator dashboard, a metrics interface. Several ship with authentication disabled or with a default that is easy to leave alone.
These deserve stricter treatment than the workload itself. They belong on a separate path from user traffic, reachable only from an administrative network or through a broker, authenticated against the corporate identity provider rather than a local account, and logged.
The notebook environment is worth naming specifically. It usually holds credentials, reaches the data and executes arbitrary code, which makes it one of the most powerful things in the estate and one of the least likely to have been segmented.
Between environments
Where a deployment spans locations, the path joining them is a component with its own security requirements, not an assumption. It needs to be encrypted, authenticated at both ends, monitored, and narrow: a link that carries only the traffic the design calls for, rather than a general route between two networks.
The temptation to widen it grows as new integrations appear, which is why the allowlist for that path deserves a review date. The wider design considerations are in hybrid AI infrastructure.
What segmentation does not do
It does not stop a legitimate caller asking for something they should not receive. If the retrieval layer will return content the asking user is not entitled to, no network control prevents it, because the request comes from exactly where it is supposed to. That is an authorization problem, solved in the retrieval query rather than at a firewall, and it belongs to Private & Secure Enterprise AI.
It also does not make containers a hard boundary. NIST notes in SP 800-190 that containers share the same kernel and do not offer as clear and concrete of a security boundary as a VM, so network policy between two containers on one host is a control worth having and not the only one to rely on where the workloads differ in sensitivity.
Monitoring the paths you allowed
A segmentation design produces two things worth watching: attempts that were denied, and traffic that was allowed but is unusual.
Denied attempts are the more useful signal and the more often discarded. A serving process trying to reach an address outside its allowlist is either a misconfiguration or something worse, and in both cases somebody should see it. Flow logs retained long enough to answer a question after the fact are the minimum; an alert on denials from components that should never generate any is better.
Allowed but unusual is harder and still worth some attention. A retrieval pipeline that normally reaches three systems of record on a schedule, suddenly reaching them continuously, is a change worth noticing even though every packet is permitted.
Retrofitting a segmentation design
Most of this is easier to describe than to apply to an environment that already exists and already works, because every rule added risks breaking something nobody documented.
The sequence that works is to observe before enforcing. Put the policy in a mode that logs what it would have blocked, run it for a period that covers the monthly jobs as well as the daily ones, and read the result. That list is simultaneously the set of exceptions you need and the most accurate architecture diagram the environment has ever had.
Then enforce outward in. Start with egress from the components that should have none, because that is where the risk concentrates and the traffic is easiest to enumerate. Move to the paths between components, which are usually few. Leave user facing ingress until last, since it is the one where a mistake is immediately visible to people.
Expect surprises in two places. Backup agents, monitoring collectors and patch mechanisms all talk to things nobody lists when drawing the diagram. And a component that has always had general internet access has usually acquired at least one dependency on it that only appears at an inconvenient moment, such as a license check or a certificate revocation lookup.
A workable baseline
- Default deny in both directions, with explicit allow rules per component pair.
- No general egress from serving or retrieval; an allowlist of named destinations.
- Private endpoints to managed services, with public access refused at the service.
- Administrative interfaces on a separate path, behind corporate identity.
- Authentication at every hop, including internal ones.
- Flow logs retained, so a question about what talked to what has an answer.
- A review date, because environments accumulate exceptions.
The joint guidance in Deploying AI Systems Securely is written for organizations deploying externally developed AI and aims at improving the confidentiality, integrity, and availability of AI systems, which is a reasonable frame for reviewing a segmentation design.
LABUSA designs this layer with the security practice rather than beside it, as part of AI security architecture services.