Containers arrived in AI infrastructure for an unglamorous reason: the dependency problem is severe. A model serving stack couples a framework, a numerical library, a device driver interface and a specific Python environment, and the versions have to agree. Reproducing that arrangement by hand on each machine is how organizations end up with an environment that works on one node and not on its neighbor.
What a container actually is
Worth being precise, because the word is used for several different things. NIST's Application Container Security Guide defines application container technologies as a form of operating system virtualization combined with application software packaging, providing a portable, reusable and automatable way to package and run applications.
Two halves. The packaging half is an image: a layered filesystem containing the application and everything it needs, addressable by digest. The virtualization half is a runtime that starts a process from that image with its own view of the filesystem, network and process table, while sharing the host's kernel.
The formats are standardized rather than proprietary. The Open Container Initiative describes itself as an open governance structure for the express purpose of creating open industry standards around container formats and runtimes, which is why an image built with one tool runs under another.
What containers buy an AI workload
Reproducibility. The exact framework, library and driver interface versions travel with the application. An instance started next month is the instance you tested, which matters more here than in most workloads because a version mismatch in a numerical stack does not always fail loudly.
A deployment unit. Scaling, restarting and rolling back become operations on an image reference rather than on a machine's state.
Density with boundaries. Several workloads share a host with their own filesystems and resource limits, which is what makes shared accelerator capacity practical.
Portability, up to a point. An image runs anywhere the runtime does, which is genuinely useful in a hybrid estate. The limit is hardware: an image built against one accelerator's runtime is not portable to another's.
What containers do not isolate, and NIST says so plainly
This is the part most worth getting right, because it is routinely overstated in both directions.
NIST is unambiguous. While containers provide a strong degree of isolation, they do not offer as clear and concrete of a security boundary as a VM. Because containers share the same kernel and can be run with varying capabilities and privileges on a host, the degree of segmentation between them is far less than that provided to VMs by a hypervisor. Elsewhere the same document notes that the use of a shared kernel invariably results in a larger inter-object attack surface, and states that the level of isolation provided by container runtimes is not as high as that provided by hypervisors.
The practical reading is not that containers are unsafe. It is that a container boundary is a good operational boundary and a weaker security boundary than a virtual machine, so it should not be the only thing standing between workloads of different sensitivity. Where two AI workloads must not reach each other, the separation belongs at a level the kernel does not share, and the network path between them belongs in AI network segmentation and security.
It follows that no container platform makes an AI system secure by itself. It changes the shape of the problem and leaves the architecture, configuration and operations to do the work.
The image size problem is specific to AI
AI images are large. A serving stack with its framework and libraries is often measured in gigabytes before any model weights are considered, and the temptation to bake the weights in makes it considerably worse.
Large images have real operational consequences. Cold starts are slower, which matters because inference instances already start slowly. Node replacement is slower. Registry storage and transfer costs rise. And a scale out event during a demand spike becomes a large synchronized download at exactly the wrong moment.
Two practices help most. Keep model weights out of the image and mount them from storage, so the image changes when the code changes and not when the model does. And use a registry close to the compute, with images pre pulled onto nodes where the platform supports it. The cold start arithmetic this affects is covered in AI inference infrastructure.
Device access breaks the usual container promise
A container's appeal is that it needs nothing from the host but a runtime. Accelerators break that. The container needs access to a physical device, and the driver on the host has to be compatible with the libraries inside the image.
This produces the one genuine coupling in an otherwise portable arrangement: host driver version and image contents have to agree, so upgrading a driver is a fleet wide event rather than a per node decision. Plan driver upgrades as coordinated changes with a tested image pairing, and record the working combinations, because discovering an incompatibility during an incident is expensive.
Building images that behave
- Pin versions. A floating tag makes the image non reproducible, which defeats the main reason for using one. Reference images by digest where it matters.
- Do not run as root. Combined with a shared kernel, a root process inside a container is a materially larger risk than a root process inside a VM.
- Keep the image minimal. Build tooling, compilers and package managers left in a runtime image are attack surface and bulk.
- Scan images and mean it. Scanning that nobody acts on is a report, not a control.
- Keep secrets out. An image layer is a distributable artifact, and a credential in one is a credential published. See secrets management for AI systems.
- Set resource limits. Unbounded memory in a container that holds a model is a node level outage waiting for a long request.
Storage, state and the stateless assumption
Containers are usually described as stateless, and AI workloads test that description immediately.
A serving container holds an enormous amount of state in memory and none of it should be persisted, which is the easy case. The awkward cases are the things around it: model weights, embedding caches, tokenizer files and downloaded artifacts, all of which are large, slow to fetch and shared between instances.
The workable pattern is to treat the container as disposable and the artifacts as mounted. Weights live on a shared volume or an object store and are mounted read only, so a hundred instances reference one copy rather than pulling a hundred. Caches live on node local storage where losing them costs time rather than correctness. Nothing that matters lives inside the writable layer of a running container, because that layer disappears when the container does, and nobody notices until a restart loses something.
Two questions settle most designs. If this container is destroyed right now, what is lost? And if fifty of them start at once, what do they all read, and can it serve them?
A note on the platform you already run
Most organizations considering containers for AI already run containers for something else, and the sensible default is to use that platform rather than build a second one.
The reasons to reuse it are strong: one set of operational practice, one identity integration, one monitoring stack, one set of people who know how it behaves at three in the morning. The reasons not to are specific and worth checking before committing. Accelerator nodes are unlike the rest of the fleet in driver lifecycle and cost, and mixing them into a general pool can mean expensive nodes running ordinary workloads. Image sizes are far larger, which can strain a registry sized for application containers. And a model serving pod that holds a device for minutes behaves nothing like a web pod that lives for milliseconds, which affects how the cluster should be allowed to evict and reschedule it.
None of those is a reason for a separate platform. All of them are reasons for a separate node pool with its own policies inside the platform you have.
When containers are the wrong answer
Two cases. A single dedicated machine serving one model for one purpose gains little from containerization and pays its costs, and there is no shame in a systemd unit. And where tenants genuinely must not share a kernel, containers alone are the wrong boundary regardless of convenience, for the reasons NIST sets out above.
What comes next
Containers solve packaging and isolation. They do not decide which machine a workload runs on, what happens when it dies, or how ten of them are kept running. That is orchestration, and for AI workloads it brings its own constraints, covered in Kubernetes for AI workloads.
LABUSA builds and operates container platforms for AI workloads as part of its AI platform and container engineering work, including the driver and image lifecycle that keeps them reproducible.
Sources and further reading
- NIST SP 800-190, Application Container Security Guide, September 2017. The source of the definition and of the security boundary statement.
- Open Container Initiative.