Availability engineering for AI is mostly ordinary availability engineering applied to a workload with one awkward property: replacement instances take minutes to become useful. Every technique below is shaped by that fact.
A note on what this page will not do. It will not offer an uptime figure. Availability is a property of a specific design, in a specific environment, operated by a specific team, and any number quoted without those three is decoration.
Start by deciding what unavailable means
Before designing anything, establish what the business actually needs, because AI features vary enormously in how much their absence matters.
An assistant that helps staff draft documents can be unavailable for an hour with mild irritation and no material consequence. A retrieval feature embedded in a customer facing process cannot. The same infrastructure team will happily build either, and building the second when the first was needed is how money is wasted.
NIST's Contingency Planning Guide supplies the vocabulary. The recovery time objective, where RTO defines the maximum amount of time that a system can be unavailable, is the number that should drive this design. Agree it with the business before choosing an architecture, because the architecture is the expensive expression of that number.
Redundancy that accounts for slow starts
The base pattern is unremarkable: more than one serving instance, behind something that distributes across them and stops sending traffic to unhealthy ones.
What changes for AI is the margin. In a web tier, N plus one is comfortable because a lost instance is replaced in seconds. In an inference tier, replacement takes minutes, so the survivors carry the entire load for that whole period. If they cannot, the loss of one instance cascades into the loss of all of them.
So the sizing question is not whether the system survives losing an instance, but whether it survives losing one for the length of time a replacement takes. That usually argues for more headroom than a web workload would need, and it makes warm capacity a resilience investment rather than waste. The arithmetic belongs in AI infrastructure capacity planning.
Health checking that tells the truth
Redundancy only works if the load balancer can identify a bad instance, and an AI serving process fails in ways a port check cannot see. It can be listening while the device underneath it is in a bad state, or while a model has partially failed to load.
A readiness check should therefore exercise the real path with a small fixed inference request, not merely confirm the process is up. Separate liveness from readiness, so an instance loading a model is marked unready rather than restarted, and give the readiness check a generous initial delay for the same reason.
The failure this prevents is the worst kind: an instance in the pool receiving traffic it cannot serve, so a proportion of users fail while the dashboard shows the service healthy.
Graceful degradation is the highest value technique
The most useful resilience work in AI systems is rarely more redundancy. It is deciding what the product does when capacity is short, and building that deliberately.
- Queue and tell the user. A visible wait is far better than a timeout, and it bounds load rather than amplifying it.
- Shed load explicitly. Refusing some requests with a clear message keeps the system usable for the rest. Letting everything slow down together fails for everybody at once.
- Reduce the work. Shorter responses or less retrieved context cost less capacity per request, and are often barely noticed.
- Disable the feature cleanly. An application that continues to work without its AI assistance is a far better outcome than one that breaks entirely, and this is a design decision made long before the incident.
The last point deserves emphasis. Whether the surrounding system depends on the model or merely benefits from it is an architectural choice, and choosing benefit rather than dependency converts a class of outages into a class of inconveniences.
Fallback models, and their honest limits
Routing to an alternative model when the primary is unavailable is attractive and comes with conditions worth stating.
The fallback answers differently. That is acceptable for drafting assistance and may not be for anything where consistency matters, so it is a product decision rather than an infrastructure one. It needs to be tested regularly, because a fallback path exercised only during incidents is a path nobody knows works. It should be visible in logs, so a later question about an odd answer can be traced to the model that produced it. And if the fallback is a different provider, it introduces a second contractual and data path to have already reviewed.
Where fallback is a genuine requirement, the routing belongs in an AI gateway rather than in each application.
What is actually redundant
An availability design should be able to name, for each component, what happens when it fails.
- Serving instances. Multiple, spread across nodes and failure domains.
- The gateway. Redundant, or it becomes the single point every model call shares.
- The vector store. Replicated, or retrieval fails while the model is perfectly healthy and answers become noticeably worse rather than absent.
- The identity provider. Frequently the true single point of failure, and frequently outside the AI team's control.
- Model artifact storage. Needed on every cold start, so its unavailability prevents recovery precisely when recovery is happening.
- Upstream providers. Not redundant at all unless you arranged for it, and their status page is not your monitoring.
The vector store is the one most often missed. A retrieval system whose index is unavailable does not stop; it produces confident answers with nothing retrieved, which is worse than an error because nobody notices.
Multi zone, and when multi region is not the answer
Spreading instances across availability zones is usually straightforward and worth doing. Spreading across regions is a much larger commitment, and for AI it carries a specific cost: model artifacts and vector indexes must exist in both, which means storage, synchronization and a consistency question about which one is authoritative.
For most enterprise AI, multi zone within one region plus a tested rebuild procedure is a better use of the budget than a second region nobody exercises. Multi region earns its place where the RTO is short enough that a rebuild cannot meet it, which is a decision the business makes, not the platform team.
The dependency most likely to take you down
In a self hosted environment the usual cause of an AI outage is capacity, and it arrives as a slow squeeze rather than a failure. In a hosted environment the usual cause is somebody else, and it arrives all at once.
Either way the mitigation begins with knowing which dependencies are shared. An AI feature typically depends on the identity provider, the network path to the model, the retrieval store, the artifact store and, if hosted, an external provider. Several of those are shared with systems that have nothing to do with AI, which means an AI outage is often a symptom rather than an event.
Two practices help. Map the dependencies once and mark which are outside your control, because that set is where a design decision is needed rather than an engineering one. And monitor them from the AI system's point of view rather than trusting their own status reporting, since a provider's dashboard reflects their view of health, not your ability to reach them.
Retries, and how they make things worse
Automatic retry is the most common accidental amplifier in AI systems, because requests are slow enough that clients are configured to retry aggressively and expensive enough that retries hurt.
A saturated endpoint responding slowly triggers client timeouts. Those clients retry. The retries add load to an already saturated endpoint, which slows further, which triggers more timeouts. The system converges on failure without anything having actually broken.
The fixes are standard and worth applying deliberately: bound the number of retries, back off exponentially with jitter so retries do not synchronize, and stop retrying entirely when the endpoint signals overload rather than error. A circuit breaker that fails fast during saturation protects the endpoint and gives users an immediate answer instead of a long wait ending in nothing.
Testing, because untested redundancy is a belief
Every mechanism above should have been observed working. Terminate an instance during business hours and watch the recovery. Mark the vector store unavailable and confirm the system degrades as designed rather than answering from nothing. Force the fallback path. Drain a node and time it.
These are unremarkable exercises and they are the difference between a design and an assumption. They also produce the number that matters for capacity: how long a replacement actually takes in your environment, rather than how long you expected.
Recovery from a failure that redundancy cannot absorb is a different discipline, covered in AI backup and disaster recovery. LABUSA designs and tests these arrangements as part of resilient AI infrastructure design.
Sources and further reading
- NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems. The source of the RTO definition used above.
- Kubernetes documentation, Horizontal Pod Autoscaling.