The instinct with a new system is to add it to the backup schedule. For an AI environment that instinct produces something expensive and incomplete, because the components have genuinely different recovery properties. Some must be restored from a copy, some are better rebuilt from configuration, and one of the largest is derived data that can be regenerated from a source you are already protecting.
Working out which is which is most of the job.
The vocabulary, and why it is worth using precisely
NIST's Contingency Planning Guide supplies the two numbers every recovery design turns on. The recovery time objective, where RTO defines the maximum amount of time that a system can be unavailable, and the recovery point objective, where the RPO represents the point in time to which data must be recovered.
The same document sets out why backup type matters: a full backup captures all files on the disk or within the folder selected for backup, which is simple to restore and slow to take. That trade off is ordinary, and it becomes interesting in AI environments because some of the data is very large and changes rarely, while other parts are small and change constantly.
Sort the environment into three piles
Every component belongs in one of them, and putting a component in the wrong pile is where cost or risk appears.
Restore from backup
Data that is authoritative, cannot be recreated, and would be lost with the system.
- Source documents, where the AI system is the only place they live. Usually it is not, and they belong to a system of record that has its own backup.
- Application databases: conversation history, feedback, evaluation results, usage records.
- Prompts and templates that were authored rather than generated, if they live outside version control.
- Audit logs, whose retention is often set by policy rather than by preference.
Rebuild from configuration
Anything that is reproducible from a definition you already keep, where the definition is far smaller than the thing.
- Container images and the cluster resources that describe the environment.
- Network policy, identity mappings and platform configuration.
- Serving configuration and routing rules.
This pile is a strong argument for treating environment definitions as code. The recovery plan for anything in it is a pipeline run, which is faster and more reliable than a restore, provided the definitions are genuinely complete. The test of that is whether the environment has ever been built from them into an empty account.
Regenerate from source
Derived data. Chiefly the vector index, which is large, expensive to back up, and reproducible from documents you already protect.
The vector index deserves its own decision
This is the component that most often ends up in the wrong pile.
Backing it up is straightforward and can be substantial in size, with an RPO that lags the source content. Regenerating it is free of storage cost and takes as long as re-embedding the corpus, which for a large estate can be many hours and, if a hosted embedding service is used, carries a real bill.
The deciding question is the RTO. If the business can tolerate retrieval being degraded for the length of a full re-index, regeneration is the cheaper design and it has a useful property: the rebuilt index is consistent with the current source content rather than with the content as it was at backup time. If it cannot, the index needs backing up like anything else.
A middle path is often best: back up the index at a low frequency so a restore gives most of the corpus quickly, and re-embed only what changed since. That requires knowing what changed, which means the pipeline needs to track document versions, which is worth building for other reasons anyway.
Whatever is chosen, note that an unavailable index does not stop a retrieval system. It produces answers with nothing retrieved, which is why this failure belongs in the monitoring described in high availability for AI systems.
Model artifacts are large and usually not yours
Weights are among the biggest objects in the environment, and how they are protected depends on where they came from.
A model obtained from a public source can be re-obtained, so it does not need a backup so much as a recorded provenance: which model, which version, and from where. What that record must include is the exact version, because a re-download months later may not produce the same file.
A model you fine tuned is different. It is expensive to reproduce and may be impossible to reproduce exactly, so it is authoritative data and belongs in the first pile alongside the dataset and parameters used to produce it.
Either way, the artifact store is part of the recovery path rather than merely a backup target: instances cannot start without it. An artifact store that is unavailable during a regional failure prevents recovery in the region you failed over to, which is the sort of dependency that is only discovered during a test.
Secrets are a recovery problem too
A restored environment with no credentials does nothing. Yet secrets should not sit in ordinary backups, because a backup is a copy that travels.
The workable pattern is that the secret manager is itself the backed up system, with its own documented recovery procedure, and that everything else holds references rather than values. The recovery runbook then contains a step to restore or re-establish the secret manager before anything that depends on it. Handling of the values themselves is covered in secrets management for AI systems.
Ransomware changes the calculation
An AI environment is an attractive target for two reasons that are easy to miss. The vector index is a distilled, searchable copy of the organization's most useful documents, and the artifact store holds large files that many systems load and execute against without much scrutiny.
Two consequences for backup design. Copies need to be genuinely separated, so that credentials able to reach the live environment cannot reach the backups, and immutability for a retention period is worth more here than backup frequency. And integrity checking matters as much as availability: a restored model artifact that was tampered with is worse than no artifact, because the environment comes back up and behaves subtly differently.
Recording a checksum for every artifact at the point it enters the environment, and verifying it on load, is a small piece of work that turns an unanswerable question during an incident into a check.
What a hosted service does and does not cover
Where models are consumed from a provider, much of the above becomes their problem, and it is worth being precise about which parts.
The provider is responsible for the availability of their service and for the model artifacts behind it. You are still responsible for everything that makes the feature work in your organization: the retrieval index over your content, the application and its database, the configuration, the prompts, the credentials, and the record of what was asked and answered.
That remaining set is smaller and it is not small. An organization that concluded backup was handled because the model is hosted has usually not protected the index, which is the component that took longest to build.
What a plan should actually say
A usable recovery plan for an AI environment answers five questions on one page.
- What is the RTO and RPO for this system, agreed with the business rather than assumed?
- For each component, is it restored, rebuilt or regenerated, and from what?
- What is the order of recovery, given that some components cannot start until others exist?
- What does the system do while it is partially recovered, particularly if retrieval is available before the index is?
- When was this last tested, and how long did it take?
The order question is the one most often missing. An AI environment has a genuine dependency sequence: secrets, then artifact storage, then serving, then retrieval. Attempting it in a different order produces confusing failures during the exact hour nobody wants to be confused.
Test it, and time it
A recovery procedure that has not been executed is a hypothesis. The useful exercise is a rebuild into a clean environment, timed, with the resulting system checked for whether it actually answers questions correctly rather than merely starting.
That test produces the number the business asked for at the beginning, which is how long recovery genuinely takes. It very often differs from the estimate, and it is far better to discover that on a Tuesday.
LABUSA builds and tests these arrangements as part of AI infrastructure operations, alongside the resilience design that determines how often they are needed.
Sources and further reading
- NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems, May 2010. The source of the RTO, RPO and full backup definitions used above.
- NSA, CISA and international partners, Deploying AI Systems Securely, April 2024.