AI environments accumulate credentials unusually fast. A pilot needs a provider key. Retrieval needs to read three systems of record. The vector store needs a password. The pipeline needs to write to object storage. Within a few weeks an environment assembled by two people contains a dozen secrets, several of them in places that were meant to be temporary.
None of this is specific to AI. What is specific is the rate of accumulation, the fact that several of the credentials cost money when used, and that the components holding them are frequently built by people whose day job is not operations.
What is actually held
Worth enumerating, because the list is longer than most teams expect.
- Model provider API keys. Directly billable, and often the only thing standing between an internal service and a very large invoice.
- Credentials for systems of record that the retrieval pipeline reads. Frequently the most sensitive item, because they typically carry broad read access.
- Vector store and database credentials.
- Object storage credentials for model artifacts and datasets.
- Service identities between internal components: gateway to serving, application to gateway.
- Registry credentials for pulling container images.
- Signing and encryption keys where the environment uses them.
The second entry deserves attention. A retrieval pipeline usually needs to read broadly in order to index, which means the credential it holds is often more powerful than any individual user's. That makes it a high value target and an argument for narrowing its scope deliberately rather than accepting whatever was easiest to issue.
The case for centralizing
OWASP puts the argument plainly in its Secrets Management Cheat Sheet: there is a growing need for organizations to centralize the storage, provisioning, auditing, rotation and management of secrets to control access to secrets and prevent them from leaking and compromising the organization.
Each of those five verbs answers a question an AI environment raises quickly. Storage, so a credential is not in a repository. Provisioning, so a new instance gets one without a person copying it. Auditing, so a question about who used a key has an answer. Rotation, so a leak has a bounded lifetime. Management, so somebody can list what exists.
The last is the one most often missing. An environment where nobody can enumerate its credentials cannot rotate them, cannot revoke them on a departure, and cannot answer a straightforward audit question.
Where they end up instead
Six places, all of them normal and all of them wrong.
- In the container image. An image layer is a distributable artifact. A key baked into one is published to everyone who can pull it, and it persists in the layer history after being removed in a later step.
- In environment variables. Convenient, and visible to anything that can inspect the process, and routinely included in crash dumps and diagnostic output.
- In a notebook. Pasted during exploration, committed with the notebook, and shared with the colleague who asked for it.
- In configuration in version control. Discovered later by a scanner, at which point the rotation has to happen anyway.
- In a personal account. A provider key issued against an individual's login, which becomes an incident when that person leaves.
- In the logs. Discussed below, because AI systems have an unusual route to this one.
Rotation is where good intentions fail
OWASP also identifies the operational reality: automated rotation of static secrets matters because key rotation is a challenging process when implemented manually and can lead to mistakes.
In an AI environment the difficulty concentrates in two places. A model provider key is often used by several applications, so rotating it is a coordinated change rather than a local one, and the usual result is that it is not rotated. And a long lived credential held by a retrieval pipeline that runs nightly may only reveal that rotation broke it a day later.
Two practices remove most of the pain. Support two valid credentials at once so rotation is a rolling change rather than a synchronized one. And prefer short lived credentials issued on demand over long lived ones stored anywhere, because a credential that expires in an hour is a much smaller problem when it leaks.
Workload identity is the better answer
The strongest position is not to store a secret at all. Where the platform supports it, a workload can be given an identity that the surrounding infrastructure attests to, and exchange that identity for a short lived token when it needs to reach something.
The advantages are structural rather than incremental. There is no static value to leak. Access is tied to a workload rather than to a copyable string. Revocation is immediate and central. And the audit trail records which workload acted rather than which key was used.
This aligns with how NIST frames access generally in Zero Trust Architecture, where authentication and authorization are discrete functions performed before a session to an enterprise resource is established, and there is no implicit trust granted based on location. A workload that must authenticate for each session is exactly that principle applied to a machine.
Where a component reaches an external provider that only offers static API keys, workload identity does not extend past your boundary. The realistic pattern is that a broker inside the environment holds the external key and every internal caller reaches the provider through it, authenticating with a workload identity. That is one of the arguments for an AI gateway.
Keeping secrets out of logs
AI systems have a route to this problem that ordinary services do not. Logging is genuinely useful for debugging model behavior, so there is real pressure to record prompts and responses, and a prompt is whatever a user typed. Users paste credentials into chat interfaces. It happens routinely and it is not a hypothetical.
Three defenses, in order of reliability. Decide deliberately whether prompt content is logged at all, with a retention period and an access control, rather than defaulting to yes because it was useful once. Redact known credential patterns on the way in, accepting that pattern matching catches the common shapes and not all of them. And treat the log store as a sensitive system in its own right, because if prompts are recorded it now holds a sample of everything anybody typed.
The same caution applies to error output. A stack trace from a failed connection frequently contains the connection string.
Scoping a credential properly
Most of the damage a leaked credential can do is decided long before it leaks, when somebody chooses how much it can reach.
The retrieval pipeline is the clearest example. It needs to read a great deal in order to index, and the path of least resistance is an account with broad read access across the estate. A narrower arrangement is almost always available: read access to named repositories or sites rather than to everything, issued per source system rather than as one credential, and reviewed when a new source is added rather than reused because it happens to work.
The same reasoning applies to the model provider key. Many providers support multiple keys with separate spend limits and separate revocation, which turns one shared credential into several attributable ones. That is worth doing before it is needed, because the moment it is needed is an incident.
A useful test for any credential in the environment: if this value appeared in a public repository this afternoon, what exactly could someone do with it, and how long would it take us to stop them? A design where the second half of that answer is measured in minutes is a good one, and it is achieved by scope and rotation rather than by secrecy alone.
Detection, because prevention is incomplete
Every practice above reduces the chance of a leak and none eliminates it, so it is worth having something that notices.
Repository scanning catches the most common route and should run on history as well as on new commits, since a credential removed in a later commit is still present in the earlier one. Provider side monitoring is the other half: unusual spend, calls from an unfamiliar network, or use at an hour the workload does not run are all signals available without any additional tooling, provided somebody has decided who reads them.
Finally, rehearse the response. Rotating a model provider key under pressure, across every component that uses it, is much easier the second time. Doing it once deliberately turns an unknown into a documented procedure with a known duration.
A workable baseline
- One secret manager, and everything else holds references rather than values.
- No secret in an image, a repository, a notebook or a personal account.
- Workload identity wherever the platform supports it; short lived tokens where it does not.
- External provider keys held by one component, reached through it by everything else.
- Rotation automated, supporting two valid credentials during the change.
- Scope narrowed deliberately, especially for the retrieval pipeline's read access.
- A deliberate decision about prompt logging, and redaction if the answer is yes.
- An owner for each credential, so a departure triggers a rotation rather than a search.
One consequence for recovery is worth stating: the secret manager becomes a dependency of everything, so it needs its own restoration procedure and it has to come back first. That ordering belongs in AI backup and disaster recovery, and the network paths these credentials travel in AI network segmentation and security.
LABUSA covers this as part of managed AI infrastructure, alongside the identity design that makes workload identity possible in the first place.
Sources and further reading
- OWASP, Secrets Management Cheat Sheet. Community security guidance, not regulation.
- NIST SP 800-207, Zero Trust Architecture, August 2020.