This question is usually asked in the wrong order. It arrives as "how many GPUs do we need", which assumes the answer is more than zero, and it frequently is not.
The honest sequence is: decide whether you are running models yourself, then decide whether the workload needs acceleration, then size it. Skipping the first two steps is how organizations end up with capacity they cannot keep busy and a support contract they did not need.
The question behind the question
Buying accelerators only makes sense if you are hosting models. If your organization consumes a hosted endpoint, the provider owns the hardware problem entirely and the question does not arise. That is not a lesser architecture; it is the majority case, and it is the right answer for a large number of enterprise deployments.
So the first fork is a control decision, not a performance one: does the model need to run inside your boundary? That question belongs to Private & Secure Enterprise AI, and it is settled by data classification, regulatory obligation and contractual terms rather than by throughput.
Only if the answer is yes does hardware enter the conversation.
What accelerators are actually for
A GPU is good at doing the same arithmetic to a great deal of data at once. Model inference is largely repeated matrix multiplication, which fits that shape well, and the practical constraint is usually memory rather than arithmetic: the model has to fit, along with the working state of every request being served concurrently.
That framing explains most of the sizing confusion. Two organizations running the same model can need very different hardware, because one serves three requests at a time and the other serves sixty. Capacity is driven by concurrency and context length at least as much as by model size.
When a CPU is genuinely enough
More often than the market implies. Several real enterprise workloads run acceptably without acceleration.
- Embedding generation. The models that turn text into vectors are small, and an indexing job that runs overnight has no latency requirement worth paying for.
- Classification and extraction. Deciding which of eight categories a document belongs to, or pulling structured fields out of a form, is usually served well by a small purpose built model.
- Low volume assistance. A handful of concurrent users with a small model and a tolerant latency expectation is within reach of ordinary server CPUs.
- Batch work. Anything that can run on a schedule rather than while somebody waits. If the deadline is tomorrow morning, throughput per dollar matters and latency does not.
The tell is the latency requirement. Where a person is waiting, acceleration usually earns its keep. Where a queue is waiting, it often does not.
When a hosted endpoint removes the question
Between the two extremes sits the option most organizations should evaluate first: a commercial model reached over an API, or a dedicated endpoint inside your own cloud tenancy. You get current models, no capacity planning and no hardware lifecycle.
The trade offs are real and they are not about performance. Cost becomes variable and attaches to usage rather than to an asset, which is better for uncertain demand and worse for steady heavy demand. The data path leaves your infrastructure and is governed by a contract. And the model can change underneath you, which matters for anything you have tested and signed off. The shapes available are set out in AI model hosting options for enterprise.
The four conditions that justify your own accelerators
Buying makes sense when several of these hold, and rarely when only one does.
Sustained utilization. Accelerators are expensive to own and cheap to rent for short periods. The arithmetic favors ownership at high, steady utilization, and punishes it at bursty or occasional use. If you cannot describe the demand curve, you are not ready to buy.
A boundary that cannot be crossed. Where a regulatory obligation, a contract or a classification genuinely prevents content leaving, local serving stops being a preference. This is the condition that most often decides it.
Latency that a network cannot meet. Where the round trip to an external endpoint is itself the problem, proximity is the reason to own hardware.
Predictable, controllable behavior. Where a model must not change without your involvement, self hosting pins the version. A hosted endpoint can be pinned contractually, but the control is somebody else's to honor.
What owning them actually commits you to
The purchase is the small part. Ownership brings a set of ongoing obligations that belong on the business case.
- Power and cooling, which are a facilities conversation before they are an IT one.
- Driver and firmware lifecycle, which is a patching stream with its own compatibility matrix.
- Scheduling, so expensive capacity is not held idle by one team.
- Utilization monitoring, without which nobody can tell whether the investment was correct.
- A refresh cycle, because this hardware ages against a fast moving field.
Sharing a device between workloads is possible and is worth understanding before assuming one workload equals one card. NVIDIA documents a mechanism for it in its Multi-Process Service documentation, where the MPS server allocates one copy of GPU storage and scheduling resources shared by all its clients, rather than each process holding its own. Whether that suits your workload is a measurement, not an assumption.
If the workloads will be orchestrated, note that the platform imposes its own constraints. Kubernetes documents that GPUs are only supposed to be specified in the limits section of a pod specification, which means the usual request and limit flexibility that lets CPU be oversubscribed does not apply. The detail is in Kubernetes for AI workloads.
Training is a different question again
Almost everything above concerns inference: running a model that already exists. Training one is a different workload with a different hardware profile, and conflating the two is how organizations arrive at a capacity estimate an order of magnitude too large.
Training is throughput bound rather than latency bound, runs for hours or days rather than seconds, and benefits from many devices working together with fast interconnect between them. Fine tuning sits in between: shorter than training, larger than inference, and usually occasional rather than continuous, which makes it the classic case for rented capacity rather than owned.
For most enterprises the practical position is that they will not train foundation models, may occasionally fine tune, and will run inference constantly. Sizing an environment for the workload you actually have means sizing for the third of those.
The questions worth asking a vendor
Once the decision genuinely is to buy, a handful of questions separate a useful conversation from a configurator.
- How much memory does each device have, and what is the largest model we intend to serve on it at our expected concurrency?
- What is the power draw per device under sustained load, and does the intended rack support it with the cooling we have?
- Can the devices be shared between workloads, and what does that do to the latency of each?
- What is the driver support lifetime, and how does it interact with the operating system and container platform we already run?
- What is the lead time, and what does it become if we need more in six months?
The last one matters more than it looks. An environment sized precisely to today's demand, in a market with long lead times, is an environment that will be undersized at the moment it becomes popular.
A practical way to decide
Run the workload before buying anything for it. Almost every organization can answer this question with a measurement rather than a projection.
- Deploy against a hosted endpoint first, even if the long term intent is to self host. It establishes real demand, real concurrency and real latency expectations.
- Record the distribution, not the average. Peak concurrency and long requests size the environment; the mean does not.
- Price both options against that measured curve, including the operational costs above.
- Re-ask the question on a date. Demand grows, models get smaller and more capable, and the answer that was right in the first quarter often is not in the fourth.
The full sizing method is in AI infrastructure capacity planning, and the hardware layer in GPU infrastructure for enterprise AI. LABUSA runs this as a measured exercise in its AI infrastructure assessment rather than as a procurement conversation.
Sources and further reading
- NVIDIA, Multi-Process Service documentation. Cited for how GPU sharing works on that vendor's hardware, not as a general industry claim.
- Kubernetes documentation, Schedule GPUs. The source of the limits section constraint.