Capacity planning for AI has an unusually high ratio of confident numbers to defensible ones. The reason is that the familiar unit, requests per second, does not describe the workload, and everything built on top of it inherits the error.
What follows is the method LABUSA uses, and an honest statement of its limits. Capacity for this workload cannot be derived accurately from generic formulas. It can be estimated, bounded and then corrected by measurement, which is a different and more useful claim.
Why request counts mislead
Two requests to the same endpoint can differ by more than an order of magnitude in what they cost, because cost is driven by how much text goes in and how much comes out.
Input length determines how much working state the request occupies while it is being served. Output length determines how long it occupies it, because generation is sequential and produces one token at a time. A short question with a one line answer and a long document summarization are the same request in a log and completely different in an environment.
So a plan built on requests per minute is a plan built on an average that conceals the thing that matters. It will be roughly right in normal conditions and badly wrong precisely when the workload changes shape, which is what happens when people find a new use for a feature.
The unit that works
Concurrency at a given context length. That is, how many requests are being served at once, and how much text each is carrying.
This unit works because it maps directly to the binding constraint. Serving requires the model to be resident in device memory, and each in flight request holds working state alongside it. The number of requests that fit is whatever memory the model leaves over, divided by what each one occupies. Everything else follows.
The practical form is a small table rather than a single number: expected concurrency at the median context length, at the ninetieth percentile, and at the worst case the product allows. Sizing to the median produces an environment that fails during the interesting hours. Sizing to the worst case produces one that is mostly idle. The useful design usually serves the ninetieth percentile with capacity and handles the tail with queueing.
The seven inputs
A defensible estimate needs these, and most stalled capacity conversations are stalled because two or three are missing.
- Population and adoption. How many people have access, and what fraction use it in a given hour. Adoption is usually the most uncertain input and the one with the largest effect.
- Shape of the day. Peak to average ratio. Enterprise AI is unusually peaky because it follows the working day, and a global organization has a different curve from a single site one.
- Input length distribution. Especially whether retrieval is inserting large passages into every request, which most enterprise deployments do.
- Output length distribution. Set as much by product design as by users. A feature that returns a paragraph and one that returns a page are different workloads.
- Latency requirement. What counts as acceptable, expressed as a percentile rather than an average.
- Model choice. Its memory footprint sets the ceiling before any request arrives.
- Availability target. Headroom to survive losing an instance for the minutes a replacement takes, which is real capacity and belongs in the number.
Method
Six steps, in this order.
1. Measure rather than estimate wherever possible. Deploy against a hosted endpoint first, even if the intention is to self host later. A month of real usage answers four of the seven inputs and costs a fraction of guessing wrong about hardware.
2. Establish the per instance ceiling empirically. Load a candidate model on candidate hardware and increase concurrency until latency leaves the acceptable band. That number, at a stated context length, is the unit everything else multiplies.
3. Convert demand into required concurrency. Peak concurrent users, times the share actively awaiting a response, gives concurrent requests. This is where a plan should record its assumptions explicitly, because they will be revisited.
4. Add headroom for failure, not just for growth. Enough that losing one instance leaves the rest able to serve peak demand for the several minutes a replacement takes. This is the step most often skipped, and it is the one that turns a single failure into an outage.
5. Decide the queue policy. Capacity and queueing are alternatives for the same demand. A system with a good queue and a visible wait needs less hardware than one where every request must be served immediately.
6. Re-measure on a schedule. Adoption changes, models change and product decisions change output length. A capacity plan is a document with a review date, not an artifact.
Autoscaling is not a substitute for planning
Kubernetes describes horizontal scaling as deploying more pods in response to increased load, and the mechanism automatically updates a workload resource to do it. For inference, that response takes minutes because an instance must acquire a device, pull a large image and load a model.
Autoscaling therefore handles the daily curve well and sudden spikes badly. It reduces the cost of provisioning for the peak; it does not remove the need to know what the peak is. And it can only add instances if there is somewhere to put them, which for accelerator workloads means either idle devices or a node autoscaler that is itself slower still.
Sharing devices, and what it does to the arithmetic
Utilization can be improved by placing more than one workload on a device, and the mechanisms differ in isolation. NVIDIA documents one in its Multi-Process Service documentation, noting reduced GPU context switching and that the MPS server allocates one copy of GPU storage and scheduling resources shared by all its clients.
Sharing raises average utilization and couples the tenants, so a plan that assumes it must also assume the neighbors behave. For interactive workloads with a latency commitment, dedicating devices and accepting lower utilization is frequently the right trade. For batch work it rarely is. The mechanisms are covered in GPU infrastructure for enterprise AI.
Cost is the other half of the same exercise
Capacity and cost are the same question asked by two departments, and answering only one produces a plan that gets overturned.
The distinction that matters is between fixed and variable. Owned capacity is largely fixed: it costs the same whether it is busy or idle, so the economics improve with utilization and punish overprovisioning. Consumed capacity is variable: it costs nothing when idle and scales linearly with success, which is comfortable at low volume and can become the largest line item at high volume.
That gives a crossover point, and knowing roughly where it sits is more useful than a precise figure for either option. Below it, consumption wins on both cost and effort. Above it, ownership wins provided utilization is genuinely high, which returns the question to the measurement above.
Two costs are routinely left out of the comparison. Operating an owned environment has a staffing cost that does not appear on any invoice, and it does not scale down for a small estate. And retrieval heavy designs move a great deal of data, so egress and storage can be a material fraction of the total in a way that a model bill alone conceals.
Common ways the estimate goes wrong
- Sizing on averages. The mean request is not the one that saturates the environment. Plan on distributions.
- Forgetting retrieval. Adding retrieved context to every prompt can multiply input length several times over, and it is usually introduced after the capacity estimate was made.
- Assuming adoption is linear. Internal AI features tend to have a slow start and a sharp inflection when a team finds a use that works.
- Ignoring the second model. Capacity planned for one model rarely survives the arrival of a second one on the same hardware.
- Counting headroom as waste. Idle capacity that exists to absorb a failure is doing its job. Removing it converts an incident into an outage.
What the plan should contain
A capacity plan worth having states, on one page: the measured per instance ceiling and the conditions it was measured under, the demand assumptions with their source, the resulting instance count, the headroom and what failure it covers, the queue policy, and the date it will be revisited.
The second item is the one that makes it defensible. A capacity number whose assumptions are written down can be corrected when an assumption turns out to be wrong. One without them can only be argued about.
Whether any of this requires owning hardware at all is the prior question, addressed in do you need GPUs for enterprise AI. LABUSA runs sizing as a measured exercise within its AI infrastructure capacity work, and re-runs it once real demand exists.
Sources and further reading
- Kubernetes documentation, Horizontal Pod Autoscaling.
- NVIDIA, Multi-Process Service documentation. Cited for that vendor's own product behavior only.