Sizing GPUs for an On-Prem Agent Deployment

The SI walked into the SDC cage with a quote for eight of the largest cards the distributor had that month. The workload was a scheme-helpdesk agent, a 7B-class model, a few dozen officers, and a retrieval corpus that fit on a disk. The SDC head asked why eight. The SI said future proof and also the demo had used a bigger model in the lab.

That quote was not sizing. It was a fear of being blamed for slowness, converted into capital. Honest sizing starts from four numbers you can defend: which model class actually wins the eval, how long the context will be, how many generations happen at once, and whether retrieval is on the same box.

This guide uses bands, not trophies. Hardware generations change. Frameworks change. Your measurement on a loaner card beats this article. What this article will not do is invent a tokens-per-second table and pretend it is a benchmark you can cite.

Four numbers before a SKU

1. Model class and quantisation. A 7B–8B Q4-class generator is a different machine from a 70B Q4-class generator. Full precision is rarely how you serve in a department.
2. Context. RAG prefill on a 16k or 32k window eats memory for the KV cache even when the weight file is small. A short-answer FAQ and a long-document drafter are different sizes.
3. Concurrency. Simultaneous generations, not named users. Forty officers who each send a query every few minutes are not forty concurrent decodes.
4. What else shares the card. Embedding models, vision, judges, and training jobs will evict your serve if you let them.

Bands for planning conversations. Re-measure on the framework you will actually run.

Planning band	Typical VRAM story	Who it is for
7B–8B instruct, Q4-class, short-to-medium context, low concurrency	Often about 8–12GB for weights plus modest KV; a single mainstream card can be enough	Most departmental drafters and FAQ agents
Same model, long RAG context or higher concurrency	The weights still fit; the cache and batch do not. Step up memory or add a second serve replica	Busy helpdesks, long circulars
30B-class Q4	A mid-to-high memory card, or more than one smaller card with a framework that shards	When 7B fails a frozen eval on reasoning-heavy tasks
70B-class Q4	Tens of GB for weights plus cache; multiple heavy cards or a high-memory node; power and cooling become first-class	When you have evidence, not a demo hangover
Embedding only	Often CPU or a small GPU; do not steal the generator’s card	Index builds and query encode

RAG, context and the quiet memory thief

Institutions love long context because they love pasting circulars. Every retrieved chunk you stuff into the window costs memory and latency on prefill. A better index that returns fewer, better chunks can save you a card. That is sizing work, not only search work.

Keep the generator’s GPU for generation. Run the vector database on CPU RAM unless you have a measured reason not to. Departmental corpora are usually RAM-sized, not cluster-sized. Buying a GPU because the word vector sounds like CUDA is how quotes bloat.

CPU, spare cards and the job you should not GPU

A small 7B-class model on CPU is ugly and sometimes honest. If concurrency is one or two, and the SDC cannot power a card this quarter, a CPU serve with a clear latency expectation is better than a hosted API you called temporary. Write the latency. Do not surprise officers who thought they had a GPU.

Spare policy is part of sizing. One live card and no spare means the first failure is an outage that pushes people back onto phones. For a citizen-facing desk, budget a second card that can also run eval. For an internal drafter used by five people, a measured risk acceptance may be cheaper than a trophy spare. Write which you chose.

Do not GPU the jobs that do not want it. Chunking, OCR on a batch of PDFs overnight, and bulk embedding of a stable corpus can often wait on CPU. The generator’s card should not be stolen by a Sunday index rebuild. Schedule the rebuild, or give the rebuild its own small device, or run it on the spare.

Objections

Give us TPS so we can put it in the SLA. Give a latency band on a named eval prompt at a named concurrency, measured on the purchased card, after go-live. That can live in an SLA. A brochure TPS cannot.

We should buy NVIDIA only / we should buy the other brand only. Buy what your framework supports and what your SDC can power. This article will not run a brand war or invent scores.

Quantisation will ruin quality. Sometimes it hurts. That is why you have a frozen eval. If Q4 fails and a higher precision on the same class passes, size for that. Do not jump a model class to avoid measuring quantisation.

What an RFP may ask, and what it must not

An RFP may ask for a loaner measurement on a named model class, a named context length, and a named concurrency, with the method attached. It may ask for live-plus-spare. It may ask for a degrade path to a smaller class.

It must not ask for a single tokens-per-second number without a method. It must not require a 70B-class serve without an eval that a 7B-class failed. It must not require a particular consumer brand unless the SDC has a documented reason. Brand-as-sovereignty is theatre with a SKU.

Integrators should attach the loaner graph even when the RFP forgot to ask. Committees remember the graph when the card arrives late and someone claims the SI undersized. The graph is your defence and the department’s memory.

A three-week sizing sprint

• Week 1: freeze a 50-item eval and a realistic prompt length including RAG.
• Week 2: run 7B-class Q4 on a loaner in the target band. Record latency at 1, 2, 4 concurrent generations. Do not hunt TPS for slides.
• Week 3: only if eval fails, try the next class. Write the buy as live plus spare, not as a trophy row.

What goes in the file

The four numbers, the eval outcome, the loaner measurements, the power-and-cooling note from the SDC, and a sentence that no unverifiable TPS was used to choose the card. If finance wants a single chart, give them the concurrency-latency graph from the loaner.

Keep the fail-notes too. The 7B-class run that failed two consistency items is why you rented the larger node. Without that note, the next committee will assume you buy large cards for status. With it, the next committee can ask whether the corpus can be improved instead.

Write the degrade plan in the same file: if the heavy card dies, the 7B-class spare takes the load, officers see a banner, and the long-chapter job waits. A sizing file without a degrade plan is a hope that hardware does not fail.

Prcept deployments should start on the smallest class that passes your eval. We will not pad a quote with idle accelerators to look serious.

How to prove this on a rack, not on a slide

“Sizing GPUs for an On-Prem Agent Deployment” only matters if a CISO can fail it. A P4 System Integrator should be able to point at a cable, a registry, a licence file, a PDU reading or a SIEM index and say: this is the control. If the only evidence is a brochure that mentions “on-prem AI GPU sizing”, you do not have the control.

Size from model class, context, concurrency and retrieval, not from a keynote. A 7B-class Q4 often fits in 8–12GB VRAM. A 70B-class serve is a different machine. Air-gap and on-prem programmes die in the second month, when the first update, the first crash, or the first GPU lead-time slip arrives. Budget the boring path — media, offline licence, local registry, local traces — in the same note as the model name.

On-prem is not air-gapped. An India region is not either. Write the forbidden path (outbound HTTPS, licence phone-home, crash reporter, hidden model API) as a numbered list and test it with the internet off. Whatever still dies was a dependency you did not draw.

1. Draw the data path for one user-visible answer under “on-prem AI GPU sizing”.
2. Disable outbound internet on staging and run the demo script.
3. List every remaining hop: update, licence, registry, NTP, DNS, SIEM.
4. Give each hop an owner inside the department, not only the SI.
5. Minute the restore or the media-transfer once before go-live.

Close this loop before the next CAB

Put “Sizing GPUs for an On-Prem Agent Deployment” on the next change-advisory or bid-opening agenda as a single line item with an owner. If it cannot earn a line item, it will not earn a control. The owner should be a P4 System Integrator, not “the vendor.”

Revisit the item when the model, the GeM term, the region, or the SI changes. “on-prem AI GPU sizing” is not a one-time workshop. It is a watch item. Date the last check. Unsigned watch items are souvenirs.