Best GPUs for
AI Inference in 2026
How to choose an inference accelerator: size the weights and the KV cache together, decide whether you are optimizing latency or throughput, then work out what that implies for capacity, bandwidth, and deployment. The arithmetic is shown so you can check it against your own traffic.
Updated
The best inference GPU is the one that fits the model and its KV cache while meeting your latency, throughput, concurrency, power, and budget targets. Memory capacity determines whether the workload fits at all, but real performance also depends on model architecture, quantization, serving software, batching, and system topology. Evaluate the complete serving configuration against your own traffic pattern.
Which Class Fits the Serving Job
Grouped by what decides suitability for inference: how much fits, how fast memory feeds the compute, and whether the part can run continuously.
| Class | Memory and interconnect | Suited to | Where it stops |
|---|---|---|---|
| Consumer | 12 to 32 GB GDDR, no GPU-to-GPU link, no ECC | Local and single-user serving of quantized small and mid-size models | Concurrency, long context, and continuous unattended operation |
| Workstation | Up to 96 GB GDDR7 with ECC, PCIe only | On-premises serving of quantized large models, data-residency constraints | Bandwidth-bound decode at scale, and multi-accelerator model splitting |
| Datacenter, single accelerator | 80 to 288 GB HBM at several TB/s, ECC, high-bandwidth interconnect | Production serving with real concurrency and latency targets | Models or KV budgets larger than one node can hold |
| Datacenter, platform and rack | 8-GPU platforms and 72-GPU rack systems as the purchasing unit | Large-model serving at high concurrency, disaggregated prefill and decode | Sites that cannot deliver the power and cooling the unit requires |
Specifications for every accelerator in these classes, with the manufacturer page each figure came from and the date it was last checked, are in the GPU specifications database. This guide does not restate them, so the two cannot disagree.
Sizing Inference Memory
Two separate costs: the weights, which are fixed, and the KV cache, which grows with context length and with every concurrent request. Sizing only the weights is the most common way an inference deployment is under-specified.
Weights
| Parameters | FP16, 2 B/param | 8-bit, 1 B/param | 4-bit, ~0.5 B/param |
|---|---|---|---|
| 7B | 14 GB | 7 GB | ~3.5 GB |
| 13B | 26 GB | 13 GB | ~6.5 GB |
| 32B | 64 GB | 32 GB | ~16 GB |
| 70B | 140 GB | 70 GB | ~35 GB |
| 175B | 350 GB | 175 GB | ~88 GB |
KV cache
| Attention scheme | Per token, FP16 | At 8k context, one request |
|---|---|---|
| Multi-head, no KV sharing | ~2.6 MB | ~21 GB |
| Grouped-query, 8:1 KV sharing | ~0.33 MB | ~2.7 GB |
| Grouped-query, 8:1, 8-bit KV | ~0.16 MB | ~1.3 GB |
Per token of context, the cache holds a key and a value for every layer:
bytes per token = 2 x layers x hidden size x bytes per element / KV sharing ratio
The worked figures above use 80 layers, a hidden size of 8,192, and FP16 at 2 bytes, which gives 2 x 80 x 8,192 x 2 = 2,621,440 bytes, or about 2.6 MB per token. Grouped-query attention divides that by the ratio of attention heads to key-value heads, and quantizing the cache divides it again by the precision ratio.
Three things follow, and they are the whole reason to do this arithmetic:
- Cost scales with context length. The same request at 128k context costs sixteen times what it costs at 8k.
- Cost scales with concurrency. Every simultaneous request carries its own cache, so serving fifty users at 8k costs fifty times the single-request figure.
- Architecture dominates. KV sharing changes the figure by nearly an order of magnitude, which is why two models of the same parameter count can have completely different serving economics.
Total requirement is weights plus peak concurrent KV cache plus runtime and framework overhead. Leave headroom: a server that fits exactly at your target concurrency has none for a traffic spike.
Memory capacity
Decides whether the model loads, and then how many concurrent requests you can hold. Capacity is the constraint that eliminates candidates; nothing else on this page can substitute for it.
Memory bandwidth
Bounds decode at low batch size, where each generated token reads the whole model from memory. It does not bound prompt processing, and it stops being the limit once batches get large enough for compute to dominate.
Supported precisions
A narrower weight format reduces both the footprint and the bytes moved per token. Whether it also reduces latency depends on whether your kernels are memory-bound and whether the framework has an optimized path for that format.
Interconnect
Only matters once a model is split across accelerators. Tensor parallelism communicates within every layer, so a slow link shows up directly in inter-token latency rather than just in aggregate throughput.
ECC and continuous operation
A serving deployment runs indefinitely rather than for the length of a job. ECC memory and datacenter thermal design exist for that duty cycle; consumer parts are not built for it.
Serving software
Continuous batching, paged KV cache, speculative decoding, and prefix caching change throughput by large factors on unchanged hardware. The serving stack is part of the configuration you are choosing, not a detail.
Latency and Throughput Are Different Problems
They trade against each other, and a configuration tuned for one is the wrong configuration for the other. Decide which you are optimizing before comparing hardware.
| Phase | What it does | Usually limited by | The metric it drives |
|---|---|---|---|
| Prefill | Processes the whole prompt in parallel and fills the KV cache | Compute, because all prompt tokens are processed at once | Time to first token |
| Decode | Generates one token at a time, reading the model each step | Memory bandwidth at low batch, compute at high batch | Inter-token latency, and tokens per second per user |
Optimizing for interactive latency
- Long prompts make prefill dominate, so time to first token is a compute and context-length problem
- Small batches keep inter-token latency low and leave bandwidth as the decode limit
- Splitting a model across accelerators adds per-layer communication, which lands directly in latency
- Accelerator utilization will be low, and that is the correct trade for an interactive product
Optimizing for aggregate throughput
- Large batches amortize the weight read across many requests, so bandwidth stops being the limit
- KV cache capacity becomes the ceiling on batch size, which makes memory capacity the throughput constraint
- Per-user latency gets worse as batch size grows; offline work does not care
- Continuous batching in the serving layer matters more here than the choice between two similar accelerators
This is why memory capacity, not bandwidth, is often the throughput constraint on a modern serving stack: batch size is bounded by how much KV cache fits, and throughput is bounded by batch size.
What Quantization Does and Does Not Buy
The memory saving is predictable. The speedup and the quality impact are not.
| Effect | Predictable in advance? | Why |
|---|---|---|
| Weight memory reduction | Yes, arithmetic | 8-bit halves the FP16 footprint and 4-bit quarters it, subject to a small overhead for scales and zero points. |
| KV cache reduction | Yes, arithmetic | Quantizing the cache scales its size by the precision ratio, which directly raises the concurrency a given accelerator supports. |
| Decode latency improvement | No | Only materializes where the kernel is memory-bound and the framework has an optimized path for that format. Dequantization overhead can cancel the gain. |
| Prefill improvement | No | Prefill is compute-bound, so reducing bytes moved does not necessarily help it. |
| Output quality impact | No | Depends on the model, the quantization method, and the task. Reasoning, code generation, and long-context behaviour are more sensitive than short-form generation. |
Treat any claim that quantization costs negligible quality as a hypothesis about your model rather than a property of the technique. Run your own evaluations before and after, on the tasks you actually care about, and check long-context and reasoning behaviour specifically.
Consumer GPUs
Genuinely capable for single-user serving of quantized models. Constrained by capacity, concurrency, and duty cycle.
Where consumer cards work for inference
Local and single-user serving
Reasonable uses
- Serving a quantized small or mid-size model to one user or a handful
- Local development against a real model rather than a hosted endpoint
- Edge and on-premises deployments where the traffic is genuinely low
- Work where the data must not leave the machine
Workstation GPUs
Up to 96 GB of ECC GDDR7, which is enough weight capacity for a quantized large model plus a usable KV budget.
What the capacity buys
Large quantized models, served on premises
Reasonable uses
- A 4-bit 70B-class model plus KV cache for modest concurrency on a single card
- On-premises serving without datacenter power, cooling, or rack space
- Deployments under data-residency or air-gap constraints
- Builds where board-power variants let the card match the chassis and its cooling
Datacenter GPUs, Single Accelerator
HBM capacity and bandwidth, ECC, and a high-bandwidth interconnect. Where production serving with real concurrency and latency targets happens.
| Difference | Why it changes the decision |
|---|---|
| Memory capacity per accelerator | Sets both whether the model fits on one accelerator and how much KV cache is left over, which sets the batch size and therefore the throughput ceiling. Fitting on one accelerator also removes tensor-parallel communication from the latency path. |
| Memory bandwidth | Caps decode speed at low batch size, which is the regime interactive serving runs in. It stops being the limit at high batch, so its importance depends on whether you are latency- or throughput-constrained. |
| Air-cooled PCIe against SXM or OAM modules | The module form factors carry the high-bandwidth interconnect and higher board power. The PCIe variants exist so that sites which cannot take that power or cooling can still deploy the same memory capacity, sometimes with more of it. |
| Narrow-precision support | Newer generations add narrower formats that reduce both footprint and bytes moved per token. The benefit is real only where your serving framework has an optimized path for that format on your model. |
| Vendor and software stack | AMD Instinct parts publish the largest memory capacities in this class, which is directly relevant to KV budget and batch size. Whether that converts into throughput on your model depends on framework and kernel maturity, which is a question to settle by testing. |
Platforms and Racks
At the top of the range the purchasing unit stops being an accelerator, which changes both what you compare and what the site has to deliver.
8-GPU platforms
The unit is a baseboard
What to know
- Memory and performance are published as platform totals across all eight accelerators
- Dividing a platform total by eight does not give a supported per-accelerator figure
- The pooled memory across the platform is what lets a large model serve with a large KV budget
- Networking bandwidth out of the platform constrains multi-node serving topologies
Rack-scale systems
The unit is a rack of 72 accelerators
What to know
- Published memory, bandwidth, and performance are whole-rack totals
- The in-rack interconnect is what makes 72 accelerators behave as one memory pool, which is the point for large-model serving
- It also enables serving topologies that separate prefill from decode across different accelerators
- These are liquid-cooled architectures, so the site has to be able to reject the heat
Reading an Inference Benchmark
A tokens-per-second figure is not a property of an accelerator. It is a property of a configuration, and the configuration moves the number more than the hardware does.
| Must be stated | Why it moves the number |
|---|---|
| Model and revision | Two models of the same parameter count can differ by nearly an order of magnitude in KV cost through their attention scheme alone. |
| Quantization of weights and of the KV cache | Changes both footprint and bytes moved per token, and can change output quality. |
| Serving framework and version | Continuous batching, paged attention, and speculative decoding change throughput by large factors on unchanged hardware. |
| Batch size or concurrency | Determines whether decode is bandwidth-bound or compute-bound, and trades directly against per-user latency. |
| Input and output length | Input length drives prefill cost and KV size; output length drives how many decode steps the result averages over. |
| The latency target it was measured under | A throughput figure with no latency constraint is an offline number and says nothing about interactive capacity. |
| System and accelerator count | An eight-accelerator result is not a per-accelerator result, and the interconnect is part of the system. |
| Whether it is offline or server scenario | Standardized inference benchmarks separate these deliberately, because they measure different things. |
SLYD does not republish specific submission results on this page. Reproducing them accurately for a moving target is a worse service than pointing at the record. Read the submissions directly at MLCommons MLPerf Inference, and treat any manufacturer performance claim as a manufacturer claim until you can tie it to a submission. The only benchmark that settles a decision is your own model, on your own traffic pattern, on the candidate configuration.
Cost per Token, Not Cost per Hour
Serving economics depend on how many tokens an hour of accelerator time actually produces at your latency target, which is a property of the whole configuration.
What sits between hourly cost and cost per token
- Achieved throughput at your latency target, not peak offline throughput
- Utilization across the day, since idle capacity is still paid for
- Prefill against decode mix, because long prompts and short answers cost differently
- Prefix and cache reuse, which can remove a large share of prefill work
- Whether the model fits on one accelerator, which removes communication overhead
Rent or own
- Rented capacity charges only for hours used, which suits spiky or early-stage traffic
- Owned capacity only amortizes at sustained utilization, and adds power, cooling, space, networking, and staffing
- Inference traffic is often steadier than training, which moves the crossover in favour of owning earlier than it does for training
- Data residency and latency to the user can decide it regardless of cost
Run it on your own numbers
Model the comparison with your own utilization, electricity rate, staffing, and horizon in the TCO calculator, and size the facility side with the power and cooling calculator. Current on-demand cloud rates and the shape of a break-even calculation are worked through in the training guide, and the same method applies here.
The figure to drive the decision is cost per million tokens at your latency target. An hourly rate on its own cannot be compared between two configurations that produce different numbers of tokens per hour.
Working Through the Decision
In order, because each answer constrains the next.
State the traffic pattern
Concurrent requests at peak, typical and maximum context length, typical output length, and the latency target for time to first token and for inter-token latency. Without these there is nothing to size against.
Size weights plus peak KV cache
Weight footprint at your quantization, plus KV cache per token times maximum context times peak concurrency, plus runtime overhead, plus headroom. This produces a capacity floor that eliminates most candidates immediately.
Decide whether it fits on one accelerator
If it does, you avoid tensor-parallel communication in the latency path and the decision is largely capacity, bandwidth, and software. If it does not, interconnect bandwidth becomes first-order and the purchasing unit may become a platform or a rack.
Choose the quantization deliberately
Take the memory saving as arithmetic, then measure the quality impact on your own evaluations before committing. Quantizing the KV cache often does more for concurrency than quantizing the weights does, and it is a separate decision.
Check the site can take it
Board power, server input power, rack design load, and heat rejection, for a deployment that runs continuously rather than in bursts. Air-cooled PCIe variants exist because many sites cannot take the module form factors.
Measure the shortlist on your traffic
Run your model, your quantization, your serving stack, and your traffic shape on the candidates, and compare cost per million tokens at your latency target. This is the only comparison that settles it.
Where the Numbers Live
This guide deliberately holds no specification table of its own.
GPU specifications database
18 NVIDIA and AMD accelerator records with memory, bandwidth, board power, precision performance, and form factor, each showing the manufacturer page it came from and the date it was checked.
Training, for comparison
Training memory budgets are dominated by optimizer state rather than KV cache, which is why the two workloads point at different hardware for the same model.
Facility power and cooling
Turn an accelerator count into server, rack, cooling, and facility input power for a deployment that runs continuously.
Rented capacity
On-demand GPU compute from SLYD marketplace providers, for traffic that does not yet justify owned capacity.
Frequently Asked Questions
Common questions about selecting hardware for inference
How much GPU memory does inference need?
Model weights plus the KV cache, and the KV cache is the part usually left out. Weights cost about 2 bytes per parameter at FP16 or about 0.5 bytes at 4-bit. The KV cache then costs a fixed amount per token of context, per concurrent request, and at long context or high concurrency it can exceed the weights. Size both, because the weights decide whether the model loads and the KV cache decides how many users you can serve at once.
Why is memory bandwidth not a reliable predictor of inference performance?
Because only part of inference is bandwidth-bound. Generating each token after the first reads the whole model from memory, so decode at low batch size does track bandwidth closely. Processing the prompt is compute-bound instead, and at high batch size decode becomes compute-bound too. Bandwidth sets a ceiling on one phase of one regime. Actual throughput and latency also depend on model architecture, quantization, serving software, batching policy, and system topology.
What is the difference between throughput and latency for inference?
They trade against each other. Throughput is total tokens per second across all requests and is maximized by large batches. Latency is what one user waits, split into time to first token, which is dominated by prompt processing, and inter-token latency, which is dominated by decode. Larger batches raise throughput and worsen per-user latency. A configuration tuned for offline throughput is not the configuration you want behind an interactive product.
Does quantization hurt model quality?
It can, and by how much depends on the model, the quantization method, and the task. The memory saving is arithmetic and predictable: 8-bit halves the weight footprint against FP16 and 4-bit quarters it. The quality impact is not predictable in advance and has to be measured on your own evaluations. Treat any claim of negligible loss as something to verify rather than assume, especially on reasoning, code, and long-context tasks.
Do I need a datacenter GPU for inference?
Not always. If a quantized model and its KV cache fit in a consumer or workstation card at your context length and concurrency, that card can serve it. Datacenter parts earn their place when you need HBM bandwidth, memory capacity beyond what GDDR parts offer, ECC for continuous operation, a high-bandwidth interconnect to split a model across accelerators, or the density to serve many concurrent users from one rack.
How should I read a published inference benchmark?
A tokens-per-second figure means nothing without its configuration. Check the model and its exact revision, the quantization, the serving framework and version, the batch size, the input and output lengths, the concurrency, the latency target the result was measured under, and the system including its accelerator count. Change any one of those and the number moves, often by more than the difference between two accelerators.
Methodology and limits
Where the memory figures come from
The weight footprints and KV cache figures are arithmetic from the formula shown on this page, not measurements and not manufacturer claims. The worked KV figures use 80 layers, a hidden size of 8,192, and FP16 at 2 bytes per element, stated so you can substitute your own model's parameters. They exclude framework and runtime overhead, which is real and varies by serving stack.
Quantized weight footprints are approximate because quantization carries a small overhead for scales and zero points that depends on the method and group size.
What this guide does not do
- It publishes no accelerator specifications. Those are in the GPU database with a manufacturer source and check date per record.
- It publishes no hardware prices or per-accelerator cloud rates. The previous version attached rate ranges to individual accelerators with no provider, region, instance shape, commitment, or date, so none of them could be checked.
- It republishes no benchmark results. A throughput figure without its model, quantization, framework, batch size, context lengths, concurrency, latency target, and system cannot be verified.
- It states no speedup multiplier for a quantization or an architecture generation. The memory reduction is arithmetic; the speedup depends on whether the kernel is memory-bound and on framework support.
- It does not characterize quantization quality loss. That is model, method, and task dependent and has to be measured.
What this guide is not
Planning guidance, not engineering. It does not replace the OEM system specification, a site survey, or measurement of your own model and traffic on candidate hardware. It is not accounting, tax, credit, or investment advice.
Sources and basis
What supports each part of this guide, and when it was last checked.
The KV cache formula, the per-token and 8k-context figures derived from it, and the weight footprints by precision.
Arithmetic from the formula and architecture parameters stated on this page. Not a measurement and not a manufacturer claim. Excludes framework and runtime overhead.
Which phase of inference is bandwidth-bound and which is compute-bound, how batching trades throughput against latency, and what quantization does and does not change.
Reviewed explanation of how transformer serving behaves. The direction of each effect is durable; the magnitude on your model is not, and has to be measured.
The class groupings, the decision framework, and the guidance on which differences matter for a serving workload.
SLYD's guidance. A way of approaching the decision rather than a sourced fact.
All accelerator specifications referenced by this guide.
This guide holds no specification table of its own so the two cannot disagree.
What has to match before two inference benchmark results can be compared, and the offline against server scenario distinction.
SLYD publishes no specific submission result on this page. Read the submissions directly.
MLPerf Inference datacenter benchmark results and submission detail
The absence of hardware prices, per-accelerator cloud rates, throughput figures, and quantization speedup multipliers.
Volatile commercial facts and configuration-specific results. SLYD publishes no governed public record for them, so this guide omits them rather than showing figures that cannot be checked.
Sizing an Inference Deployment?
Tell us the model, the context length, the concurrency at peak, and your latency target, and we will work through the options with you against a specific configuration.