AI Answer Library
Short answer
Size everything backwards from VRAM; the rest is secondary. The rule of thumb is: weight memory ≈ parameter count × bytes per parameter × 1.2, where FP16 is about 2 bytes per parameter and INT4 about 0.5 — then add KV cache on top, which grows linearly with concurrency and context length. A 70B dense model therefore needs roughly 150 GB of VRAM at FP16 (multi-GPU, unavoidably) and can be squeezed to roughly 40 GB at INT4. CPU-only inference works, but only for one user at low frequency.
The table uses empirical values from common open-source serving stacks as of August 2026 and counts weight memory only, excluding KV cache. Use it for a first pass; real selection requires load testing at your target concurrency and context length, because at 128K context under high concurrency the KV cache can exceed the weights themselves.
| Model scale | FP16 weight memory (approx.) | After INT4 quantisation (approx.) | Typical single-machine setup | Fits |
|---|---|---|---|---|
| 7B–8B (small tiers of Llama 4 and Qwen 3.x) | About 16 GB | About 6 GB | A single 24 GB consumer or workstation card | Departmental pilots, summarisation, lightweight RAG |
| 14B–32B (mid tiers of Qwen 3.x, distilled GLM-5.x) | About 32–70 GB | About 10–20 GB | One or two 48 GB workstation cards | Enterprise RAG, agent assist, structured extraction |
| 70B dense (the Llama 4 dense line and peers) | About 150 GB | About 40 GB | A 4–8 GPU chassis with 48–80 GB per card | Company-wide primary inference, coding assistants, harder reasoning tasks |
| Trillion-parameter MoE (full DeepSeek V4, full GLM-5.x) | Hundreds of gigabytes up to terabyte scale | Still hundreds of gigabytes | A multi-node, multi-GPU cluster with high-speed interconnect | Customer-facing high-concurrency workloads, shared multi-team platforms |
Two parts. Weights: parameter count × bytes per parameter, times roughly 1.2 of headroom for activations and framework overhead. FP16 is 2 bytes per parameter, INT8 about 1, INT4 about 0.5. For example a 32B model needs about 32 × 2 × 1.2 ≈ 77 GB at FP16 and about 32 × 0.5 × 1.2 ≈ 19 GB at INT4. KV cache: it grows linearly with concurrent requests × context tokens, and under long context or heavy concurrency it easily exceeds the weights. In practice, pick cards from the weight figure, then treat leftover memory as the dial that sets your concurrency ceiling — when memory runs short, cut concurrency or context length before you downgrade the model.
System RAM should be at least as large as total VRAM, since weights pass through it on load. Use NVMe storage: a single model file runs from tens to hundreds of gigabytes, and once you add the vector store, embedding models and a rollback checkpoint, 1–2 TB is the starting point. On multi-GPU boxes, inter-card bandwidth matters — the same eight cards behave very differently over PCIe versus a high-speed fabric, especially at long context. Power and cooling are routinely underestimated: a loaded multi-GPU chassis draws kilowatts and a standard office rack usually cannot take it, so schedule facility work early. Finally, the network: in a RAG deployment the model is one component among several, and the vector store, document service and auth gateway need planning too — do not discover after delivery that there is nowhere to put the database.
Where this applies
People also ask
How much does it cost for a company to self-host a large language model?
Self-hosted LLM or public API — how do I choose?
Should an enterprise knowledge base use RAG or a fine-tuned LLM?
Ygg Lab Token — API Quota Gateway
Every model call, clear and in control — nine upstream provider types behind a single sk- key, with quota, logs and billing you own.
YggLab AI Agent Desktop
One desktop where 20+ CLI agents actually collaborate — assign work solo, form a team, or command them from your phone while work keeps moving 24/7.