Determining VRAM Requirements for Local LLM Execution
When deploying local Large Language Models (LLMs), the primary consideration is whether the model fits within your GPU memory. This capacity depends on three key factors: model size, quantization level, and context length. This guide provides a practical framework for selecting the appropriate amount of VRAM.
The impact of quantization on VRAM usage
Quantization involves reducing the precision used to store model weights. Lower-bit quantization results in a smaller model footprint and reduced VRAM consumption, though this comes with a trade-off in quality.
| Quant | Bits per weight | Typical use |
|---|---|---|
| Q8_0 | 8 | Very high quality |
| Q6_K | ~6.6 | Very good quality |
| Q5_K_M | ~5.5 | Good quality and size |
| Q4_K_M | ~4.5 | Good balance of size and quality |
| Q3_K_M | ~3.5 | Lower VRAM, more quality loss |
In scenarios where VRAM is constrained, Q4_K_M is a popular choice. Conversely, if sufficient VRAM is available, using Q5 or Q6 allows the same model to be run with less aggressive quantization, preserving higher fidelity.
Estimated VRAM consumption by model size
The figures below are rough estimates for model weights alone. The total VRAM requirement will be higher, as the runtime, KV cache, and context processing also consume memory.
| Model size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
These values are estimates rather than strict limits, as different model architectures and quantization formats can influence the actual memory footprint.
Capabilities based on VRAM capacity
| VRAM | Practical range | Current examples |
|---|---|---|
| 8 GB | Small models around 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-sized models around 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models at lower quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models at higher quantization | 70B-class models at Q4 to Q6 |
These ranges apply to models where the weights can reside entirely on the GPU. Large MoE models operate differently: while only a subset of parameters is active for any given token, the model must still store its complete set of weights. Consequently, a model with over 100B total parameters cannot fit within a VRAM budget sized for 100B simply due to its lower active parameter count.
Understanding MoE models
Mixture-of-Experts (MoE) models consist of multiple parameter groups known as experts. Since only specific experts are activated per token, inference can be more efficient compared to dense models with the same total parameter count.
However, inactive experts still occupy memory space. Therefore, large MoE models often demand significantly more memory than their active parameter count implies. Very large models may necessitate multiple GPUs or offloading to system RAM.
Context length and VRAM consumption
Model weights represent only a portion of the total memory requirement. The KV cache expands as the context length increases, meaning that running the same model with a 64K context can demand substantially more VRAM than with a 4K context.
- Longer context lengths increase VRAM usage.
- The precision of the KV cache impacts memory consumption.
- Batch size and the number of concurrent users also contribute to memory load.
- It is advisable to reserve some VRAM for the runtime rather than filling the GPU entirely with model weights.
Practical recommendations
- Verify the actual size of the specific quantized model you intend to run.
- Avoid relying solely on the model file size as the exact VRAM requirement; allow space for the KV cache and runtime.
- If a model exceeds VRAM capacity, parts can be offloaded to system RAM, but this typically slows down inference.
- For long-context or agentic workloads, budget for more VRAM than the model weights alone require.
- Utilize multiple GPUs to distribute the model when a single GPU lacks sufficient VRAM.
Execute with DaDesktop
There is no need to purchase hardware to run a local LLM. DaDesktop provides a cloud desktop equipped with the necessary VRAM, allowing you to run models directly without owning the underlying hardware.
Select the VRAM tier that suits your model, load it, and begin usage immediately. This eliminates the need for setup, hardware purchases, or driver troubleshooting. Explore the available GPUs to see your options.