Running local LLMs on 8 GB of VRAM
An 8 GB GPU can run smaller quantized models locally. The practical limit depends on the model, context, KV cache, runner, and other applications using GPU memory.
Updated July 11, 2026
Models to try
Start with a quantized model and monitor memory use as you increase context. Model files, KV cache, and runner overhead all share the available VRAM.
| Model | Notes |
|---|---|
Qwen3 8B Q4_K_M | The community default. Strongest at coding for its size. |
Llama 3.1 8B Q4_K_M | Polished general chat and instruction following. |
Mistral 7B Q4_K_M | The fast option when you want snappier turnaround. |
Gemma 4B Q4_K_M–Q8 | Leaves room for long context; some variants take images. |
Phi-4 Mini 3.8B Q4–Q8 | The best reasoning per gigabyte in the small class. |
Qwen3-30B-A3B (MoE) Q4, hybrid CPU/GPU | An option to test when your system memory can support hybrid inference. |
Where the 8 GB actually goes
The budget has three lines: model weights, KV cache, and runtime overhead. The model format and quantization determine the first; the context setting affects the second. Leave headroom rather than trying to fill all available VRAM.
Before changing models, close GPU-hungry applications and inspect available memory with your platform's monitoring tools. llama.cpp documents options for layer placement, flash attention, and KV-cache formats; test those options one at a time with the model and context you use.
The silent slowdown to check first
A model can load while some work is still handled outside the GPU. That can make performance feel unexpectedly slow, so check how your runner placed the model before changing everything else.
Run ollama ps after loading to inspect placement. If the result does not suit your setup, reduce the model or context, free GPU memory, or use a runner that exposes placement controls.
The 14B temptation, and the MoE loophole
Partial offload can make a larger model usable, but it adds a CPU-memory dependency and may not be the best experience. Compare it with a smaller model that fits fully on the GPU using your own prompts.
Mixture-of-experts models can be worth testing with hybrid CPU/GPU inference. llama.cpp documents the relevant backend and placement options. Expect the trade-off to depend on both system memory and the model's behavior on your tasks.
The runners for tight VRAM
The easiest start. Use “ollama ps” after loading a model to inspect how it is placed.
The engine underneath most of the ecosystem. Running it directly gives you the flags that matter at 8 GB: -ngl for layer placement, -fa for flash attention, and quantized KV cache types.
A llama.cpp wrapper that shows you exactly which layers landed on the GPU. Preferred by people who got burned by silent offload once and don't want it happening again.
When to stop tweaking and upgrade
Upgrade when the model quality, context capacity, or responsiveness no longer suits your work. For casual chat, an 8 GB setup may be enough; coding assistance and larger models can make its limits more apparent.
If you regularly need more memory, consider hardware with additional VRAM or a system with more unified memory. For going fully without a discrete GPU, see running an LLM without a GPU. And for what to put on top of the runner once it works, the self-hosted ChatGPT stack page covers interfaces and models.
Questions people actually ask
Can I run a larger model on limited VRAM?
Check the model card and quantization documentation before trying a larger model. Available memory also depends on context settings and the inference backend.
Will a laptop GPU behave like a desktop GPU?
Do not assume a laptop and desktop GPU with the same memory will behave identically. Test the intended runner and model on the machine you will use.
How much context can I afford?
Context consumes memory in addition to model weights. Start with a conservative context setting and increase it while monitoring memory use.
Do AMD cards work?
llama.cpp and Ollama document supported GPU backends. Confirm backend support for your operating system and GPU before selecting a runner.
Sources
Every runner above is one row in the Awesome Open Source AI registry, which resyncs twice a day from the curated GitHub list. The Inference Engines & Serving category holds the rest of the local-inference toolbox, from ExLlamaV2 to MLC-LLM.
