Running a local LLM without a GPU
Smaller quantized models can run locally without a discrete GPU. Whether the experience works for you depends on available memory, processor performance, model size, and the time you can tolerate before a response begins.
Updated July 11, 2026
What your RAM can run
The model, context, and operating system must all fit in memory. Start with a smaller quantized model and avoid relying on swap for active inference.
| RAM | Run this | Good for |
|---|---|---|
8 GB older laptops | Small quantized models | Light tasks and quick drafts |
16 GB typical machines | Small and some medium quantized models | A practical starting point for local chat |
32 GB good desktops | Larger quantized models or hybrid workloads | More room for models and context |
64 GB+ servers, old Xeons | Larger models and batch workloads | Useful when capacity matters more than responsiveness |
Memory bandwidth sets the speed limit
Generating a token repeatedly reads model data from memory, so memory bandwidth affects CPU inference. Memory configuration and processor design vary widely; use the configuration supported by your machine and benchmark the model you intend to run.
Thread settings also matter. Start with your runner's defaults, then tune them against a representative prompt instead of assuming every available core improves throughput.
The wait before the first word
Generation speed is not the only delay. Before the first token appears, the runner processes the prompt, and long prompts can make that wait noticeable.
Keep contexts appropriate to the task and reserve long-document work for batch jobs when responsiveness matters. On laptops, monitor sustained performance and thermal behavior during longer runs.
The MoE trick that changed CPU-only
Mixture-of-experts models are another option to test without a discrete GPU. Their memory needs and practical responsiveness still depend on the model format, runner, and host machine.
If you already have a desktop with ample memory, test a supported quantized model before buying dedicated hardware.
The tools that matter on CPU
A CPU-capable inference engine with documented backends and runtime options.
Runs on CPU automatically when there is no GPU to find, with none of the silent-fallback confusion that page's GPU owners deal with. The comfortable choice if you'd rather not compile anything.
llama.cpp packed into a single executable that runs on macOS, Linux, and Windows unchanged. The shortest path to a working CPU chat on a machine you don't administer.
When CPU-only is the right call
The strongest case is hardware you already own plus work that tolerates latency: private scripts over personal documents, overnight batch summarization, or an always-on homelab assistant. Measure your own machine's power use if operating cost is part of the decision.
If you are buying hardware anyway, compare a discrete GPU with a CPU-only setup using the models and responsiveness you need. The details are on the 8 GB VRAM page. Hosted APIs can be more economical for light use; compare current hardware, electricity, and service costs for your expected workload. Privacy and control can be stronger reasons to stay local. For assembling the interface and model layers on top of whichever hardware wins, see the self-hosted ChatGPT stack.
Questions people actually ask
Can I run a large model without a GPU?
A model must fit in available memory alongside the operating system and context. Consult the model format and runner documentation before attempting a large model on CPU.
Do more CPU cores make it faster?
CPU performance depends on the processor, memory configuration, model format, and runner settings. Benchmark thread settings on your own workload.
Which small model should I start with?
Start with a small model whose model card and format are supported by your runner. Compare it on the prompts and languages that matter to your work.
Does a Mac count as running without a GPU?
Apple Silicon is supported by Ollama on supported macOS systems. Assess memory capacity and the model requirements for the machine you plan to use.
Sources
Every tool above is one row in the Awesome Open Source AI registry, which resyncs twice a day from the curated GitHub list. The Inference Engines & Serving category holds more CPU-capable engines, including MLC-LLM and LocalAI.
