Running Qwen locally, size by size
Qwen3 is the most complete size ladder in open weights: 0.6B to 235B, dense and mixture-of-experts, Apache-2.0 on the sizes you would actually run at home. An 8 GB card takes the 8B, a 24 GB card takes the 32B dense or the 30B-A3B MoE, and the interesting decisions live between those two. This page maps the ladder, then covers the thinking-mode and tool-calling quirks that dominate real usage reports.
Updated July 12, 2026
The size ladder
| Model | VRAM | Hardware | Notes |
|---|---|---|---|
| Qwen3 0.6B–4B | ~1–3 GB (Q4) | Anything, including phones | The 4B runs on an iPhone at 5–12 t/s |
| Qwen3 8B | ~5–6 GB (Q4) | 8–12 GB cards, 16 GB Macs | The community default for small rigs |
| Qwen3 14B | ~9–10 GB (Q4) | 12–16 GB cards | Comfortable middle of the dense line |
| Qwen3 32B dense | ~18–20 GB (Q4) | 24 GB (RTX 4090) | 40–80+ t/s on a 4090 |
| Qwen3-30B-A3B (MoE) | ~19–24 GB (Q4) | 24 GB cards, hybrid CPU/GPU | Only ~3B active per token; can outpace the dense 32B |
| Qwen3-235B-A22B (MoE) | Hundreds of GB | Not consumer hardware | The flagship stays in the datacenter |
Q4_K_M figures at modest context. Q8 roughly doubles the weight size, and KV cache grows with context on top of the weights; llama.cpp's quantized KV cache option roughly halves that cost for long sessions.
The MoE trick, and its catch
The 30B-A3B is the family's clever move: 30B parameters on disk, 128 experts inside, and only about 3B active for any given token. All the weights still have to load, so it needs a 24 GB card or a hybrid CPU/GPU split, but the compute per token is small, which is why it can generate faster than the dense 32B while matching it on many tasks, with reports of 100–200 tokens per second on well-tuned setups. Newer family revisions extended the same pattern.
The catch surfaces in long agentic sessions. Practitioners consistently report the low active-parameter count showing up as lost context and repetition on multi-step work, where a dense 27B or 32B stays coherent longer. Speed is not free.
Thinking mode is the real learning curve
Qwen3 toggles chain-of-thought reasoning per request: a global enable_thinking flag, or /think and /no_think inside a conversation. It works, and on hard problems the larger models genuinely benefit. The plumbing is where the complaints concentrate. Leaked <think> tags accumulating in multi-turn context were still an open llama.cpp issue in mid-2026, and tool calls emitted after a thinking block get misparsed by several backends, leaking XML as plain text instead of a structured call.
The field-tested configuration is unglamorous: thinking off for agentic and tool-calling work, repeat_penalty at 1.0, community quants with fixed chat templates, and thinking turned on only for the turns that need deep reasoning. Set up that way, the models are dependable; set up naively, the first week is spent debugging tags.
The runners
Native qwen3 tags for every consumer size. Supports the /think and /no_think per-turn toggles, and handled thinking-tag cleanup better after v0.17.
The flexible option: MoE offloading, quantized KV cache for long context, Metal on Macs. Also where leaked thinking tags were still an open issue as of mid-2026.
For serving Qwen to more than one user. Prefers models that fit fully on the GPU; use --tool-call-parser designed for Qwen when doing agentic work.
On Macs, MLX-based serving tends to lead llama.cpp for Qwen throughput, and the 8B class is a comfortable fit for the 8 GB VRAM tier.
Qwen or a DeepSeek distill?
The R1 distills run on Qwen bodies, so the choice is really between Qwen's own instruction tuning and DeepSeek's reasoning-first training. Native Qwen wins on breadth: multilingual work, coding variety, and tool use in ordinary flows. The distills win when the task is a hard reasoning or math problem and nothing else matters. Licensing tilts the same way for derivative-builders, MIT on the R1 line against Apache on Qwen, both clean. Details and hardware fits are in the DeepSeek local guide, and the license fine print in commercially usable AI.
Questions people actually ask
Thinking mode: on or off?
Off for tool calling and agentic work, on for hard single-shot reasoning on the larger models. Practitioner reports through early 2026 are consistent: with thinking enabled, tool calls leak as plain text or get misparsed, and smaller models can loop. The per-turn toggles (/think and /no_think in Ollama) let you pay the reasoning cost only on the turns that need it.
Which size for an 8 GB card?
The 8B at Q4_K_M, which needs about 5–6 GB and leaves room for context. If your system RAM is generous, the 30B-A3B MoE with hybrid CPU/GPU placement is worth testing against it; sparsity means only about 3B parameters are active per token, so it runs faster than its size suggests.
Is Qwen really Apache-licensed?
The sizes you would run locally, yes: the dense line and consumer MoE variants ship under plain Apache-2.0. Some flagship releases have historically used a custom license with a 100M monthly-active-user clause and attribution requirements instead, so check the LICENSE file on the specific Hugging Face repo rather than assuming the family rule. The commercially usable AI page covers what the custom terms actually require.
MoE or dense at the same memory budget?
Try both; they fail differently. The MoE is faster and often matches the dense model on general tasks, but practitioners report the low active-parameter count shows on long agentic sessions, where dense 27B–32B models hold together better. If your use is chat and single-file coding, the MoE's speed usually wins; for multi-step tool work, dense is the safer default.
Sources
Every runner above is one row in the Awesome Open Source AI registry, which resyncs twice a day from the curated GitHub list, including Qwen3-VL and the coder variants in the model families category.
