llama.cpp on CPU: Running a Local Model Without a Graphics Card

llama.cpp was built for exactly this scenario from the start: running a model where there's no expensive graphics card. It still works today — the only question is which model to pick and what speed to expect.

The answer to the speed question usually disappoints less than people assume, but it depends on something other than what most people think about first.

What to download

You need a CPU build: an archive like llama-<number>-bin-win-cpu-x64.zip from the releases page. It needs no CUDA, no drivers — unpack it into a folder and it works.

There's a separate article on the other build variants and picking one for your hardware.

Running it is no different:

llama-server.exe -m models/qwen3-4b-q4_k_m.gguf --port 8080

The -ngl flag isn't needed on a CPU build — there's nothing to offload layers to.

What actually limits speed

This is where the biggest misunderstanding on the topic lives. People look at core count and clock speed, but the bottleneck is usually memory.

Text generation works so that for each next token, the model's weights have to be read from RAM. A 4-gigabyte model means 4 gigabytes of reading per token. The CPU finishes computing faster than memory can deliver the data, and sits idle.

That leads to practical consequences that break the usual intuition:

• fast memory gives a bigger boost than a more powerful CPU;
• dual-channel mode beats single-channel noticeably, and that's about how your RAM sticks are seated;
• adding cores past a certain point changes almost nothing;
• a smaller model speeds up generation nearly linearly, because there's less to read.

Which model to pick

Simple rule: on a CPU, size matters more than it does anywhere else.

Models in the 3–4 billion parameter range at Q4_K_M quantization are a reasonable starting point. They give a few tokens per second on an ordinary laptop — text shows up slower than a person reads, but it's usable.

7–8 billion parameter models run noticeably slower on a CPU. Tolerable for short answers, a real test of patience for long ones.

Anything bigger only makes sense without a GPU if you're willing to wait minutes. What quantization tags mean and why Q4_K_M is the default choice is covered in the article on the GGUF format.

What speed to expect

I won't give exact numbers — they depend too much on memory, CPU generation, and settings. The rough shape: a small model on a modern laptop gives a speed where chat feels slow but usable. A medium model turns every answer into a tea break.

Measuring your own setup takes five minutes: one model, the same question, watch the tokens per second. That's more honest than anyone else's tables, because your memory and your CPU never showed up in those tables.

Where a CPU stops keeping up

A chat conversation on a CPU is doable. Agentic scenarios are much worse, and not because of raw speed.

An agent takes many steps, and on each one it reprocesses the accumulated context: tool descriptions, history, file contents. What feels "slow" in a chat gets multiplied by the number of steps in a multi-step task. On top of that, the small model you're forced into for speed's sake holds onto the goal worse and gets tool arguments wrong more often.

So the honest answer to "can I run a local agent without a GPU" is: you can try, but start with simple tasks and short contexts. What a GPU gets you, and how much memory to budget, is covered separately.

If there's no GPU but you need a result

There's a middle path that's often overlooked. A local model and a cloud one aren't mutually exclusive — you can keep a local model for everything ordinary and reach for a cloud one on your own key for rare, heavy tasks.

Doka works exactly this way: a local GGUF model installs from the interface and matches your hardware, and a cloud model can be connected separately for when the local one isn't enough. For a machine with no GPU, that's more sensible than committing to one option: everyday work runs local and free, and the heavy stuff doesn't mean a ten-minute wait.

The easiest way to start is a small model on your own real tasks — download it and see exactly where your machine hits its limit. Practice tells you more here than any calculation.