llama-cpp-python: A Local Model From Python in Ten Minutes

If you need a local model inside your own code rather than someone else's app, llama-cpp-python is usually the shortest path there. These are Python bindings for llama.cpp: the same engine, the same GGUF files, but everything gets called from a script.

One detail ruins the first impression for about half of everyone who tries it, and it's better to know about it up front.

Installation

The basic command looks harmless:

pip install llama-cpp-python

Here's where the detail hides: by default, the package builds llama.cpp from source. On a machine without a compiler, the install fails; on a machine with one, it takes a noticeable amount of time. If the build breaks, add --verbose — otherwise the reason stays hidden.

Second point: a version built this way runs on the CPU. A GPU doesn't turn itself on.

How to enable a GPU

Two paths.

Set build flags through an environment variable:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

Or grab a prebuilt wheel for your CUDA version and skip building entirely:

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

Swap cu121 for your own version — builds exist for several CUDA releases. The second path is faster and breaks less often, so start there.

A model straight from Hugging Face

You don't have to download the file by hand. The from_pretrained method pulls a GGUF from a repository by filename pattern (you'll need huggingface-hub installed):

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="lmstudio-community/Qwen3.5-0.8B-GGUF",
    filename="*Q8_0.gguf"
)

Handy for experimenting: swap the model string and compare. What tags like Q8_0 mean and which to pick is covered in the GGUF format breakdown.

Your own server, one command

If you need an address other programs can connect to, rather than an object inside your code, the package has a server mode:

pip install 'llama-cpp-python[server]'
python3 -m llama_cpp.server --model models/7B/llama-model.gguf

API docs open at http://localhost:8000/docs.

This is essentially an alternative to the ready-made llama-server from llama.cpp's own builds. The difference is that everything here lives inside a Python environment, so it plugs more easily into an existing project and is managed by the same dependencies. If you don't need Python glue code, the regular llama-server is simpler: no pip, no build.

What to pick

Task	Tool
The model gets called from your own Python code	llama-cpp-python
You just need a local API for other programs	llama-server from the official builds
You want a chat and to switch models by clicking	LM Studio or Jan
You need an agent that does the work	see below

About the glue code everyone ends up writing

The same story tends to play out. First a twenty-line script: load the model, ask a question, print the answer. Then file reading shows up, because the questions are about documents. Then answer parsing, because the model answers in text and you need the result in a file. Then error handling, retries, running it on a schedule.

A month later, it turns out you're writing an agent. That's a normal path and useful experience, but it's worth comparing it to a ready-made solution at least once before sinking another three evenings into it.

Doka is that agent, already assembled: working with files and the terminal, carrying a task through several steps, MCP servers, and a local model that installs from the interface. Your Python code doesn't go anywhere in the process — the agent can run it, so your scripts become tools instead of getting replaced.

A reasonable test: take the task you were about to write another script for, and run it through the agent instead. If it handles it, you don't have to write the script. If it doesn't, llama-cpp-python isn't going anywhere.