Embeddings in llama.cpp: Local Search Over Your Own Documents

For a model to answer based on your documents, it first has to be able to find them. That's what embeddings do: text turns into a set of numbers, and fragments close in meaning end up near each other. Search then happens by meaning, not by matching words.

llama.cpp can do this too, but the setup isn't quite like a regular chat. The general idea behind RAG is covered in a separate article — this one is about the specific implementation.

A separate model for embeddings

The first thing worth accepting: a regular chat model doesn't work for this. You need a specialized embedding model — they're noticeably smaller, run faster, and are trained specifically to turn text into a vector.

Second: it runs as a separate server instance. llama-server has a flag that switches it into this mode:

llama-server -m models/embedding-model.gguf --embeddings --port 8081

The documentation has a direct caveat about the flag: use it only with models actually built for embeddings. The mode restricts the server to that one task.

Why you end up with two servers

Here's the practical consequence that surprises people on their first setup: RAG needs two running processes. One holds the chat model and answers questions, the other computes embeddings for search.

llama-server -m models/qwen3-8b-q4_k_m.gguf --port 8080
llama-server -m models/embedding-model.gguf --embeddings --port 8081

Each takes its own memory. On a machine where the chat model already barely fits, adding a second process can be the last straw — the embedding model is small, but not free.

Endpoints

In embeddings mode, the server exposes two addresses: POST /v1/embeddings in OpenAI-compatible format, and POST /embedding in llama.cpp's native format. The first one is more convenient, since off-the-shelf libraries connect to it without modification. The server's other endpoints are covered in the article on llama-server.

Reranking

There's another flag people usually learn about late:

llama-server -m models/reranker-model.gguf --reranking --port 8082

It turns on the POST /reranking endpoint. Reranking's job is to pick out which of the retrieved fragments are actually relevant.

Why this matters: embedding-based search finds things that are similar, but "similar" and "actually answers the question" aren't the same thing. A reranker model looks at the pair "question — fragment" more carefully and re-sorts the results. In practice this noticeably improves answers, especially when there are lots of similar-looking documents.

The price is predictable: a third model, a third process.

What you'll have to write yourself

Worth pausing here, because llama.cpp only gives you the bricks.

The server computes embeddings. It doesn't read your PDFs, doesn't chunk them, doesn't store the vectors, doesn't search them, and doesn't assemble what it found into a prompt. All of that is code you write: parsing documents, chunking with overlap, a vector database, the query, assembling context, updating the index when files change.

That's a week of work for someone who's done this before, and noticeably more for someone who hasn't. What you get back is tailored exactly to your task — which is the point, if the task is non-standard.

When it's not worth building yourself

If the task is ordinary — "I want to ask questions about my documents" — assembling a pipeline of three servers and your own code isn't necessary. Ready-made apps give you the same result: some desktop apps close the "chat over documents" scenario with almost no setup, and some knowledge-base tools ship their own retrieval built in.

And if the documents aren't for a conversation but for actual work — pulling together a summary, filling in a spreadsheet, preparing a document from a template — RAG might be an unnecessary layer entirely. An index matters when there are a lot of documents and you don't know in advance where to look. When files sit in a specific folder, it's simpler for an agent to just open them directly.

Doka works exactly that way: it reads files where they are and does what you asked with them, no upfront loading into a database. A local model installs from the interface, and working with documents and spreadsheets doesn't require standing up three servers and writing a pipeline.

So llama.cpp with embeddings is the right choice if you're building your own search system and know why you need it. For everything else, a ready-made agent saves you that exact week.