llama-server: A Local OpenAI-Compatible API on Your Own Computer

Almost every AI-facing program knows how to talk to the OpenAI API. That turned out to be a convenient coincidence: stand up a server that speaks the same language, and all those programs can switch to a local model by changing one address.

llama-server, part of llama.cpp, does exactly that. Here's what it exposes and which settings are worth setting before connecting anything that matters to it.

Starting it

The server ships in llama.cpp's prebuilt releases — nothing to compile. Picking a build for your hardware is covered in the article on llama.cpp on Windows.

Minimal command:

llama-server -m models/qwen3-8b-q4_k_m.gguf --port 8080

By default, the server listens on 127.0.0.1:8080 and serves a web UI on the same address. --no-ui turns the interface off if you don't need it.

Which endpoints are available

The OpenAI-compatible part — the whole point of this:

Endpoint	Purpose
POST /v1/chat/completions	chat, the main mode
POST /v1/completions	text continuation
POST /v1/embeddings	embeddings

Alongside these live llama.cpp's own native endpoints: POST /completion and POST /embedding in native format, POST /reranking for reranking (enabled with the --reranking flag), plus utility endpoints GET /health, GET /props, and GET /slots.

The base address for clients ends up being http://127.0.0.1:8080/v1. From there, wherever a program's settings ask for an "OpenAI API base URL," you plug that in, and it starts working with your local model. Ollama and LM Studio follow the same pattern, just on different ports.

Settings worth setting right away

--ctx-size N — context size. It's taken from the model by default, but for agentic scenarios you almost always end up setting it by hand.

--parallel N — the number of slots, meaning simultaneous requests. If a code editor and a separate client are both connected to the server, one slot isn't enough and requests will queue up.

--jinja — enables OpenAI-style function calling. Without it, tools won't work.

--embeddings — switches the server into embeddings-only mode. Needed if you're standing up a separate instance for local RAG with a dedicated model.

About --host 0.0.0.0

This is the biggest trap on the topic, so it gets its own section.

Guides often suggest changing the address to 0.0.0.0 to connect from another device. It works. But 127.0.0.1 isn't the default just for looks — it means the server is only reachable from this machine.

Change the address, and you open the model up to anyone who can reach the computer over the network. Without additional measures, that's an open, password-free endpoint: a stranger can send requests, occupy your GPU, and read the answers.

If outside access is genuinely needed, the server has authentication built in:

llama-server -m model.gguf --host 0.0.0.0 --api-key mysecretkey

You can set several keys separated by commas, or move them into a file with --api-key-file.

What people actually do with this

Three typical scenarios.

First: connect a local model to a code editor instead of a cloud provider. Second: move your own scripts off a paid API onto a local one — the code barely changes, just the base address. Third: keep one server running on a home machine and reach it from a laptop — but that's exactly the case where you need both --api-key and a clear understanding of what you're doing with the network.

Where the server's usefulness ends

llama-server solves the delivery problem: the model exists, it answers, you can connect to it. Everything that happens after the answer, it doesn't do.

That leads to a familiar trajectory. Someone stands up a server, connects a couple of clients, and a week later finds themselves writing glue code: a script that gathers files into a prompt, another that parses the answer, a third for running things on a schedule. They've essentially started writing an agent on top of their own server.

Doka is that agent, already written. It works with files and the terminal directly, carries a task through several steps, and connects MCP servers. You can install a local model right from the interface, or keep your own llama-server and point Doka at it — the server doesn't go anywhere in this setup, it just stops being the only thing you have.

So standing up llama-server almost always makes sense. The question is what you're going to connect on top of it, and whether it's worth writing that yourself. Download Doka and see how much of your glue code turns out unnecessary.