llama.cpp server cheat sheet: GGUF models, flags, endpoints and metrics

llama-server is the HTTP server that comes with llama.cpp. It serves GGUF models through an OpenAI-compatible API. Use it for single-GPU or CPU hosts, edge boxes and dev machines where vLLM is too heavy. This page lists the commands you'll use day to day.

Tip

llama.cpp releases a new build almost daily, and flags get renamed or change their defaults. For your build, llama-server --help is the only reliable source. Pin the build number (for example b6xxx) in production, just as you would pin an image tag.

#Install and check the build

# macOS / Linux with Homebrew
brew install llama.cpp

# Build from source with CUDA
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# What did I get?
./build/bin/llama-server --version
./build/bin/llama-server --help | less

Official container images are published to ghcr.io/ggml-org/llama.cpp. The server tag is the CPU build, and there are GPU variants such as server-cuda:

docker run -d --name llm --gpus all -p 8080:8080 -v /srv/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/model-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 99

To confirm the GPU build is really using the GPU, look for the CUDA device lines and the "offloaded N/N layers to GPU" message in the startup log:

docker logs llm 2>&1 | grep -iE "cuda|offload"

#Get a model

# Download straight from Hugging Face (cached under LLAMA_CACHE or the default cache dir)
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

# Pick a specific quantization from a repo
llama-server -hf <org>/<repo>-GGUF:Q4_K_M

# Or download with the hf CLI and point at the file
hf download <org>/<repo>-GGUF <file>.gguf --local-dir /srv/models
llama-server -m /srv/models/<file>.gguf

For production, download once, record the file hash and serve with -m from local disk. Don't let every restart download from the internet.

sha256sum /srv/models/*.gguf > /srv/models/SHA256SUMS

#Launch flags you will actually use

Most flags can also be set as environment variables named LLAMA_ARG_* (for example LLAMA_ARG_CTX_SIZE). This is useful in Kubernetes manifests. Check --help for the exact names.

A sensible single-GPU production start:

llama-server -m /srv/models/model-Q4_K_M.gguf --alias chat-small \
  --host 0.0.0.0 --port 8080 -ngl 99 -c 32768 -np 4 -fa on \
  --api-key-file /etc/llama/keys --metrics

This gives 4 slots of about 8K tokens each. If users send long documents, use fewer slots with more context each.

#Endpoints and test calls

# Health: 200 when ready, 503 while the model is still loading
curl -s localhost:8080/health

# Model list (OpenAI-compatible)
curl -s localhost:8080/v1/models -H "Authorization: Bearer $KEY" | jq

# Chat
curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"chat-small","messages":[{"role":"user","content":"Say ok"}],"max_tokens":20}' \
  | jq '.choices[0].message.content, .usage, .timings'

# Tokenize: check how many tokens a prompt really is
curl -s localhost:8080/tokenize -H "Content-Type: application/json" \
  -d '{"content":"How many tokens is this?"}' | jq '.tokens | length'

# Server settings as loaded (context size, chat template, etc.)
curl -s localhost:8080/props -H "Authorization: Bearer $KEY" | jq

The timings object in responses gives prompt and generation tokens per second for that request. It's the quickest speed check you have.

From Python, any OpenAI SDK works:

from openai import OpenAI
client = OpenAI(base_url="http://llm-host:8080/v1", api_key=KEY)
r = client.chat.completions.create(model="chat-small",
        messages=[{"role": "user", "content": "Say ok"}], max_tokens=20)
print(r.choices[0].message.content)

#Slots: who is using the server right now

curl -s localhost:8080/slots -H "Authorization: Bearer $KEY" \
  | jq '.[] | {id, is_processing, n_ctx}'

When all slots are processing, new requests wait in a queue. If users report slow first tokens and every slot is busy, you need more slots or more replicas. A faster model won't help. Some builds disable /slots by default or let you turn it off with --no-slots, so check --help.

#Prometheus metrics

Start with --metrics, then scrape /metrics. Common series (names vary by version; check your build's output):

rate(llamacpp:tokens_predicted_total[5m])
max_over_time(llamacpp:requests_deferred[10m]) > 0

curl -s localhost:8080/metrics -H "Authorization: Bearer $KEY" | grep '^llamacpp:'

#Quantize, convert and benchmark

# Hugging Face safetensors -> GGUF (script in the repo root)
python convert_hf_to_gguf.py /path/to/hf-model --outfile model-f16.gguf --outtype f16

# Quantize to 4-bit
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Throughput test: 512-token prompt, 128 generated tokens
llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -ngl 99

Compare Q4_K_M and Q8_0 with llama-bench and with your own evaluation prompts before you pick one. Lower quantizations save VRAM, but you pay for it in answer quality.

#Run it as a service

# /etc/systemd/system/llama-server.service
[Unit]
Description=llama.cpp server
After=network-online.target

[Service]
User=llama
ExecStart=/usr/local/bin/llama-server -m /srv/models/model-Q4_K_M.gguf --alias chat-small \
  --host 127.0.0.1 --port 8080 -ngl 99 -c 32768 -np 4 --api-key-file /etc/llama/keys --metrics
Restart=on-failure
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now llama-server
journalctl -u llama-server -f

Warning

llama-server doesn't provide per-user access control, rate limiting or TLS that you should rely on for internet exposure. Bind to localhost or a private interface and put NGINX, a gateway or Open WebUI in front for authentication and TLS. An open port with no --api-key lets anyone on the network use your GPU.

#Symptom → cause → fix

#Prevent it next time

• Pin the llama.cpp build and the GGUF file hash, and upgrade both through staging.
• Always run with --api-key or --api-key-file and a non-public bind address.
• Enable --metrics and alert on deferred requests and on generation speed dropping.
• Record the exact launch command (or LLAMA_ARG_* values) in source control, not in shell history.