command-center/omp-docs/tools/llama-cpp/index.md

llama.cpp Server — Overview

llama.cpp is a pure C/C++ LLM inference library. The llama-server binary is a lightweight, OpenAI-compatible HTTP server for serving GGUF models via REST API.

See Setup Guide for build instructions (CPU/CUDA/Metal/Vulkan), Docker, and systemd deployment.

---

Key Features

• OpenAI API-compatible chat completions, completions, and embeddings
• Anthropic Messages API-compatible chat completions
• Parallel decoding with multi-user support and continuous batching
• Multimodal (vision/audio) support
• Speculative decoding with draft models
• Schema-constrained JSON output and function calling
• Built-in Web UI
• Prometheus metrics
• Router mode for multi-model deployment

---

GGUF Model Format

GGUF (GGML Universal File) is the binary model file format used by llama.cpp.

Quantization Methods

Method	Bits/weight	8B model size	Quality
Q4_K_M	4.9	4.58 GB	Good (recommended default)
Q5_K_M	5.7	5.33 GB	High
Q6_K	6.6	6.14 GB	Very High
Q8_0	8.5	7.95 GB	Near-lossless
F16	16.0	14.96 GB	Full precision

Q4_K_M is the recommended default — best balance of quality and size.

---

Quick Start

# Serve a local model on default port 8080
llama-server -m /path/to/model.gguf

# From Hugging Face (auto-downloads)
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Defaults: Listens on 127.0.0.1:8080. Single slot (one concurrent request).

System note: On this network the fleet inference hosts do not use the default port. As of 2026-10 the only llama.cpp endpoint is frank-zealot on dankclaw :7771 (PrismML fork build); frank-dragoon (dankclaw :7770, halogen-flash-server) and frank-archon (dankserver :7770, radiance) are different engines — see System hosts. The pi-era frank-default-small/frank-default-large llama.cpp pair (each :7770) is retired; Pi Models Configuration is a pi-era archive.

---

Health Check

curl http://localhost:8080/health
# 200: {"status": "ok"}
# 503: {"error": {"code": 503, "message": "Loading model"}}

---

API Endpoints

Chat Completions (OpenAI-compatible)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a limerick about Python"}
    ],
    "max_tokens": 256,
    "stream": true
  }'

Other Endpoints

# Completions
curl http://localhost:8080/v1/completions

# Embeddings
curl http://localhost:8080/v1/embeddings

# Native completion
curl -X POST http://localhost:8080/completion -d '{"prompt": "Hello", "n_predict": 64}'

# Anthropic-compatible
curl http://localhost:8080/v1/messages

# Monitoring
curl http://localhost:8080/props
curl http://localhost:8080/slots
curl http://localhost:8080/metrics

---

Python Client

import openai
client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-no-key-required"
)
completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "What is the meaning of life?"}],
    max_tokens=256
)
print(completion.choices[0].message.content)

---

Docker

docker run -p 8080:8080 -v /models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/model.gguf --host 0.0.0.0 --port 8080

# CUDA variant
docker run -p 8080:8080 --gpus all \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/model.gguf --n-gpu-layers 99

---

Environment Variables

All CLI options can be set via LLAMA_ARG_ prefix:

LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server

---

Cross-References

• See Setup Guide for build instructions, GPU offloading, and deployment.
• See Ornith Setup for running Ornith models.