# llama.cpp Server — Overview

[llama.cpp](https://github.com/ggml-org/llama.cpp) is a pure C/C++ LLM inference library. The `llama-server` binary is a lightweight, OpenAI-compatible HTTP server for serving GGUF models via REST API.

See [Setup Guide](setup.md) for build instructions (CPU/CUDA/Metal/Vulkan), Docker, and systemd deployment.

---

## Key Features

- OpenAI API-compatible chat completions, completions, and embeddings
- Anthropic Messages API-compatible chat completions
- Parallel decoding with multi-user support and continuous batching
- Multimodal (vision/audio) support
- Speculative decoding with draft models
- Schema-constrained JSON output and function calling
- Built-in Web UI
- Prometheus metrics
- Router mode for multi-model deployment

---

## GGUF Model Format

**GGUF** (GGML Universal File) is the binary model file format used by llama.cpp.

### Quantization Methods

| Method | Bits/weight | 8B model size | Quality |
|--------|------------|---------------|---------|
| Q4_K_M | 4.9 | 4.58 GB | **Good (recommended default)** |
| Q5_K_M | 5.7 | 5.33 GB | High |
| Q6_K | 6.6 | 6.14 GB | Very High |
| Q8_0 | 8.5 | 7.95 GB | Near-lossless |
| F16 | 16.0 | 14.96 GB | Full precision |

**Q4_K_M is the recommended default** — best balance of quality and size.

---

## Quick Start

```bash
# Serve a local model on default port 8080
llama-server -m /path/to/model.gguf

# From Hugging Face (auto-downloads)
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
```

**Defaults**: Listens on `127.0.0.1:8080`. Single slot (one concurrent request).

> **System note:** On this network the fleet inference hosts do not use the default port. As of 2026-10 the only llama.cpp endpoint is `frank-zealot` on dankclaw **:7771** (PrismML fork build); `frank-dragoon` (dankclaw :7770, halogen-flash-server) and `frank-archon` (dankserver :7770, radiance) are different engines — see [System hosts](/system/index.md). The pi-era `frank-default-small`/`frank-default-large` llama.cpp pair (each :7770) is retired; [Pi Models Configuration](../../pi/models.md) is a pi-era archive.

---

## Health Check

```bash
curl http://localhost:8080/health
# 200: {"status": "ok"}
# 503: {"error": {"code": 503, "message": "Loading model"}}
```

---

## API Endpoints

### Chat Completions (OpenAI-compatible)

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Write a limerick about Python"}
    ],
    "max_tokens": 256,
    "stream": true
  }'
```

### Other Endpoints

```bash
# Completions
curl http://localhost:8080/v1/completions

# Embeddings
curl http://localhost:8080/v1/embeddings

# Native completion
curl -X POST http://localhost:8080/completion -d '{"prompt": "Hello", "n_predict": 64}'

# Anthropic-compatible
curl http://localhost:8080/v1/messages

# Monitoring
curl http://localhost:8080/props
curl http://localhost:8080/slots
curl http://localhost:8080/metrics
```

---

## Python Client

```python
import openai
client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-no-key-required"
)
completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "What is the meaning of life?"}],
    max_tokens=256
)
print(completion.choices[0].message.content)
```

---

## Docker

```bash
docker run -p 8080:8080 -v /models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/model.gguf --host 0.0.0.0 --port 8080

# CUDA variant
docker run -p 8080:8080 --gpus all \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/model.gguf --n-gpu-layers 99
```

---

## Environment Variables

All CLI options can be set via `LLAMA_ARG_` prefix:

```bash
LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server
```

---

## Cross-References

- See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment.
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for running Ornith models.