Frank
3713339ffc
Uncommitted machine/session edits (live config, models, docs drift) were blocking the merge of origin/main. Committed as-is to preserve them; the merge that follows resolves overlapping prose files toward upstream canonical, with local variants recoverable from this commit. Pre-merge copies also in .AGENTTRASH/pre-merge-2026-10-18/.
156 lines
4.1 KiB
Markdown
156 lines
4.1 KiB
Markdown
# llama.cpp Server — Overview
|
|
|
|
[llama.cpp](https://github.com/ggml-org/llama.cpp) is a pure C/C++ LLM inference library. The `llama-server` binary is a lightweight, OpenAI-compatible HTTP server for serving GGUF models via REST API.
|
|
|
|
See [Setup Guide](setup.md) for build instructions (CPU/CUDA/Metal/Vulkan), Docker, and systemd deployment.
|
|
|
|
---
|
|
|
|
## Key Features
|
|
|
|
- OpenAI API-compatible chat completions, completions, and embeddings
|
|
- Anthropic Messages API-compatible chat completions
|
|
- Parallel decoding with multi-user support and continuous batching
|
|
- Multimodal (vision/audio) support
|
|
- Speculative decoding with draft models
|
|
- Schema-constrained JSON output and function calling
|
|
- Built-in Web UI
|
|
- Prometheus metrics
|
|
- Router mode for multi-model deployment
|
|
|
|
---
|
|
|
|
## GGUF Model Format
|
|
|
|
**GGUF** (GGML Universal File) is the binary model file format used by llama.cpp.
|
|
|
|
### Quantization Methods
|
|
|
|
| Method | Bits/weight | 8B model size | Quality |
|
|
|--------|------------|---------------|---------|
|
|
| Q4_K_M | 4.9 | 4.58 GB | **Good (recommended default)** |
|
|
| Q5_K_M | 5.7 | 5.33 GB | High |
|
|
| Q6_K | 6.6 | 6.14 GB | Very High |
|
|
| Q8_0 | 8.5 | 7.95 GB | Near-lossless |
|
|
| F16 | 16.0 | 14.96 GB | Full precision |
|
|
|
|
**Q4_K_M is the recommended default** — best balance of quality and size.
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Serve a local model on default port 8080
|
|
llama-server -m /path/to/model.gguf
|
|
|
|
# From Hugging Face (auto-downloads)
|
|
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
|
|
```
|
|
|
|
**Defaults**: Listens on `127.0.0.1:8080`. Single slot (one concurrent request).
|
|
|
|
> **System note:** On this network the fleet inference hosts do not use the default port. As of 2026-10 the only llama.cpp endpoint is `frank-zealot` on dankclaw **:7771** (PrismML fork build); `frank-dragoon` (dankclaw :7770, halogen-flash-server) and `frank-archon` (dankserver :7770, radiance) are different engines — see [System hosts](/system/index.md). The pi-era `frank-default-small`/`frank-default-large` llama.cpp pair (each :7770) is retired; [Pi Models Configuration](../../pi/models.md) is a pi-era archive.
|
|
|
|
---
|
|
|
|
## Health Check
|
|
|
|
```bash
|
|
curl http://localhost:8080/health
|
|
# 200: {"status": "ok"}
|
|
# 503: {"error": {"code": 503, "message": "Loading model"}}
|
|
```
|
|
|
|
---
|
|
|
|
## API Endpoints
|
|
|
|
### Chat Completions (OpenAI-compatible)
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer no-key" \
|
|
-d '{
|
|
"model": "gpt-3.5-turbo",
|
|
"messages": [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Write a limerick about Python"}
|
|
],
|
|
"max_tokens": 256,
|
|
"stream": true
|
|
}'
|
|
```
|
|
|
|
### Other Endpoints
|
|
|
|
```bash
|
|
# Completions
|
|
curl http://localhost:8080/v1/completions
|
|
|
|
# Embeddings
|
|
curl http://localhost:8080/v1/embeddings
|
|
|
|
# Native completion
|
|
curl -X POST http://localhost:8080/completion -d '{"prompt": "Hello", "n_predict": 64}'
|
|
|
|
# Anthropic-compatible
|
|
curl http://localhost:8080/v1/messages
|
|
|
|
# Monitoring
|
|
curl http://localhost:8080/props
|
|
curl http://localhost:8080/slots
|
|
curl http://localhost:8080/metrics
|
|
```
|
|
|
|
---
|
|
|
|
## Python Client
|
|
|
|
```python
|
|
import openai
|
|
client = openai.OpenAI(
|
|
base_url="http://localhost:8080/v1",
|
|
api_key="sk-no-key-required"
|
|
)
|
|
completion = client.chat.completions.create(
|
|
model="gpt-3.5-turbo",
|
|
messages=[{"role": "user", "content": "What is the meaning of life?"}],
|
|
max_tokens=256
|
|
)
|
|
print(completion.choices[0].message.content)
|
|
```
|
|
|
|
---
|
|
|
|
## Docker
|
|
|
|
```bash
|
|
docker run -p 8080:8080 -v /models:/models \
|
|
ghcr.io/ggml-org/llama.cpp:server \
|
|
-m /models/model.gguf --host 0.0.0.0 --port 8080
|
|
|
|
# CUDA variant
|
|
docker run -p 8080:8080 --gpus all \
|
|
ghcr.io/ggml-org/llama.cpp:server-cuda \
|
|
-m /models/model.gguf --n-gpu-layers 99
|
|
```
|
|
|
|
---
|
|
|
|
## Environment Variables
|
|
|
|
All CLI options can be set via `LLAMA_ARG_` prefix:
|
|
|
|
```bash
|
|
LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server
|
|
```
|
|
|
|
---
|
|
|
|
## Cross-References
|
|
|
|
- See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment.
|
|
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for running Ornith models.
|