Frank
ec19771b8b
The overview page carried `docker run` invocations (including a CUDA variant, on a fleet with no NVIDIA card left in it) and the `LLAMA_ARG_` environment rule, while `setup.md` — the page whose own intro promises "build instructions, Docker, and systemd deployment" — carried the Compose example and the same env form in use. Two places to look, one place to be wrong. `setup.md` now states the env-prefix rule where the flags live (and why units prefer it: one flag per line is auditable, a forty-flag command line is not), and the overview keeps only the API surface and the GGUF format, pointing at the sections that absorbed the rest. Refs #64
132 lines
3.7 KiB
Markdown
132 lines
3.7 KiB
Markdown
# llama.cpp Server — Overview
|
|
|
|
[llama.cpp](https://github.com/ggml-org/llama.cpp) is a pure C/C++ LLM inference library. The `llama-server` binary is a lightweight, OpenAI-compatible HTTP server for serving GGUF models via REST API.
|
|
|
|
See [Setup Guide](setup.md) for build instructions (CPU/CUDA/Metal/Vulkan), Docker, and systemd deployment.
|
|
|
|
---
|
|
|
|
## Key Features
|
|
|
|
- OpenAI API-compatible chat completions, completions, and embeddings
|
|
- Anthropic Messages API-compatible chat completions
|
|
- Parallel decoding with multi-user support and continuous batching
|
|
- Multimodal (vision/audio) support
|
|
- Speculative decoding with draft models
|
|
- Schema-constrained JSON output and function calling
|
|
- Built-in Web UI
|
|
- Prometheus metrics
|
|
- Router mode for multi-model deployment
|
|
|
|
---
|
|
|
|
## GGUF Model Format
|
|
|
|
**GGUF** (GGML Universal File) is the binary model file format used by llama.cpp.
|
|
|
|
### Quantization Methods
|
|
|
|
| Method | Bits/weight | 8B model size | Quality |
|
|
|--------|------------|---------------|---------|
|
|
| Q4_K_M | 4.9 | 4.58 GB | **Good (recommended default)** |
|
|
| Q5_K_M | 5.7 | 5.33 GB | High |
|
|
| Q6_K | 6.6 | 6.14 GB | Very High |
|
|
| Q8_0 | 8.5 | 7.95 GB | Near-lossless |
|
|
| F16 | 16.0 | 14.96 GB | Full precision |
|
|
|
|
**Q4_K_M is the recommended default** — best balance of quality and size.
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Serve a local model on default port 8080
|
|
llama-server -m /path/to/model.gguf
|
|
|
|
# From Hugging Face (auto-downloads)
|
|
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
|
|
```
|
|
|
|
**Defaults**: Listens on `127.0.0.1:8080`. Single slot (one concurrent request).
|
|
|
|
> **System note:** On this network the fleet inference hosts do not use the default port. As of 2026-10 the only llama.cpp endpoint is `frank-zealot` on dankclaw **:7771** (PrismML fork build); `frank-dragoon` (dankclaw :7770, halogen-flash-server) and `frank-archon` (dankserver :7770, radiance) are different engines — see [System hosts](/system/index.md). The pi-era `frank-default-small`/`frank-default-large` llama.cpp pair (each :7770) is retired; [Pi Models Configuration](../../pi/models.md) is a pi-era archive.
|
|
|
|
---
|
|
|
|
## Health Check
|
|
|
|
```bash
|
|
curl http://localhost:8080/health
|
|
# 200: {"status": "ok"}
|
|
# 503: {"error": {"code": 503, "message": "Loading model"}}
|
|
```
|
|
|
|
---
|
|
|
|
## API Endpoints
|
|
|
|
### Chat Completions (OpenAI-compatible)
|
|
|
|
```bash
|
|
curl http://localhost:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer no-key" \
|
|
-d '{
|
|
"model": "gpt-3.5-turbo",
|
|
"messages": [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Write a limerick about Python"}
|
|
],
|
|
"max_tokens": 256,
|
|
"stream": true
|
|
}'
|
|
```
|
|
|
|
### Other Endpoints
|
|
|
|
```bash
|
|
# Completions
|
|
curl http://localhost:8080/v1/completions
|
|
|
|
# Embeddings
|
|
curl http://localhost:8080/v1/embeddings
|
|
|
|
# Native completion
|
|
curl -X POST http://localhost:8080/completion -d '{"prompt": "Hello", "n_predict": 64}'
|
|
|
|
# Anthropic-compatible
|
|
curl http://localhost:8080/v1/messages
|
|
|
|
# Monitoring
|
|
curl http://localhost:8080/props
|
|
curl http://localhost:8080/slots
|
|
curl http://localhost:8080/metrics
|
|
```
|
|
|
|
---
|
|
|
|
## Python Client
|
|
|
|
```python
|
|
import openai
|
|
client = openai.OpenAI(
|
|
base_url="http://localhost:8080/v1",
|
|
api_key="sk-no-key-required"
|
|
)
|
|
completion = client.chat.completions.create(
|
|
model="gpt-3.5-turbo",
|
|
messages=[{"role": "user", "content": "What is the meaning of life?"}],
|
|
max_tokens=256
|
|
)
|
|
print(completion.choices[0].message.content)
|
|
```
|
|
|
|
---
|
|
|
|
## Cross-References
|
|
|
|
- See [Setup Guide](setup.md) for build instructions, GPU offloading, the `LLAMA_ARG_*`
|
|
environment form, the Docker Compose example, and running under systemd. Deployment lives
|
|
there; this page is the API surface and the GGUF format.
|