docs: rebuild KB in OKF v0.1 with live-verified facts
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
This commit is contained in:
Frank
@@ -0,0 +1,155 @@
|
||||
# llama.cpp Server — Overview
|
||||
|
||||
[llama.cpp](https://github.com/ggml-org/llama.cpp) is a pure C/C++ LLM inference library. The `llama-server` binary is a lightweight, OpenAI-compatible HTTP server for serving GGUF models via REST API.
|
||||
|
||||
See [Setup Guide](setup.md) for build instructions (CPU/CUDA/Metal/Vulkan), Docker, and systemd deployment.
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
- OpenAI API-compatible chat completions, completions, and embeddings
|
||||
- Anthropic Messages API-compatible chat completions
|
||||
- Parallel decoding with multi-user support and continuous batching
|
||||
- Multimodal (vision/audio) support
|
||||
- Speculative decoding with draft models
|
||||
- Schema-constrained JSON output and function calling
|
||||
- Built-in Web UI
|
||||
- Prometheus metrics
|
||||
- Router mode for multi-model deployment
|
||||
|
||||
---
|
||||
|
||||
## GGUF Model Format
|
||||
|
||||
**GGUF** (GGML Universal File) is the binary model file format used by llama.cpp.
|
||||
|
||||
### Quantization Methods
|
||||
|
||||
| Method | Bits/weight | 8B model size | Quality |
|
||||
|--------|------------|---------------|---------|
|
||||
| Q4_K_M | 4.9 | 4.58 GB | **Good (recommended default)** |
|
||||
| Q5_K_M | 5.7 | 5.33 GB | High |
|
||||
| Q6_K | 6.6 | 6.14 GB | Very High |
|
||||
| Q8_0 | 8.5 | 7.95 GB | Near-lossless |
|
||||
| F16 | 16.0 | 14.96 GB | Full precision |
|
||||
|
||||
**Q4_K_M is the recommended default** — best balance of quality and size.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Serve a local model on default port 8080
|
||||
llama-server -m /path/to/model.gguf
|
||||
|
||||
# From Hugging Face (auto-downloads)
|
||||
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
|
||||
```
|
||||
|
||||
**Defaults**: Listens on `127.0.0.1:8080`. Single slot (one concurrent request).
|
||||
|
||||
> **System note:** On this network the Pi inference hosts do not use the default port. Pi serves `frank-default-small` (dankserver) and `frank-default-large` (dankclaw), each on port `7770`. The local Ornith host runs on port `8080`. See [Pi Models Configuration](../../pi/models.md).
|
||||
|
||||
---
|
||||
|
||||
## Health Check
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/health
|
||||
# 200: {"status": "ok"}
|
||||
# 503: {"error": {"code": 503, "message": "Loading model"}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Chat Completions (OpenAI-compatible)
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer no-key" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Write a limerick about Python"}
|
||||
],
|
||||
"max_tokens": 256,
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
### Other Endpoints
|
||||
|
||||
```bash
|
||||
# Completions
|
||||
curl http://localhost:8080/v1/completions
|
||||
|
||||
# Embeddings
|
||||
curl http://localhost:8080/v1/embeddings
|
||||
|
||||
# Native completion
|
||||
curl -X POST http://localhost:8080/completion -d '{"prompt": "Hello", "n_predict": 64}'
|
||||
|
||||
# Anthropic-compatible
|
||||
curl http://localhost:8080/v1/messages
|
||||
|
||||
# Monitoring
|
||||
curl http://localhost:8080/props
|
||||
curl http://localhost:8080/slots
|
||||
curl http://localhost:8080/metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python Client
|
||||
|
||||
```python
|
||||
import openai
|
||||
client = openai.OpenAI(
|
||||
base_url="http://localhost:8080/v1",
|
||||
api_key="sk-no-key-required"
|
||||
)
|
||||
completion = client.chat.completions.create(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[{"role": "user", "content": "What is the meaning of life?"}],
|
||||
max_tokens=256
|
||||
)
|
||||
print(completion.choices[0].message.content)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8080 -v /models:/models \
|
||||
ghcr.io/ggml-org/llama.cpp:server \
|
||||
-m /models/model.gguf --host 0.0.0.0 --port 8080
|
||||
|
||||
# CUDA variant
|
||||
docker run -p 8080:8080 --gpus all \
|
||||
ghcr.io/ggml-org/llama.cpp:server-cuda \
|
||||
-m /models/model.gguf --n-gpu-layers 99
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All CLI options can be set via `LLAMA_ARG_` prefix:
|
||||
|
||||
```bash
|
||||
LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment.
|
||||
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for running Ornith models.
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
type: howto
|
||||
title: llama.cpp — Build and Deployment Setup
|
||||
description: Build guides for CPU, CUDA, Metal, Vulkan, and other GPU backends, plus Docker Compose and systemd service configuration.
|
||||
tags:
|
||||
- llm
|
||||
- inference
|
||||
- llama-cpp
|
||||
- build
|
||||
- deployment
|
||||
verified: 2026-08-28
|
||||
---
|
||||
|
||||
# llama.cpp — Build and Deployment Setup
|
||||
|
||||
See [Overview](index.md) for server usage and API reference.
|
||||
|
||||
---
|
||||
|
||||
## Building llama.cpp
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Git
|
||||
- CMake (3.12+)
|
||||
- C++ compiler (GCC 9+, Clang 12+, MSVC 2019+)
|
||||
|
||||
### CPU-only Build
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp
|
||||
cd llama.cpp
|
||||
|
||||
cmake -B build
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
The `llama-server` binary will be at `./build/bin/llama-server`.
|
||||
|
||||
**Faster compilation:**
|
||||
- Use Ninja: `cmake -B build -G Ninja`
|
||||
- Install `ccache` for incremental builds
|
||||
|
||||
**Debug build:**
|
||||
```bash
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Debug
|
||||
cmake --build build
|
||||
```
|
||||
|
||||
### CUDA Build (NVIDIA GPU)
|
||||
|
||||
```bash
|
||||
# Requires CUDA toolkit installed
|
||||
cmake -B build -DGGML_CUDA=ON
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
**Non-native build (works on all CUDA GPUs):**
|
||||
```bash
|
||||
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
|
||||
cmake --build build --config Release
|
||||
```
|
||||
|
||||
**Override compute capabilities:**
|
||||
```bash
|
||||
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89"
|
||||
```
|
||||
|
||||
### Metal Build (Apple Silicon)
|
||||
|
||||
Metal is enabled by default on macOS. No special flags needed:
|
||||
|
||||
```bash
|
||||
cmake -B build
|
||||
cmake --build build --config Release
|
||||
```
|
||||
|
||||
Disable GPU offloading at runtime with `--n-gpu-layers 0`.
|
||||
|
||||
### Other GPU Backends
|
||||
|
||||
| Backend | Flag | Platform |
|
||||
|---------|------|----------|
|
||||
| HIP/ROCm (AMD GPU) | `-DGGML_HIP=ON` | Linux |
|
||||
| Vulkan | `-DGGML_VULKAN=ON` | All |
|
||||
| SYCL (Intel GPU) | `-DGGML_SYCL=ON` | Linux |
|
||||
| CANN (Ascend NPU) | `-DGGML_CANN=ON` | Linux |
|
||||
| OpenVINO | `-DGGML_OPENVINO=ON` | Intel |
|
||||
| ZenDNN (AMD CPU) | `-DGGML_ZENDNN=ON` | Linux |
|
||||
| WebGPU | `-DGGML_WEBGPU=ON` | All (WASM) |
|
||||
|
||||
### SSL/TLS Support
|
||||
|
||||
```bash
|
||||
# Requires OpenSSL dev libraries
|
||||
cmake -B build -DLLAMA_OPENSSL=ON
|
||||
cmake --build build --config Release -t llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Command-Line Options
|
||||
|
||||
### Model Loading
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-m, --model FNAME` | Path to GGUF model file | required |
|
||||
| `-hf, --hf-repo USER/MODEL` | Download from Hugging Face | — |
|
||||
| `-mmproj FILE` | Multimodal projector file | — |
|
||||
| `-lora FNAME` | LoRA adapter (repeatable) | — |
|
||||
| `-ngl, --gpu-layers N` | Layers to offload to GPU | auto |
|
||||
|
||||
### Context & Performance
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-c, --ctx-size N` | Prompt context size | 0 (from model) |
|
||||
| `-t, --threads N` | CPU threads | -1 (all) |
|
||||
| `-b, --batch-size N` | Max logical batch size | 2048 |
|
||||
| `--flash-attn` | Flash Attention | auto |
|
||||
| `--mlock` | Keep model in RAM (no swap) | false |
|
||||
| `--mmap` | Memory-map model file | true |
|
||||
| `--fit` | Auto-fit to GPU memory | on |
|
||||
|
||||
### Server Configuration
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--host HOST` | Bind address | 127.0.0.1 |
|
||||
| `--port PORT` | Port number | 8080 |
|
||||
| `-np, --parallel N` | Concurrent slots | -1 (auto) |
|
||||
| `--api-key KEY` | API key (repeatable) | none |
|
||||
| `--metrics` | Prometheus metrics | disabled |
|
||||
| `--jinja` | Jinja template engine | enabled |
|
||||
|
||||
### Sampling Parameters
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-s, --seed N` | RNG seed | -1 (random) |
|
||||
| `--temp, --temperature N` | Temperature | 0.80 |
|
||||
| `--top-k N` | Top-k sampling | 40 |
|
||||
| `--top-p N` | Top-p sampling | 0.95 |
|
||||
| `--min-p N` | Min-p sampling | 0.05 |
|
||||
| `--repeat-penalty N` | Repetition penalty | 1.1 |
|
||||
|
||||
---
|
||||
|
||||
## GPU Offloading
|
||||
|
||||
```bash
|
||||
# Offload all layers to GPU
|
||||
llama-server -m model.gguf --n-gpu-layers 99
|
||||
|
||||
# Auto-detect (default)
|
||||
llama-server -m model.gguf --n-gpu-layers auto
|
||||
|
||||
# Disable GPU (CPU-only)
|
||||
llama-server -m model.gguf --n-gpu-layers 0
|
||||
|
||||
# Multi-GPU (layer mode)
|
||||
llama-server -m model.gguf --device 0,1 --split-mode layer
|
||||
|
||||
# Multi-GPU (row split)
|
||||
llama-server -m model.gguf --device 0,1 --split-mode row
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Context Length
|
||||
|
||||
```bash
|
||||
# Use model's native context
|
||||
llama-server -m model.gguf -c 0
|
||||
|
||||
# Set specific context
|
||||
llama-server -m model.gguf -c 8192
|
||||
|
||||
# RoPE scaling to extend beyond trained size
|
||||
llama-server -m model.gguf -c 65536 --rope-scale 2
|
||||
|
||||
# YaRN scaling
|
||||
llama-server -m model.gguf -c 65536 --rope-scaling yarn --yarn-orig-ctx 8192
|
||||
```
|
||||
|
||||
**Important**: Larger context = more VRAM/RAM usage (KV cache grows linearly). Reduce with quantization:
|
||||
|
||||
```bash
|
||||
llama-server -m model.gguf -c 32768 \
|
||||
--cache-type-k q4_0 --cache-type-v q4_0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
```bash
|
||||
# Draft model based
|
||||
llama-server -m main_model.gguf -md draft_model.gguf
|
||||
|
||||
# N-gram based (no draft model needed)
|
||||
llama-server -m model.gguf --spec-type ngram-mod --spec-ngram-mod-n-max 64
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose Example
|
||||
|
||||
```yaml
|
||||
services:
|
||||
llamacpp-server:
|
||||
image: ghcr.io/ggml-org/llama.cpp:server-cuda
|
||||
ports:
|
||||
- 8080:8080
|
||||
volumes:
|
||||
- ./models:/models
|
||||
environment:
|
||||
LLAMA_ARG_MODEL: /models/my_model.gguf
|
||||
LLAMA_ARG_CTX_SIZE: 8192
|
||||
LLAMA_ARG_N_PARALLEL: 2
|
||||
LLAMA_ARG_PORT: 8080
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running as systemd Service
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/llama-server.service
|
||||
[Unit]
|
||||
Description=llama.cpp Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \
|
||||
-m /home/dank/models/my_model.gguf -c 131072 -ngl 12 \
|
||||
--host 0.0.0.0 --port 8080
|
||||
Restart=on-failure
|
||||
WorkingDirectory=/home/dank/GitPackages/llama.cpp
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable llama-server
|
||||
sudo systemctl start llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- See [Overview](index.md) for API reference and usage examples.
|
||||
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for model-specific configuration.
|
||||
Reference in New Issue
Block a user