docs: rebuild KB in OKF v0.1 with live-verified facts
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
This commit is contained in:
Frank
@@ -0,0 +1,266 @@
|
||||
---
|
||||
type: howto
|
||||
title: llama.cpp — Build and Deployment Setup
|
||||
description: Build guides for CPU, CUDA, Metal, Vulkan, and other GPU backends, plus Docker Compose and systemd service configuration.
|
||||
tags:
|
||||
- llm
|
||||
- inference
|
||||
- llama-cpp
|
||||
- build
|
||||
- deployment
|
||||
verified: 2026-08-28
|
||||
---
|
||||
|
||||
# llama.cpp — Build and Deployment Setup
|
||||
|
||||
See [Overview](index.md) for server usage and API reference.
|
||||
|
||||
---
|
||||
|
||||
## Building llama.cpp
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Git
|
||||
- CMake (3.12+)
|
||||
- C++ compiler (GCC 9+, Clang 12+, MSVC 2019+)
|
||||
|
||||
### CPU-only Build
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp
|
||||
cd llama.cpp
|
||||
|
||||
cmake -B build
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
The `llama-server` binary will be at `./build/bin/llama-server`.
|
||||
|
||||
**Faster compilation:**
|
||||
- Use Ninja: `cmake -B build -G Ninja`
|
||||
- Install `ccache` for incremental builds
|
||||
|
||||
**Debug build:**
|
||||
```bash
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Debug
|
||||
cmake --build build
|
||||
```
|
||||
|
||||
### CUDA Build (NVIDIA GPU)
|
||||
|
||||
```bash
|
||||
# Requires CUDA toolkit installed
|
||||
cmake -B build -DGGML_CUDA=ON
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
**Non-native build (works on all CUDA GPUs):**
|
||||
```bash
|
||||
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
|
||||
cmake --build build --config Release
|
||||
```
|
||||
|
||||
**Override compute capabilities:**
|
||||
```bash
|
||||
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89"
|
||||
```
|
||||
|
||||
### Metal Build (Apple Silicon)
|
||||
|
||||
Metal is enabled by default on macOS. No special flags needed:
|
||||
|
||||
```bash
|
||||
cmake -B build
|
||||
cmake --build build --config Release
|
||||
```
|
||||
|
||||
Disable GPU offloading at runtime with `--n-gpu-layers 0`.
|
||||
|
||||
### Other GPU Backends
|
||||
|
||||
| Backend | Flag | Platform |
|
||||
|---------|------|----------|
|
||||
| HIP/ROCm (AMD GPU) | `-DGGML_HIP=ON` | Linux |
|
||||
| Vulkan | `-DGGML_VULKAN=ON` | All |
|
||||
| SYCL (Intel GPU) | `-DGGML_SYCL=ON` | Linux |
|
||||
| CANN (Ascend NPU) | `-DGGML_CANN=ON` | Linux |
|
||||
| OpenVINO | `-DGGML_OPENVINO=ON` | Intel |
|
||||
| ZenDNN (AMD CPU) | `-DGGML_ZENDNN=ON` | Linux |
|
||||
| WebGPU | `-DGGML_WEBGPU=ON` | All (WASM) |
|
||||
|
||||
### SSL/TLS Support
|
||||
|
||||
```bash
|
||||
# Requires OpenSSL dev libraries
|
||||
cmake -B build -DLLAMA_OPENSSL=ON
|
||||
cmake --build build --config Release -t llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Command-Line Options
|
||||
|
||||
### Model Loading
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-m, --model FNAME` | Path to GGUF model file | required |
|
||||
| `-hf, --hf-repo USER/MODEL` | Download from Hugging Face | — |
|
||||
| `-mmproj FILE` | Multimodal projector file | — |
|
||||
| `-lora FNAME` | LoRA adapter (repeatable) | — |
|
||||
| `-ngl, --gpu-layers N` | Layers to offload to GPU | auto |
|
||||
|
||||
### Context & Performance
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-c, --ctx-size N` | Prompt context size | 0 (from model) |
|
||||
| `-t, --threads N` | CPU threads | -1 (all) |
|
||||
| `-b, --batch-size N` | Max logical batch size | 2048 |
|
||||
| `--flash-attn` | Flash Attention | auto |
|
||||
| `--mlock` | Keep model in RAM (no swap) | false |
|
||||
| `--mmap` | Memory-map model file | true |
|
||||
| `--fit` | Auto-fit to GPU memory | on |
|
||||
|
||||
### Server Configuration
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--host HOST` | Bind address | 127.0.0.1 |
|
||||
| `--port PORT` | Port number | 8080 |
|
||||
| `-np, --parallel N` | Concurrent slots | -1 (auto) |
|
||||
| `--api-key KEY` | API key (repeatable) | none |
|
||||
| `--metrics` | Prometheus metrics | disabled |
|
||||
| `--jinja` | Jinja template engine | enabled |
|
||||
|
||||
### Sampling Parameters
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-s, --seed N` | RNG seed | -1 (random) |
|
||||
| `--temp, --temperature N` | Temperature | 0.80 |
|
||||
| `--top-k N` | Top-k sampling | 40 |
|
||||
| `--top-p N` | Top-p sampling | 0.95 |
|
||||
| `--min-p N` | Min-p sampling | 0.05 |
|
||||
| `--repeat-penalty N` | Repetition penalty | 1.1 |
|
||||
|
||||
---
|
||||
|
||||
## GPU Offloading
|
||||
|
||||
```bash
|
||||
# Offload all layers to GPU
|
||||
llama-server -m model.gguf --n-gpu-layers 99
|
||||
|
||||
# Auto-detect (default)
|
||||
llama-server -m model.gguf --n-gpu-layers auto
|
||||
|
||||
# Disable GPU (CPU-only)
|
||||
llama-server -m model.gguf --n-gpu-layers 0
|
||||
|
||||
# Multi-GPU (layer mode)
|
||||
llama-server -m model.gguf --device 0,1 --split-mode layer
|
||||
|
||||
# Multi-GPU (row split)
|
||||
llama-server -m model.gguf --device 0,1 --split-mode row
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Context Length
|
||||
|
||||
```bash
|
||||
# Use model's native context
|
||||
llama-server -m model.gguf -c 0
|
||||
|
||||
# Set specific context
|
||||
llama-server -m model.gguf -c 8192
|
||||
|
||||
# RoPE scaling to extend beyond trained size
|
||||
llama-server -m model.gguf -c 65536 --rope-scale 2
|
||||
|
||||
# YaRN scaling
|
||||
llama-server -m model.gguf -c 65536 --rope-scaling yarn --yarn-orig-ctx 8192
|
||||
```
|
||||
|
||||
**Important**: Larger context = more VRAM/RAM usage (KV cache grows linearly). Reduce with quantization:
|
||||
|
||||
```bash
|
||||
llama-server -m model.gguf -c 32768 \
|
||||
--cache-type-k q4_0 --cache-type-v q4_0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
```bash
|
||||
# Draft model based
|
||||
llama-server -m main_model.gguf -md draft_model.gguf
|
||||
|
||||
# N-gram based (no draft model needed)
|
||||
llama-server -m model.gguf --spec-type ngram-mod --spec-ngram-mod-n-max 64
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose Example
|
||||
|
||||
```yaml
|
||||
services:
|
||||
llamacpp-server:
|
||||
image: ghcr.io/ggml-org/llama.cpp:server-cuda
|
||||
ports:
|
||||
- 8080:8080
|
||||
volumes:
|
||||
- ./models:/models
|
||||
environment:
|
||||
LLAMA_ARG_MODEL: /models/my_model.gguf
|
||||
LLAMA_ARG_CTX_SIZE: 8192
|
||||
LLAMA_ARG_N_PARALLEL: 2
|
||||
LLAMA_ARG_PORT: 8080
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: 1
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running as systemd Service
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/llama-server.service
|
||||
[Unit]
|
||||
Description=llama.cpp Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \
|
||||
-m /home/dank/models/my_model.gguf -c 131072 -ngl 12 \
|
||||
--host 0.0.0.0 --port 8080
|
||||
Restart=on-failure
|
||||
WorkingDirectory=/home/dank/GitPackages/llama.cpp
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable llama-server
|
||||
sudo systemctl start llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- See [Overview](index.md) for API reference and usage examples.
|
||||
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for model-specific configuration.
|
||||
Reference in New Issue
Block a user