docs: rebuild KB in OKF v0.1 with live-verified facts
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
This commit is contained in:
@@ -0,0 +1,155 @@
|
||||
# llama.cpp Server — Overview
|
||||
|
||||
[llama.cpp](https://github.com/ggml-org/llama.cpp) is a pure C/C++ LLM inference library. The `llama-server` binary is a lightweight, OpenAI-compatible HTTP server for serving GGUF models via REST API.
|
||||
|
||||
See [Setup Guide](setup.md) for build instructions (CPU/CUDA/Metal/Vulkan), Docker, and systemd deployment.
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
- OpenAI API-compatible chat completions, completions, and embeddings
|
||||
- Anthropic Messages API-compatible chat completions
|
||||
- Parallel decoding with multi-user support and continuous batching
|
||||
- Multimodal (vision/audio) support
|
||||
- Speculative decoding with draft models
|
||||
- Schema-constrained JSON output and function calling
|
||||
- Built-in Web UI
|
||||
- Prometheus metrics
|
||||
- Router mode for multi-model deployment
|
||||
|
||||
---
|
||||
|
||||
## GGUF Model Format
|
||||
|
||||
**GGUF** (GGML Universal File) is the binary model file format used by llama.cpp.
|
||||
|
||||
### Quantization Methods
|
||||
|
||||
| Method | Bits/weight | 8B model size | Quality |
|
||||
|--------|------------|---------------|---------|
|
||||
| Q4_K_M | 4.9 | 4.58 GB | **Good (recommended default)** |
|
||||
| Q5_K_M | 5.7 | 5.33 GB | High |
|
||||
| Q6_K | 6.6 | 6.14 GB | Very High |
|
||||
| Q8_0 | 8.5 | 7.95 GB | Near-lossless |
|
||||
| F16 | 16.0 | 14.96 GB | Full precision |
|
||||
|
||||
**Q4_K_M is the recommended default** — best balance of quality and size.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Serve a local model on default port 8080
|
||||
llama-server -m /path/to/model.gguf
|
||||
|
||||
# From Hugging Face (auto-downloads)
|
||||
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
|
||||
```
|
||||
|
||||
**Defaults**: Listens on `127.0.0.1:8080`. Single slot (one concurrent request).
|
||||
|
||||
> **System note:** On this network the Pi inference hosts do not use the default port. Pi serves `frank-default-small` (dankserver) and `frank-default-large` (dankclaw), each on port `7770`. The local Ornith host runs on port `8080`. See [Pi Models Configuration](../../pi/models.md).
|
||||
|
||||
---
|
||||
|
||||
## Health Check
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/health
|
||||
# 200: {"status": "ok"}
|
||||
# 503: {"error": {"code": 503, "message": "Loading model"}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Chat Completions (OpenAI-compatible)
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer no-key" \
|
||||
-d '{
|
||||
"model": "gpt-3.5-turbo",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Write a limerick about Python"}
|
||||
],
|
||||
"max_tokens": 256,
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
### Other Endpoints
|
||||
|
||||
```bash
|
||||
# Completions
|
||||
curl http://localhost:8080/v1/completions
|
||||
|
||||
# Embeddings
|
||||
curl http://localhost:8080/v1/embeddings
|
||||
|
||||
# Native completion
|
||||
curl -X POST http://localhost:8080/completion -d '{"prompt": "Hello", "n_predict": 64}'
|
||||
|
||||
# Anthropic-compatible
|
||||
curl http://localhost:8080/v1/messages
|
||||
|
||||
# Monitoring
|
||||
curl http://localhost:8080/props
|
||||
curl http://localhost:8080/slots
|
||||
curl http://localhost:8080/metrics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python Client
|
||||
|
||||
```python
|
||||
import openai
|
||||
client = openai.OpenAI(
|
||||
base_url="http://localhost:8080/v1",
|
||||
api_key="sk-no-key-required"
|
||||
)
|
||||
completion = client.chat.completions.create(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[{"role": "user", "content": "What is the meaning of life?"}],
|
||||
max_tokens=256
|
||||
)
|
||||
print(completion.choices[0].message.content)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker
|
||||
|
||||
```bash
|
||||
docker run -p 8080:8080 -v /models:/models \
|
||||
ghcr.io/ggml-org/llama.cpp:server \
|
||||
-m /models/model.gguf --host 0.0.0.0 --port 8080
|
||||
|
||||
# CUDA variant
|
||||
docker run -p 8080:8080 --gpus all \
|
||||
ghcr.io/ggml-org/llama.cpp:server-cuda \
|
||||
-m /models/model.gguf --n-gpu-layers 99
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All CLI options can be set via `LLAMA_ARG_` prefix:
|
||||
|
||||
```bash
|
||||
LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
- See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment.
|
||||
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for running Ornith models.
|
||||
Reference in New Issue
Block a user