Frank
02883d3ed2
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
98 lines
2.4 KiB
Markdown
98 lines
2.4 KiB
Markdown
---
|
|
type: howto
|
|
title: Installing Ornith + llama.cpp
|
|
description: Step-by-step setup — clone and build llama.cpp, download models, start the server, verify it works.
|
|
tags:
|
|
- ornith
|
|
- installation
|
|
- llama.cpp
|
|
- unverified
|
|
verified: 2026-08-28
|
|
---
|
|
|
|
# Installation
|
|
|
|
## 1. Clone and Build llama.cpp
|
|
|
|
```bash
|
|
cd ~/GitPackages
|
|
git clone https://github.com/ggerganov/llama.cpp
|
|
cd llama.cpp
|
|
cmake -B build -DLLAMA_CUDA=ON
|
|
cmake --build build --config Release
|
|
```
|
|
|
|
## 2. Download the Model
|
|
|
|
```bash
|
|
mkdir -p ~/models/ornith
|
|
cd ~/models/ornith
|
|
|
|
# For 9B model (faster, less memory)
|
|
wget https://huggingface.co/ornith-ai/Ornith-1.0-9B-GGUF/resolve/main/ornith-1.0-9b-Q4_K_M.gguf
|
|
|
|
# OR for 35B MoE model (better performance, more memory)
|
|
wget https://huggingface.co/ornith-ai/Ornith-1.0-35B-GGUF/resolve/main/ornith-1.0-35b-Q4_K_M.gguf
|
|
```
|
|
|
|
## 3. Start the Server
|
|
|
|
**For 9B model with full GPU offload (8GB+ VRAM):**
|
|
```bash
|
|
cd ~/GitPackages/llama.cpp
|
|
./build/bin/llama-server \
|
|
-m ~/models/ornith/ornith-1.0-9b-Q4_K_M.gguf \
|
|
-c 262144 \
|
|
-ngl 99 \
|
|
--host 0.0.0.0 \
|
|
--port 8080
|
|
```
|
|
|
|
**For 35B MoE model with partial GPU offload (8GB VRAM example):**
|
|
```bash
|
|
cd ~/GitPackages/llama.cpp
|
|
./build/bin/llama-server \
|
|
-m ~/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \
|
|
-c 131072 \
|
|
-ngl 12 \
|
|
--host 0.0.0.0 \
|
|
--port 8080
|
|
```
|
|
|
|
**Key Parameters:**
|
|
- `-m`: Path to GGUF model file
|
|
- `-c` / `--ctx-size`: Context window size (131072 or 262144)
|
|
- `-ngl` / `--n-gpu-layers`: Number of layers to offload to GPU
|
|
- 99 = all layers
|
|
- Lower values = more CPU inference
|
|
- `--host`: Bind address (0.0.0.0 for all interfaces, 127.0.0.1 for local only)
|
|
- `--port`: Server port (default 8080)
|
|
|
|
## 4. Verify Server is Running
|
|
|
|
```bash
|
|
curl http://localhost:8080/health
|
|
# Expected: {"status":"ok"}
|
|
```
|
|
|
|
## 5. Test the Model
|
|
|
|
```bash
|
|
curl -s http://localhost:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "frank-default-local",
|
|
"messages": [
|
|
{"role": "user", "content": "Hello, what is 2+2?"}
|
|
],
|
|
"max_tokens": 150,
|
|
"temperature": 0.7
|
|
}' | jq .
|
|
```
|
|
|
|
## Related
|
|
|
|
* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Model options and quantization
|
|
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Tuning GPU layer counts
|
|
* [Pi Configuration](/tools/ornith-llama-cpp-setup/pi-configuration.md) — Adding to models.json
|