dank/command-center
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(llama): delete fabricated ornith-llama-cpp-setup tree (608 lines)

Browse Source 
Night fact-check falsified the tree's subject: no `ornith` CLI exists
(npm 404), github.com/rhitching/ornith is 404, and the model IDs the
tree cites are wrong — the real family is ornith-ai/Ornith-1.0/1.5
(9B, 35B-A3B, 397B), built on Qwen3.5/Gemma4 per its model card, not
the 'Ornith-9B-Thinking-1.5T based on Qwen3-8B' the docs assert.
Nothing in the tree describes this fleet: no ornith binary on PATH
(audit 2026-10-18 said the same). Real facts it echoed (PrismML fork
on frank-zealot) already live in /system/.

Also: tools/index.md de-duplicated (same files were listed twice, once
as wikilinks once as links, live-sync missorted) and links normalized;
llama-cpp setup/index xrefs retargeted.

Refs #29
This commit is contained in:
Frank
2026-10-19 03:17:49 -07:00
parent 651924a4bc
commit 39385636be
14 changed files with 8 additions and 536 deletions
Download Patch File Download Diff File Expand all files Collapse all files
omp-docs/tools/index.md
+7 -10
View File
@@ -6,21 +6,18 @@ Configuration guides and reference material for installed software. Each tool ha
## Available Tools
* [[cachyos|cachyos]] — Arch-based rolling release with performance-optimized kernels, repositories, and gaming stack. Running on the workstation.
* [applications](applications.md) — Default userspace applications on CachyOS with Niri — browsers, editor, terminal, file manager, audio, screenshots, gaming.
* [cachyos](cachyos.md) — Arch-based rolling release with performance-optimized kernels, repositories, and gaming stack. Running on the workstation.
* [comfyui](comfyui/index.md) — Node-based AI image generation engine on dankclaw. FLUX.2 klein 4B, CPU backend, systemd managed.
* [live-sync](live-sync.md) — Heartbeat daemon committing machine-written paths (memories, managed-skills, live config, dotfiles) in the command-center checkout; systemd user timer, alerts via ntfy.
* [[applications|applications]] — Default userspace applications on CachyOS with Niri — browsers, editor, terminal, file manager, audio, screenshots, gaming.
* [ghostty](ghostty/index.md) — GPU-accelerated terminal emulator with extensive configuration, keybindings, and shaders.
* [gitea](gitea/index.md) — Private Git hosting on dankclaw. Quadlet-managed containers: Gitea, PostgreSQL, Caddy.
* [llama.cpp](llama-cpp/index.md) — Local LLM inference server supporting GGUF models with OpenAI-compatible API.
* [[linux-power-management|linux-power-management]] — Sleep/wake troubleshooting, NVIDIA GPU power management, and diagnostics for Linux systems.
* [[noctalia-shell|noctalia-shell]] — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management.
* [handy](handy.md) — Offline speech-to-text transcription with WebKitGTK backend and Tauri v2 UI.
* [live-sync](live-sync.md) — Heartbeat daemon committing machine-written paths (memories, managed-skills, live config, dotfiles) in the command-center checkout; systemd user timer, alerts via ntfy.
* [llama.cpp](llama-cpp/index.md) — Local LLM inference server supporting GGUF models with OpenAI-compatible API.
* [linux-power-management](linux-power-management.md) — Sleep/wake troubleshooting, NVIDIA GPU power management, and diagnostics for Linux systems.
* [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation.
* [ornith-llama-cpp-setup](ornith-llama-cpp-setup/index.md) — Setup and reference for running Ornith MoE models with llama.cpp server for local inference.
* [[tmux|tmux]] — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures.
* [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management.
* [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures.
## Adding a New Tool
omp-docs/tools/llama-cpp/index.md
-1
View File
@@ -152,4 +152,3 @@ LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server
## Cross-References
- See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment.
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for running Ornith models.
omp-docs/tools/llama-cpp/setup.md
-1
View File
@@ -263,4 +263,3 @@ sudo systemctl start llama-server
## Cross-References
- See [Overview](index.md) for API reference and usage examples.
- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for model-specific configuration.
omp-docs/tools/ornith-llama-cpp-setup/api-endpoints.md
-73
View File
@@ -1,73 +0,0 @@
---
type: reference
title: llama.cpp Server API Endpoints
description: Complete reference for health check, model listing, chat completions, and native completion endpoints.
tags:
- unverified
- ornith
- api
- llama.cpp
- endpoints
timestamp: 2026-07-18
verified: 2026-08-28
---
# API Endpoints
The llama.cpp server provides these endpoints:
## Health Check
```bash
curl http://localhost:8080/health
```
## List Models
```bash
curl http://localhost:8080/v1/models
```
## Chat Completions (OpenAI-compatible)
```bash
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "frank-default-local",
"messages": [...],
"max_tokens": 100
}'
```
## Native Completion Endpoint
```bash
curl http://localhost:8080/completion \
-H "Content-Type: application/json" \
-d '{
"prompt": "Hello, how are you?",
"n_predict": 100
}'
```
## Model Features
### Reasoning/Thinking Blocks
Ornith models use `<think>...</think>` blocks for reasoning. The server automatically:
- Detects and preserves reasoning content
- Returns reasoning in `reasoning_content` field
- Supports the `developer` role (or falls back to `system`)
### Chat Template
The model uses a specific chat template that supports:
- System prompts
- User/Assistant roles
- Reasoning preservation
## Related
* [Pi Configuration](/tools/ornith-llama-cpp-setup/pi-configuration.md) — How pi connects to these endpoints
* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — Starting the server
omp-docs/tools/ornith-llama-cpp-setup/gpu-offloading.md
-46
View File
@@ -1,46 +0,0 @@
---
type: reference
title: GPU Layer Offloading Guidelines
description: How many GPU layers to offload based on your VRAM size for both Ornith 9B and 35B models.
tags:
- unverified
- ornith
- gpu
- offloading
- llama.cpp
timestamp: 2026-07-18
verified: 2026-08-28
---
# GPU Layer Offloading
## For 9B Q4_K_M (5.63 GB):
- **8GB VRAM:** `-ngl 35-40` (fits most layers)
- **12GB VRAM:** `-ngl 99` (all layers)
- **16GB+ VRAM:** `-ngl 99` (all layers + room for large KV cache)
## For 35B Q4_K_M (~20 GB):
- **8GB VRAM:** `-ngl 10-14` (partial offload)
- **12GB VRAM:** `-ngl 18-22` (partial offload)
- **16GB VRAM:** `-ngl 28-32` (partial offload)
- **24GB VRAM:** `-ngl 40-48` (majority of layers)
- **40GB+ VRAM:** `-ngl 99` (all layers)
**Note:** The 35B model is a MoE architecture, so actual memory usage depends on which experts are activated.
## Context Window Considerations
- **9B model:** Supports up to 262,144 tokens
- **35B model:** Supports up to 262,144 tokens
- **KV Cache Memory:** Larger context windows require more VRAM/RAM for the key-value cache
- 131K context: ~8-16 GB depending on model and quantization
- 256K context: ~16-32 GB depending on model and quantization
**Recommendation:** Start with 131K context if you have limited VRAM, increase to 256K if you have headroom.
## Related
* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — How to set the -ngl flag
* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Memory requirements
omp-docs/tools/ornith-llama-cpp-setup/index.md
-17
View File
@@ -1,17 +0,0 @@
# Ornith llama.cpp Setup
Complete reference for running Ornith-9B/35B MoE models with llama.cpp server on the workstation.
## Concept Files
| File | Type | Description |
|------|------|-------------|
| [api-endpoints.md](api-endpoints.md) | reference | Complete reference for health check, model listing, chat completions, and nat... |
| [gpu-offloading.md](gpu-offloading.md) | reference | How many GPU layers to offload based on your VRAM size for both Ornith 9B and... |
| [installation.md](installation.md) | howto | Step-by-step setup — clone and build llama.cpp, download models, start the se... |
| [model-specs.md](model-specs.md) | reference | Specifications for Ornith-1.0-9B dense and Ornith-1.0-35B MoE models — contex... |
| [performance.md](performance.md) | reference | Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU infe... |
| [pi-configuration.md](pi-configuration.md) | howto | How to add Ornith models to models.json so pi can use the local llama.cpp ser... |
| [systemd-service.md](systemd-service.md) | howto | Set up llama.cpp server as a background service with systemd for persistent o... |
| [troubleshooting.md](troubleshooting.md) | howto | Common errors — CUDA OOM, segfaults, model not responding, poor performance —... |
omp-docs/tools/ornith-llama-cpp-setup/installation.md
-97
View File
@@ -1,97 +0,0 @@
---
type: howto
title: Installing Ornith + llama.cpp
description: Step-by-step setup — clone and build llama.cpp, download models, start the server, verify it works.
tags:
- ornith
- installation
- llama.cpp
- unverified
verified: 2026-08-28
---
# Installation
## 1. Clone and Build llama.cpp
```bash
cd ~/GitPackages
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release
```
## 2. Download the Model
```bash
mkdir -p ~/models/ornith
cd ~/models/ornith
# For 9B model (faster, less memory)
wget https://huggingface.co/ornith-ai/Ornith-1.0-9B-GGUF/resolve/main/ornith-1.0-9b-Q4_K_M.gguf
# OR for 35B MoE model (better performance, more memory)
wget https://huggingface.co/ornith-ai/Ornith-1.0-35B-GGUF/resolve/main/ornith-1.0-35b-Q4_K_M.gguf
```
## 3. Start the Server
**For 9B model with full GPU offload (8GB+ VRAM):**
```bash
cd ~/GitPackages/llama.cpp
./build/bin/llama-server \
-m ~/models/ornith/ornith-1.0-9b-Q4_K_M.gguf \
-c 262144 \
-ngl 99 \
--host 0.0.0.0 \
--port 8080
```
**For 35B MoE model with partial GPU offload (8GB VRAM example):**
```bash
cd ~/GitPackages/llama.cpp
./build/bin/llama-server \
-m ~/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \
-c 131072 \
-ngl 12 \
--host 0.0.0.0 \
--port 8080
```
**Key Parameters:**
- `-m`: Path to GGUF model file
- `-c` / `--ctx-size`: Context window size (131072 or 262144)
- `-ngl` / `--n-gpu-layers`: Number of layers to offload to GPU
- 99 = all layers
- Lower values = more CPU inference
- `--host`: Bind address (0.0.0.0 for all interfaces, 127.0.0.1 for local only)
- `--port`: Server port (default 8080)
## 4. Verify Server is Running
```bash
curl http://localhost:8080/health
# Expected: {"status":"ok"}
```
## 5. Test the Model
```bash
curl -s http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "frank-default-local",
"messages": [
{"role": "user", "content": "Hello, what is 2+2?"}
],
"max_tokens": 150,
"temperature": 0.7
}' | jq .
```
## Related
* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Model options and quantization
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Tuning GPU layer counts
* [Pi Configuration](/tools/ornith-llama-cpp-setup/pi-configuration.md) — Adding to models.json
omp-docs/tools/ornith-llama-cpp-setup/model-specs.md
-65
View File
@@ -1,65 +0,0 @@
---
type: reference
title: Ornith Model Specifications
description: Specifications for Ornith-1.0-9B dense and Ornith-1.0-35B MoE models — context length, quantization options, memory requirements.
tags:
- ornith
- models
- llama.cpp
- GGUF
- unverified
verified: 2026-08-28
---
# Model Specifications
## Ornith-1.0-9B-GGUF
- **Source:** https://huggingface.co/ornith-ai/Ornith-1.0-9B-GGUF
- **Architecture:** Dense ~9B model
- **Context Length:** 262,144 tokens (256K)
- **Quantization Options:**
- Q4_K_M: 5.3 GB (recommended for most systems)
- Q5_K_M: 6.47 GB
- Q6_K: 7.36 GB
- Q8_0: 9.53 GB
- bf16: 17.9 GB
## Ornith-1.0-35B-GGUF (MoE)
- **Source:** https://huggingface.co/ornith-ai/Ornith-1.0-35B-GGUF
- **Architecture:** 35B MoE (Mixture of Experts)
- **Context Length:** 262,144 tokens (256K)
- **Quantization Options:**
- Q4_K_M: 20 GB (recommended)
- Other quantizations available
**Note:** No fp4 quantization exists for these models. Q4_K_M is the lightest quantization.
## Comparison: 9B vs 35B MoE
| Aspect | 9B Dense | 35B MoE |
|--------|----------|---------|
| **Memory (Q4)** | 5.3 GB | 20 GB |
| **VRAM for Full GPU** | 8 GB | 24-40 GB |
| **Speed (GPU)** | Faster | Slower but better quality |
| **Quality** | Good | Excellent |
| **Context** | 256K | 256K |
| **Best For** | Quick tasks, limited RAM | Complex reasoning, best quality |
## Requirements
### Hardware
- **RAM:** Minimum 16GB for 9B model, 32GB+ for 35B model
- **GPU** (optional but recommended):
- 9B Q4: 6-8GB VRAM for full GPU offload
- 35B Q4: 24GB+ VRAM for full GPU offload
### Software
- llama.cpp built with CUDA support (for GPU acceleration)
- Git (for cloning llama.cpp repo)
## Related
* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — How to set up the server
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Layer allocation by VRAM
omp-docs/tools/ornith-llama-cpp-setup/performance.md
-39
View File
@@ -1,39 +0,0 @@
---
type: reference
title: Performance Benchmarks
description: Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU inference.
tags:
- ornith
- performance
- benchmarks
- unverified
verified: 2026-08-28
---
# Performance
# Performance
## 9B Model:
- **Prompt Processing:** ~10-15 tokens/sec (CPU), ~50-100 tokens/sec (GPU)
- **Generation:** ~8-12 tokens/sec (CPU), ~40-80 tokens/sec (GPU)
## 35B MoE Model:
- **Prompt Processing:** ~5-8 tokens/sec (CPU+GPU mix), ~30-50 tokens/sec (full GPU)
- **Generation:** ~4-8 tokens/sec (CPU+GPU mix), ~20-40 tokens/sec (full GPU)
**Note:** MoE models can be slower due to expert routing overhead, but provide better quality.
## Tuning Tips
1. **Increase GPU layers** (`-ngl`) for better speed
2. **Enable Flash Attention:** `--flash-attn on` (if supported)
3. **Use a smaller quantization** (Q4_K_M instead of Q8_0)
4. **Reduce context window size** if memory-bound
5. **KV Cache Quantization:** `--cache-type-k q8_0` to reduce KV cache memory
## Related
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Optimize GPU layers for your hardware
* [Troubleshooting](/tools/ornith-llama-cpp-setup/troubleshooting.md) — Fix performance issues
omp-docs/tools/ornith-llama-cpp-setup/pi-configuration.md
-55
View File
@@ -1,55 +0,0 @@
---
type: howto
title: Pi Configuration for Ornith Models
description: How to add Ornith models to models.json so pi can use the local llama.cpp server as a provider.
tags:
- unverified
- ornith
- pi
- models.json
- configuration
timestamp: 2026-07-18
verified: 2026-08-28
---
# Pi Configuration
Add to `~/.pi/agent/models.json`:
```json
{
"providers": {
"llamacpp (local-ornith)": {
"baseUrl": "http://localhost:8080/v1",
"api": "openai-completions",
"apiKey": "EMPTY",
"compat": {
"supportsDeveloperRole": false,
"supportsReasoningEffort": false
},
"models": [
{
"id": "frank-default-local",
"name": "Ornith-1.0-35B-Q4_K_M (Local)",
"contextWindow": 131072,
"reasoning": true,
"thinkingLevelMap": {
"max": "xhigh"
}
}
]
}
}
}
```
**Important Compatibility Settings:**
- `supportsDeveloperRole: false` — The server does not support the `developer` role, uses `system` instead
- `supportsReasoningEffort: false` — The server does not support the `reasoning_effort` parameter
- `reasoning: true` — Model supports reasoning/thinking blocks
## Related
* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — Get the server running first
* [API Endpoints](/tools/ornith-llama-cpp-setup/api-endpoints.md) — The endpoints this configuration targets
omp-docs/tools/ornith-llama-cpp-setup/systemd-service.md
-59
View File
@@ -1,59 +0,0 @@
---
type: howto
title: Running Ornith as a Systemd Service
description: Set up llama.cpp server as a background service with systemd for persistent operation.
tags:
- unverified
- ornith
- systemd
- service
timestamp: 2026-07-18
verified: 2026-08-28
---
# Systemd Service
## Using nohup:
```bash
nohup ~/GitPackages/llama.cpp/build/bin/llama-server \
-m ~/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \
-c 131072 -ngl 12 \
--host 0.0.0.0 --port 8080 \
> ~/logs/llama-server.log 2>&1 &
```
## Using systemd (advanced):
Create `/etc/systemd/system/llama-server.service`:
```ini
[Unit]
Description=llama.cpp Server
After=network.target
[Service]
Type=simple
User=dank
ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \
-m /home/dank/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \
-c 131072 -ngl 12 \
--host 0.0.0.0 --port 8080
Restart=on-failure
WorkingDirectory=/home/dank/GitPackages/llama.cpp
[Install]
WantedBy=multi-user.target
```
Then:
```bash
sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server
```
## Related
* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — The server setup this service wraps
omp-docs/tools/ornith-llama-cpp-setup/troubleshooting.md
-71
View File
@@ -1,71 +0,0 @@
---
type: howto
title: Troubleshooting Ornith + llama.cpp
description: Common errors — CUDA OOM, segfaults, model not responding, poor performance — and their fixes.
tags:
- unverified
- ornith
- troubleshooting
- llama.cpp
timestamp: 2026-07-18
verified: 2026-08-28
---
# Troubleshooting
## CUDA Out of Memory
```
Error: cudaMalloc failed: out of memory
```
**Solution:** Reduce `-ngl` value to offload fewer layers to GPU.
## Segmentation Fault
```
Segmentation fault (core dumped)
```
**Solution:**
1. Reduce context size (`-c` parameter)
2. Reduce GPU layers (`-ngl`)
3. Check if model file is corrupted
## Model Not Responding
```
curl: (7) Failed to connect to localhost port 8080
```
**Solution:** Check if server is running:
```bash
ps aux | grep llama-server
tail -f ~/logs/llama-server.log
```
## Poor Performance
**Solutions:**
1. Increase GPU layers (`-ngl`)
2. Enable Flash Attention: `--flash-attn on` (if supported)
3. Use a smaller quantization (Q4_K_M instead of Q8_0)
4. Reduce context window size
## Security Considerations
- **Local Server:** By default, binds to localhost (127.0.0.1)
- **Network Exposure:** If binding to 0.0.0.0, consider:
- Adding authentication (`--api-key`)
- Using a firewall to restrict access
- Running behind a reverse proxy with TLS
```bash
# Add API key authentication
./build/bin/llama-server ... --api-key your-secret-key
```
## Related
* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — The commands that might be causing issues
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — GPU configuration