diff --git a/omp-docs/index.md b/omp-docs/index.md index c9b9392..40c4b9f 100644 --- a/omp-docs/index.md +++ b/omp-docs/index.md @@ -23,7 +23,7 @@ Porkbun DNS-01, hot reload). ## [Gitea and tooling](tools/index.md) -Installed software: Gitea, llama.cpp/ornith, niri, ghostty, tmux, ComfyUI, +Installed software: Gitea, llama.cpp, niri, ghostty, tmux, ComfyUI, CachyOS, skills tooling — each with its verified operational facts. ## [Pi-era agent setup](pi/index.md) diff --git a/omp-docs/pi/models.md b/omp-docs/pi/models.md index dc4dcda..6039aaf 100644 --- a/omp-docs/pi/models.md +++ b/omp-docs/pi/models.md @@ -145,4 +145,3 @@ The subagent model IDs (`frank-default-small`, `frank-default-large`) resolve ag - [[settings|settings]] — Main configuration file - [[dankserver|dankserver]] — Primary llama.cpp server - [[dankclaw|dankclaw]] — Secondary llama.cpp server -- [ornith-llama-cpp-setup](../tools/ornith-llama-cpp-setup/index.md) — Local Ornith model setup diff --git a/omp-docs/tools/index.md b/omp-docs/tools/index.md index 6376ee4..e0833f5 100644 --- a/omp-docs/tools/index.md +++ b/omp-docs/tools/index.md @@ -6,21 +6,18 @@ Configuration guides and reference material for installed software. Each tool ha ## Available Tools -* [[cachyos|cachyos]] — Arch-based rolling release with performance-optimized kernels, repositories, and gaming stack. Running on the workstation. +* [applications](applications.md) — Default userspace applications on CachyOS with Niri — browsers, editor, terminal, file manager, audio, screenshots, gaming. +* [cachyos](cachyos.md) — Arch-based rolling release with performance-optimized kernels, repositories, and gaming stack. Running on the workstation. * [comfyui](comfyui/index.md) — Node-based AI image generation engine on dankclaw. FLUX.2 klein 4B, CPU backend, systemd managed. -* [live-sync](live-sync.md) — Heartbeat daemon committing machine-written paths (memories, managed-skills, live config, dotfiles) in the command-center checkout; systemd user timer, alerts via ntfy. -* [[applications|applications]] — Default userspace applications on CachyOS with Niri — browsers, editor, terminal, file manager, audio, screenshots, gaming. * [ghostty](ghostty/index.md) — GPU-accelerated terminal emulator with extensive configuration, keybindings, and shaders. * [gitea](gitea/index.md) — Private Git hosting on dankclaw. Quadlet-managed containers: Gitea, PostgreSQL, Caddy. -* [llama.cpp](llama-cpp/index.md) — Local LLM inference server supporting GGUF models with OpenAI-compatible API. -* [[linux-power-management|linux-power-management]] — Sleep/wake troubleshooting, NVIDIA GPU power management, and diagnostics for Linux systems. -* [[noctalia-shell|noctalia-shell]] — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management. * [handy](handy.md) — Offline speech-to-text transcription with WebKitGTK backend and Tauri v2 UI. +* [live-sync](live-sync.md) — Heartbeat daemon committing machine-written paths (memories, managed-skills, live config, dotfiles) in the command-center checkout; systemd user timer, alerts via ntfy. +* [llama.cpp](llama-cpp/index.md) — Local LLM inference server supporting GGUF models with OpenAI-compatible API. +* [linux-power-management](linux-power-management.md) — Sleep/wake troubleshooting, NVIDIA GPU power management, and diagnostics for Linux systems. * [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation. - -* [ornith-llama-cpp-setup](ornith-llama-cpp-setup/index.md) — Setup and reference for running Ornith MoE models with llama.cpp server for local inference. -* [[tmux|tmux]] — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures. - +* [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management. +* [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures. ## Adding a New Tool diff --git a/omp-docs/tools/llama-cpp/index.md b/omp-docs/tools/llama-cpp/index.md index ca0a4ff..8425263 100644 --- a/omp-docs/tools/llama-cpp/index.md +++ b/omp-docs/tools/llama-cpp/index.md @@ -152,4 +152,3 @@ LLAMA_ARG_MODEL=/models/model.gguf LLAMA_ARG_CTX_SIZE=8192 llama-server ## Cross-References - See [Setup Guide](setup.md) for build instructions, GPU offloading, and deployment. -- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for running Ornith models. diff --git a/omp-docs/tools/llama-cpp/setup.md b/omp-docs/tools/llama-cpp/setup.md index 8a9963b..1bc7b35 100644 --- a/omp-docs/tools/llama-cpp/setup.md +++ b/omp-docs/tools/llama-cpp/setup.md @@ -263,4 +263,3 @@ sudo systemctl start llama-server ## Cross-References - See [Overview](index.md) for API reference and usage examples. -- See [Ornith Setup](../ornith-llama-cpp-setup/index.md) for model-specific configuration. diff --git a/omp-docs/tools/ornith-llama-cpp-setup/api-endpoints.md b/omp-docs/tools/ornith-llama-cpp-setup/api-endpoints.md deleted file mode 100644 index b73b984..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/api-endpoints.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -type: reference -title: llama.cpp Server API Endpoints -description: Complete reference for health check, model listing, chat completions, and native completion endpoints. -tags: - - unverified - - ornith - - api - - llama.cpp - - endpoints -timestamp: 2026-07-18 -verified: 2026-08-28 ---- - -# API Endpoints - -The llama.cpp server provides these endpoints: - -## Health Check - -```bash -curl http://localhost:8080/health -``` - -## List Models - -```bash -curl http://localhost:8080/v1/models -``` - -## Chat Completions (OpenAI-compatible) - -```bash -curl http://localhost:8080/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "frank-default-local", - "messages": [...], - "max_tokens": 100 - }' -``` - -## Native Completion Endpoint - -```bash -curl http://localhost:8080/completion \ - -H "Content-Type: application/json" \ - -d '{ - "prompt": "Hello, how are you?", - "n_predict": 100 - }' -``` - -## Model Features - -### Reasoning/Thinking Blocks - -Ornith models use `...` blocks for reasoning. The server automatically: -- Detects and preserves reasoning content -- Returns reasoning in `reasoning_content` field -- Supports the `developer` role (or falls back to `system`) - -### Chat Template - -The model uses a specific chat template that supports: -- System prompts -- User/Assistant roles -- Reasoning preservation - -## Related - -* [Pi Configuration](/tools/ornith-llama-cpp-setup/pi-configuration.md) — How pi connects to these endpoints -* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — Starting the server diff --git a/omp-docs/tools/ornith-llama-cpp-setup/gpu-offloading.md b/omp-docs/tools/ornith-llama-cpp-setup/gpu-offloading.md deleted file mode 100644 index 5b6930e..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/gpu-offloading.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -type: reference -title: GPU Layer Offloading Guidelines -description: How many GPU layers to offload based on your VRAM size for both Ornith 9B and 35B models. -tags: - - unverified - - ornith - - gpu - - offloading - - llama.cpp -timestamp: 2026-07-18 -verified: 2026-08-28 ---- - -# GPU Layer Offloading - -## For 9B Q4_K_M (5.63 GB): - -- **8GB VRAM:** `-ngl 35-40` (fits most layers) -- **12GB VRAM:** `-ngl 99` (all layers) -- **16GB+ VRAM:** `-ngl 99` (all layers + room for large KV cache) - -## For 35B Q4_K_M (~20 GB): - -- **8GB VRAM:** `-ngl 10-14` (partial offload) -- **12GB VRAM:** `-ngl 18-22` (partial offload) -- **16GB VRAM:** `-ngl 28-32` (partial offload) -- **24GB VRAM:** `-ngl 40-48` (majority of layers) -- **40GB+ VRAM:** `-ngl 99` (all layers) - -**Note:** The 35B model is a MoE architecture, so actual memory usage depends on which experts are activated. - -## Context Window Considerations - -- **9B model:** Supports up to 262,144 tokens -- **35B model:** Supports up to 262,144 tokens -- **KV Cache Memory:** Larger context windows require more VRAM/RAM for the key-value cache - - 131K context: ~8-16 GB depending on model and quantization - - 256K context: ~16-32 GB depending on model and quantization - -**Recommendation:** Start with 131K context if you have limited VRAM, increase to 256K if you have headroom. - -## Related - -* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — How to set the -ngl flag -* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Memory requirements diff --git a/omp-docs/tools/ornith-llama-cpp-setup/index.md b/omp-docs/tools/ornith-llama-cpp-setup/index.md deleted file mode 100644 index 5ee3521..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/index.md +++ /dev/null @@ -1,17 +0,0 @@ -# Ornith llama.cpp Setup - -Complete reference for running Ornith-9B/35B MoE models with llama.cpp server on the workstation. - -## Concept Files - -| File | Type | Description | -|------|------|-------------| -| [api-endpoints.md](api-endpoints.md) | reference | Complete reference for health check, model listing, chat completions, and nat... | -| [gpu-offloading.md](gpu-offloading.md) | reference | How many GPU layers to offload based on your VRAM size for both Ornith 9B and... | -| [installation.md](installation.md) | howto | Step-by-step setup — clone and build llama.cpp, download models, start the se... | -| [model-specs.md](model-specs.md) | reference | Specifications for Ornith-1.0-9B dense and Ornith-1.0-35B MoE models — contex... | -| [performance.md](performance.md) | reference | Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU infe... | -| [pi-configuration.md](pi-configuration.md) | howto | How to add Ornith models to models.json so pi can use the local llama.cpp ser... | -| [systemd-service.md](systemd-service.md) | howto | Set up llama.cpp server as a background service with systemd for persistent o... | -| [troubleshooting.md](troubleshooting.md) | howto | Common errors — CUDA OOM, segfaults, model not responding, poor performance —... | - diff --git a/omp-docs/tools/ornith-llama-cpp-setup/installation.md b/omp-docs/tools/ornith-llama-cpp-setup/installation.md deleted file mode 100644 index 7c2683d..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/installation.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -type: howto -title: Installing Ornith + llama.cpp -description: Step-by-step setup — clone and build llama.cpp, download models, start the server, verify it works. -tags: - - ornith - - installation - - llama.cpp - - unverified -verified: 2026-08-28 ---- - -# Installation - -## 1. Clone and Build llama.cpp - -```bash -cd ~/GitPackages -git clone https://github.com/ggerganov/llama.cpp -cd llama.cpp -cmake -B build -DLLAMA_CUDA=ON -cmake --build build --config Release -``` - -## 2. Download the Model - -```bash -mkdir -p ~/models/ornith -cd ~/models/ornith - -# For 9B model (faster, less memory) -wget https://huggingface.co/ornith-ai/Ornith-1.0-9B-GGUF/resolve/main/ornith-1.0-9b-Q4_K_M.gguf - -# OR for 35B MoE model (better performance, more memory) -wget https://huggingface.co/ornith-ai/Ornith-1.0-35B-GGUF/resolve/main/ornith-1.0-35b-Q4_K_M.gguf -``` - -## 3. Start the Server - -**For 9B model with full GPU offload (8GB+ VRAM):** -```bash -cd ~/GitPackages/llama.cpp -./build/bin/llama-server \ - -m ~/models/ornith/ornith-1.0-9b-Q4_K_M.gguf \ - -c 262144 \ - -ngl 99 \ - --host 0.0.0.0 \ - --port 8080 -``` - -**For 35B MoE model with partial GPU offload (8GB VRAM example):** -```bash -cd ~/GitPackages/llama.cpp -./build/bin/llama-server \ - -m ~/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \ - -c 131072 \ - -ngl 12 \ - --host 0.0.0.0 \ - --port 8080 -``` - -**Key Parameters:** -- `-m`: Path to GGUF model file -- `-c` / `--ctx-size`: Context window size (131072 or 262144) -- `-ngl` / `--n-gpu-layers`: Number of layers to offload to GPU - - 99 = all layers - - Lower values = more CPU inference -- `--host`: Bind address (0.0.0.0 for all interfaces, 127.0.0.1 for local only) -- `--port`: Server port (default 8080) - -## 4. Verify Server is Running - -```bash -curl http://localhost:8080/health -# Expected: {"status":"ok"} -``` - -## 5. Test the Model - -```bash -curl -s http://localhost:8080/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "frank-default-local", - "messages": [ - {"role": "user", "content": "Hello, what is 2+2?"} - ], - "max_tokens": 150, - "temperature": 0.7 - }' | jq . -``` - -## Related - -* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Model options and quantization -* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Tuning GPU layer counts -* [Pi Configuration](/tools/ornith-llama-cpp-setup/pi-configuration.md) — Adding to models.json diff --git a/omp-docs/tools/ornith-llama-cpp-setup/model-specs.md b/omp-docs/tools/ornith-llama-cpp-setup/model-specs.md deleted file mode 100644 index 829cdef..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/model-specs.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -type: reference -title: Ornith Model Specifications -description: Specifications for Ornith-1.0-9B dense and Ornith-1.0-35B MoE models — context length, quantization options, memory requirements. -tags: - - ornith - - models - - llama.cpp - - GGUF - - unverified -verified: 2026-08-28 ---- - -# Model Specifications - -## Ornith-1.0-9B-GGUF - -- **Source:** https://huggingface.co/ornith-ai/Ornith-1.0-9B-GGUF -- **Architecture:** Dense ~9B model -- **Context Length:** 262,144 tokens (256K) -- **Quantization Options:** - - Q4_K_M: 5.3 GB (recommended for most systems) - - Q5_K_M: 6.47 GB - - Q6_K: 7.36 GB - - Q8_0: 9.53 GB - - bf16: 17.9 GB - -## Ornith-1.0-35B-GGUF (MoE) - -- **Source:** https://huggingface.co/ornith-ai/Ornith-1.0-35B-GGUF -- **Architecture:** 35B MoE (Mixture of Experts) -- **Context Length:** 262,144 tokens (256K) -- **Quantization Options:** - - Q4_K_M: 20 GB (recommended) - - Other quantizations available - -**Note:** No fp4 quantization exists for these models. Q4_K_M is the lightest quantization. - -## Comparison: 9B vs 35B MoE - -| Aspect | 9B Dense | 35B MoE | -|--------|----------|---------| -| **Memory (Q4)** | 5.3 GB | 20 GB | -| **VRAM for Full GPU** | 8 GB | 24-40 GB | -| **Speed (GPU)** | Faster | Slower but better quality | -| **Quality** | Good | Excellent | -| **Context** | 256K | 256K | -| **Best For** | Quick tasks, limited RAM | Complex reasoning, best quality | - -## Requirements - -### Hardware -- **RAM:** Minimum 16GB for 9B model, 32GB+ for 35B model -- **GPU** (optional but recommended): - - 9B Q4: 6-8GB VRAM for full GPU offload - - 35B Q4: 24GB+ VRAM for full GPU offload - -### Software -- llama.cpp built with CUDA support (for GPU acceleration) -- Git (for cloning llama.cpp repo) - -## Related - -* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — How to set up the server -* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Layer allocation by VRAM diff --git a/omp-docs/tools/ornith-llama-cpp-setup/performance.md b/omp-docs/tools/ornith-llama-cpp-setup/performance.md deleted file mode 100644 index c691557..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/performance.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -type: reference -title: Performance Benchmarks -description: Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU inference. -tags: - - ornith - - performance - - benchmarks - - unverified -verified: 2026-08-28 ---- - -# Performance -# Performance - -## 9B Model: - -- **Prompt Processing:** ~10-15 tokens/sec (CPU), ~50-100 tokens/sec (GPU) -- **Generation:** ~8-12 tokens/sec (CPU), ~40-80 tokens/sec (GPU) - -## 35B MoE Model: - -- **Prompt Processing:** ~5-8 tokens/sec (CPU+GPU mix), ~30-50 tokens/sec (full GPU) -- **Generation:** ~4-8 tokens/sec (CPU+GPU mix), ~20-40 tokens/sec (full GPU) - -**Note:** MoE models can be slower due to expert routing overhead, but provide better quality. - -## Tuning Tips - -1. **Increase GPU layers** (`-ngl`) for better speed -2. **Enable Flash Attention:** `--flash-attn on` (if supported) -3. **Use a smaller quantization** (Q4_K_M instead of Q8_0) -4. **Reduce context window size** if memory-bound -5. **KV Cache Quantization:** `--cache-type-k q8_0` to reduce KV cache memory - -## Related - -* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Optimize GPU layers for your hardware -* [Troubleshooting](/tools/ornith-llama-cpp-setup/troubleshooting.md) — Fix performance issues diff --git a/omp-docs/tools/ornith-llama-cpp-setup/pi-configuration.md b/omp-docs/tools/ornith-llama-cpp-setup/pi-configuration.md deleted file mode 100644 index dbec1b3..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/pi-configuration.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -type: howto -title: Pi Configuration for Ornith Models -description: How to add Ornith models to models.json so pi can use the local llama.cpp server as a provider. -tags: - - unverified - - ornith - - pi - - models.json - - configuration -timestamp: 2026-07-18 -verified: 2026-08-28 ---- - -# Pi Configuration - -Add to `~/.pi/agent/models.json`: - -```json -{ - "providers": { - "llamacpp (local-ornith)": { - "baseUrl": "http://localhost:8080/v1", - "api": "openai-completions", - "apiKey": "EMPTY", - "compat": { - "supportsDeveloperRole": false, - "supportsReasoningEffort": false - }, - "models": [ - { - "id": "frank-default-local", - "name": "Ornith-1.0-35B-Q4_K_M (Local)", - "contextWindow": 131072, - "reasoning": true, - "thinkingLevelMap": { - "max": "xhigh" - } - } - ] - } - } -} -``` - -**Important Compatibility Settings:** - -- `supportsDeveloperRole: false` — The server does not support the `developer` role, uses `system` instead -- `supportsReasoningEffort: false` — The server does not support the `reasoning_effort` parameter -- `reasoning: true` — Model supports reasoning/thinking blocks - -## Related - -* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — Get the server running first -* [API Endpoints](/tools/ornith-llama-cpp-setup/api-endpoints.md) — The endpoints this configuration targets diff --git a/omp-docs/tools/ornith-llama-cpp-setup/systemd-service.md b/omp-docs/tools/ornith-llama-cpp-setup/systemd-service.md deleted file mode 100644 index 3a62e2e..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/systemd-service.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -type: howto -title: Running Ornith as a Systemd Service -description: Set up llama.cpp server as a background service with systemd for persistent operation. -tags: - - unverified - - ornith - - systemd - - service -timestamp: 2026-07-18 -verified: 2026-08-28 ---- - -# Systemd Service - -## Using nohup: - -```bash -nohup ~/GitPackages/llama.cpp/build/bin/llama-server \ - -m ~/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \ - -c 131072 -ngl 12 \ - --host 0.0.0.0 --port 8080 \ - > ~/logs/llama-server.log 2>&1 & -``` - -## Using systemd (advanced): - -Create `/etc/systemd/system/llama-server.service`: - -```ini -[Unit] -Description=llama.cpp Server -After=network.target - -[Service] -Type=simple -User=dank -ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \ - -m /home/dank/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \ - -c 131072 -ngl 12 \ - --host 0.0.0.0 --port 8080 -Restart=on-failure -WorkingDirectory=/home/dank/GitPackages/llama.cpp - -[Install] -WantedBy=multi-user.target -``` - -Then: - -```bash -sudo systemctl daemon-reload -sudo systemctl enable llama-server -sudo systemctl start llama-server -``` - -## Related - -* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — The server setup this service wraps diff --git a/omp-docs/tools/ornith-llama-cpp-setup/troubleshooting.md b/omp-docs/tools/ornith-llama-cpp-setup/troubleshooting.md deleted file mode 100644 index 19fe432..0000000 --- a/omp-docs/tools/ornith-llama-cpp-setup/troubleshooting.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -type: howto -title: Troubleshooting Ornith + llama.cpp -description: Common errors — CUDA OOM, segfaults, model not responding, poor performance — and their fixes. -tags: - - unverified - - ornith - - troubleshooting - - llama.cpp -timestamp: 2026-07-18 -verified: 2026-08-28 ---- - -# Troubleshooting - -## CUDA Out of Memory - -``` -Error: cudaMalloc failed: out of memory -``` - -**Solution:** Reduce `-ngl` value to offload fewer layers to GPU. - -## Segmentation Fault - -``` -Segmentation fault (core dumped) -``` - -**Solution:** -1. Reduce context size (`-c` parameter) -2. Reduce GPU layers (`-ngl`) -3. Check if model file is corrupted - -## Model Not Responding - -``` -curl: (7) Failed to connect to localhost port 8080 -``` - -**Solution:** Check if server is running: -```bash -ps aux | grep llama-server -tail -f ~/logs/llama-server.log -``` - -## Poor Performance - -**Solutions:** -1. Increase GPU layers (`-ngl`) -2. Enable Flash Attention: `--flash-attn on` (if supported) -3. Use a smaller quantization (Q4_K_M instead of Q8_0) -4. Reduce context window size - -## Security Considerations - -- **Local Server:** By default, binds to localhost (127.0.0.1) -- **Network Exposure:** If binding to 0.0.0.0, consider: - - Adding authentication (`--api-key`) - - Using a firewall to restrict access - - Running behind a reverse proxy with TLS - -```bash -# Add API key authentication -./build/bin/llama-server ... --api-key your-secret-key -``` - -## Related - -* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — The commands that might be causing issues -* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — GPU configuration