Frank
02883d3ed2
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
40 lines
1.2 KiB
Markdown
40 lines
1.2 KiB
Markdown
---
|
|
type: reference
|
|
title: Performance Benchmarks
|
|
description: Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU inference.
|
|
tags:
|
|
- ornith
|
|
- performance
|
|
- benchmarks
|
|
- unverified
|
|
verified: 2026-08-28
|
|
---
|
|
|
|
# Performance
|
|
# Performance
|
|
|
|
## 9B Model:
|
|
|
|
- **Prompt Processing:** ~10-15 tokens/sec (CPU), ~50-100 tokens/sec (GPU)
|
|
- **Generation:** ~8-12 tokens/sec (CPU), ~40-80 tokens/sec (GPU)
|
|
|
|
## 35B MoE Model:
|
|
|
|
- **Prompt Processing:** ~5-8 tokens/sec (CPU+GPU mix), ~30-50 tokens/sec (full GPU)
|
|
- **Generation:** ~4-8 tokens/sec (CPU+GPU mix), ~20-40 tokens/sec (full GPU)
|
|
|
|
**Note:** MoE models can be slower due to expert routing overhead, but provide better quality.
|
|
|
|
## Tuning Tips
|
|
|
|
1. **Increase GPU layers** (`-ngl`) for better speed
|
|
2. **Enable Flash Attention:** `--flash-attn on` (if supported)
|
|
3. **Use a smaller quantization** (Q4_K_M instead of Q8_0)
|
|
4. **Reduce context window size** if memory-bound
|
|
5. **KV Cache Quantization:** `--cache-type-k q8_0` to reduce KV cache memory
|
|
|
|
## Related
|
|
|
|
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Optimize GPU layers for your hardware
|
|
* [Troubleshooting](/tools/ornith-llama-cpp-setup/troubleshooting.md) — Fix performance issues
|