Frank
02883d3ed2
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
1.2 KiB
1.2 KiB
type, title, description, tags, verified
| type | title | description | tags | verified | ||||
|---|---|---|---|---|---|---|---|---|
| reference | Performance Benchmarks | Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU inference. |
|
2026-08-28 |
Performance
Performance
9B Model:
- Prompt Processing: ~10-15 tokens/sec (CPU), ~50-100 tokens/sec (GPU)
- Generation: ~8-12 tokens/sec (CPU), ~40-80 tokens/sec (GPU)
35B MoE Model:
- Prompt Processing: ~5-8 tokens/sec (CPU+GPU mix), ~30-50 tokens/sec (full GPU)
- Generation: ~4-8 tokens/sec (CPU+GPU mix), ~20-40 tokens/sec (full GPU)
Note: MoE models can be slower due to expert routing overhead, but provide better quality.
Tuning Tips
- Increase GPU layers (
-ngl) for better speed - Enable Flash Attention:
--flash-attn on(if supported) - Use a smaller quantization (Q4_K_M instead of Q8_0)
- Reduce context window size if memory-bound
- KV Cache Quantization:
--cache-type-k q8_0to reduce KV cache memory
Related
- GPU Offloading — Optimize GPU layers for your hardware
- Troubleshooting — Fix performance issues