---
type: reference
title: Performance Benchmarks
description: Token throughput benchmarks for Ornith 9B and 35B models with GPU vs CPU inference.
tags:
  - ornith
  - performance
  - benchmarks
  - unverified
verified: 2026-08-28
---

# Performance
# Performance

## 9B Model:

- **Prompt Processing:** ~10-15 tokens/sec (CPU), ~50-100 tokens/sec (GPU)
- **Generation:** ~8-12 tokens/sec (CPU), ~40-80 tokens/sec (GPU)

## 35B MoE Model:

- **Prompt Processing:** ~5-8 tokens/sec (CPU+GPU mix), ~30-50 tokens/sec (full GPU)
- **Generation:** ~4-8 tokens/sec (CPU+GPU mix), ~20-40 tokens/sec (full GPU)

**Note:** MoE models can be slower due to expert routing overhead, but provide better quality.

## Tuning Tips

1. **Increase GPU layers** (`-ngl`) for better speed
2. **Enable Flash Attention:** `--flash-attn on` (if supported)
3. **Use a smaller quantization** (Q4_K_M instead of Q8_0)
4. **Reduce context window size** if memory-bound
5. **KV Cache Quantization:** `--cache-type-k q8_0` to reduce KV cache memory

## Related

* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Optimize GPU layers for your hardware
* [Troubleshooting](/tools/ornith-llama-cpp-setup/troubleshooting.md) — Fix performance issues