Frank
02883d3ed2
Copy of the legacy ~/.pi/agent/docs tree rebuilt as the command-center knowledge base: concept files promoted to path identity, index listings stripped of frontmatter, all wikilinks/relative links resolved, every checkable claim verified against the live fleet (Dankserver = Ryzen 7 7700X + RTX 3090; Strix Halo is Dankclaw), unverifiable claims tagged. kb-lint exits 0.
1.5 KiB
1.5 KiB
type, title, description, tags, timestamp, verified
| type | title | description | tags | timestamp | verified | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| reference | GPU Layer Offloading Guidelines | How many GPU layers to offload based on your VRAM size for both Ornith 9B and 35B models. |
|
2026-07-18 | 2026-08-28 |
GPU Layer Offloading
For 9B Q4_K_M (5.63 GB):
- 8GB VRAM:
-ngl 35-40(fits most layers) - 12GB VRAM:
-ngl 99(all layers) - 16GB+ VRAM:
-ngl 99(all layers + room for large KV cache)
For 35B Q4_K_M (~20 GB):
- 8GB VRAM:
-ngl 10-14(partial offload) - 12GB VRAM:
-ngl 18-22(partial offload) - 16GB VRAM:
-ngl 28-32(partial offload) - 24GB VRAM:
-ngl 40-48(majority of layers) - 40GB+ VRAM:
-ngl 99(all layers)
Note: The 35B model is a MoE architecture, so actual memory usage depends on which experts are activated.
Context Window Considerations
- 9B model: Supports up to 262,144 tokens
- 35B model: Supports up to 262,144 tokens
- KV Cache Memory: Larger context windows require more VRAM/RAM for the key-value cache
- 131K context: ~8-16 GB depending on model and quantization
- 256K context: ~16-32 GB depending on model and quantization
Recommendation: Start with 131K context if you have limited VRAM, increase to 256K if you have headroom.
Related
- Installation — How to set the -ngl flag
- Model Specs — Memory requirements