---
type: reference
title: GPU Layer Offloading Guidelines
description: How many GPU layers to offload based on your VRAM size for both Ornith 9B and 35B models.
tags:
  - unverified
  - ornith
  - gpu
  - offloading
  - llama.cpp
timestamp: 2026-07-18
verified: 2026-08-28
---

# GPU Layer Offloading

## For 9B Q4_K_M (5.63 GB):

- **8GB VRAM:** `-ngl 35-40` (fits most layers)
- **12GB VRAM:** `-ngl 99` (all layers)
- **16GB+ VRAM:** `-ngl 99` (all layers + room for large KV cache)

## For 35B Q4_K_M (~20 GB):

- **8GB VRAM:** `-ngl 10-14` (partial offload)
- **12GB VRAM:** `-ngl 18-22` (partial offload)
- **16GB VRAM:** `-ngl 28-32` (partial offload)
- **24GB VRAM:** `-ngl 40-48` (majority of layers)
- **40GB+ VRAM:** `-ngl 99` (all layers)

**Note:** The 35B model is a MoE architecture, so actual memory usage depends on which experts are activated.

## Context Window Considerations

- **9B model:** Supports up to 262,144 tokens
- **35B model:** Supports up to 262,144 tokens
- **KV Cache Memory:** Larger context windows require more VRAM/RAM for the key-value cache
  - 131K context: ~8-16 GB depending on model and quantization
  - 256K context: ~16-32 GB depending on model and quantization

**Recommendation:** Start with 131K context if you have limited VRAM, increase to 256K if you have headroom.

## Related

* [Installation](/tools/ornith-llama-cpp-setup/installation.md) — How to set the -ngl flag
* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Memory requirements