---
type: howto
title: Installing Ornith + llama.cpp
description: Step-by-step setup — clone and build llama.cpp, download models, start the server, verify it works.
tags:
  - ornith
  - installation
  - llama.cpp
  - unverified
verified: 2026-08-28
---

# Installation

## 1. Clone and Build llama.cpp

```bash
cd ~/GitPackages
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release
```

## 2. Download the Model

```bash
mkdir -p ~/models/ornith
cd ~/models/ornith

# For 9B model (faster, less memory)
wget https://huggingface.co/ornith-ai/Ornith-1.0-9B-GGUF/resolve/main/ornith-1.0-9b-Q4_K_M.gguf

# OR for 35B MoE model (better performance, more memory)
wget https://huggingface.co/ornith-ai/Ornith-1.0-35B-GGUF/resolve/main/ornith-1.0-35b-Q4_K_M.gguf
```

## 3. Start the Server

**For 9B model with full GPU offload (8GB+ VRAM):**
```bash
cd ~/GitPackages/llama.cpp
./build/bin/llama-server \
  -m ~/models/ornith/ornith-1.0-9b-Q4_K_M.gguf \
  -c 262144 \
  -ngl 99 \
  --host 0.0.0.0 \
  --port 8080
```

**For 35B MoE model with partial GPU offload (8GB VRAM example):**
```bash
cd ~/GitPackages/llama.cpp
./build/bin/llama-server \
  -m ~/models/ornith/ornith-1.0-35b-Q4_K_M.gguf \
  -c 131072 \
  -ngl 12 \
  --host 0.0.0.0 \
  --port 8080
```

**Key Parameters:**
- `-m`: Path to GGUF model file
- `-c` / `--ctx-size`: Context window size (131072 or 262144)
- `-ngl` / `--n-gpu-layers`: Number of layers to offload to GPU
  - 99 = all layers
  - Lower values = more CPU inference
- `--host`: Bind address (0.0.0.0 for all interfaces, 127.0.0.1 for local only)
- `--port`: Server port (default 8080)

## 4. Verify Server is Running

```bash
curl http://localhost:8080/health
# Expected: {"status":"ok"}
```

## 5. Test the Model

```bash
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "frank-default-local",
    "messages": [
      {"role": "user", "content": "Hello, what is 2+2?"}
    ],
    "max_tokens": 150,
    "temperature": 0.7
  }' | jq .
```

## Related

* [Model Specs](/tools/ornith-llama-cpp-setup/model-specs.md) — Model options and quantization
* [GPU Offloading](/tools/ornith-llama-cpp-setup/gpu-offloading.md) — Tuning GPU layer counts
* [Pi Configuration](/tools/ornith-llama-cpp-setup/pi-configuration.md) — Adding to models.json