command-center/omp-docs/tools/llama-cpp/setup.md

---
type: howto
title: llama.cpp — Build and Deployment Setup
description: Build guides for CPU, CUDA, Metal, Vulkan, and other GPU backends, plus Docker Compose and systemd service configuration.
tags:
  - llm
  - inference
  - llama-cpp
  - build
  - deployment
verified: 2026-08-28
---

# llama.cpp — Build and Deployment Setup

See [Overview](index.md) for server usage and API reference.

---

## Building llama.cpp

### Prerequisites

- Git
- CMake (3.12+)
- C++ compiler (GCC 9+, Clang 12+, MSVC 2019+)

### CPU-only Build

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build
cmake --build build --config Release -j 8
```

The `llama-server` binary will be at `./build/bin/llama-server`.

**Faster compilation:**
- Use Ninja: `cmake -B build -G Ninja`
- Install `ccache` for incremental builds

**Debug build:**
```bash
cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build
```

### CUDA Build (NVIDIA GPU)

```bash
# Requires CUDA toolkit installed
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8
```

**Non-native build (works on all CUDA GPUs):**
```bash
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build build --config Release
```

**Override compute capabilities:**
```bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89"
```

### Metal Build (Apple Silicon)

Metal is enabled by default on macOS. No special flags needed:

```bash
cmake -B build
cmake --build build --config Release
```

Disable GPU offloading at runtime with `--n-gpu-layers 0`.

### Other GPU Backends

| Backend | Flag | Platform |
|---------|------|----------|
| HIP/ROCm (AMD GPU) | `-DGGML_HIP=ON` | Linux |
| Vulkan | `-DGGML_VULKAN=ON` | All |
| SYCL (Intel GPU) | `-DGGML_SYCL=ON` | Linux |
| CANN (Ascend NPU) | `-DGGML_CANN=ON` | Linux |
| OpenVINO | `-DGGML_OPENVINO=ON` | Intel |
| ZenDNN (AMD CPU) | `-DGGML_ZENDNN=ON` | Linux |
| WebGPU | `-DGGML_WEBGPU=ON` | All (WASM) |

### SSL/TLS Support

```bash
# Requires OpenSSL dev libraries
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server
```

---

## Key Command-Line Options

### Model Loading

| Flag | Description | Default |
|------|-------------|---------|
| `-m, --model FNAME` | Path to GGUF model file | required |
| `-hf, --hf-repo USER/MODEL` | Download from Hugging Face | — |
| `-mmproj FILE` | Multimodal projector file | — |
| `-lora FNAME` | LoRA adapter (repeatable) | — |
| `-ngl, --gpu-layers N` | Layers to offload to GPU | auto |

### Context & Performance

| Flag | Description | Default |
|------|-------------|---------|
| `-c, --ctx-size N` | Prompt context size | 0 (from model) |
| `-t, --threads N` | CPU threads | -1 (all) |
| `-b, --batch-size N` | Max logical batch size | 2048 |
| `--flash-attn` | Flash Attention | auto |
| `--mlock` | Keep model in RAM (no swap) | false |
| `--mmap` | Memory-map model file | true |
| `--fit` | Auto-fit to GPU memory | on |

### Server Configuration

| Flag | Description | Default |
|------|-------------|---------|
| `--host HOST` | Bind address | 127.0.0.1 |
| `--port PORT` | Port number | 8080 |
| `-np, --parallel N` | Concurrent slots | -1 (auto) |
| `--api-key KEY` | API key (repeatable) | none |
| `--metrics` | Prometheus metrics | disabled |
| `--jinja` | Jinja template engine | enabled |

### Sampling Parameters

| Flag | Description | Default |
|------|-------------|---------|
| `-s, --seed N` | RNG seed | -1 (random) |
| `--temp, --temperature N` | Temperature | 0.80 |
| `--top-k N` | Top-k sampling | 40 |
| `--top-p N` | Top-p sampling | 0.95 |
| `--min-p N` | Min-p sampling | 0.05 |
| `--repeat-penalty N` | Repetition penalty | 1.1 |

---

## GPU Offloading

```bash
# Offload all layers to GPU
llama-server -m model.gguf --n-gpu-layers 99

# Auto-detect (default)
llama-server -m model.gguf --n-gpu-layers auto

# Disable GPU (CPU-only)
llama-server -m model.gguf --n-gpu-layers 0

# Multi-GPU (layer mode)
llama-server -m model.gguf --device 0,1 --split-mode layer

# Multi-GPU (row split)
llama-server -m model.gguf --device 0,1 --split-mode row
```

---

## Context Length

```bash
# Use model's native context
llama-server -m model.gguf -c 0

# Set specific context
llama-server -m model.gguf -c 8192

# RoPE scaling to extend beyond trained size
llama-server -m model.gguf -c 65536 --rope-scale 2

# YaRN scaling
llama-server -m model.gguf -c 65536 --rope-scaling yarn --yarn-orig-ctx 8192
```

**Important**: Larger context = more VRAM/RAM usage (KV cache grows linearly). Reduce with quantization:

```bash
llama-server -m model.gguf -c 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0
```

---

## Speculative Decoding

```bash
# Draft model based
llama-server -m main_model.gguf -md draft_model.gguf

# N-gram based (no draft model needed)
llama-server -m model.gguf --spec-type ngram-mod --spec-ngram-mod-n-max 64
```

---

## Docker Compose Example

```yaml
services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 8192
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_PORT: 8080
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

---

## Running as systemd Service

```ini
# /etc/systemd/system/llama-server.service
[Unit]
Description=llama.cpp Server
After=network.target

[Service]
Type=simple
User=dank
ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \
  -m /home/dank/models/my_model.gguf -c 131072 -ngl 12 \
  --host 0.0.0.0 --port 8080
Restart=on-failure
WorkingDirectory=/home/dank/GitPackages/llama.cpp

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server
```

---

## Cross-References

- See [Overview](index.md) for API reference and usage examples.