command-center/omp-docs/tools/llama-cpp/setup.md

type, title, description, tags, verified

type	title	description	tags	verified
howto	llama.cpp — Build and Deployment Setup	Build guides for CPU, CUDA, Metal, Vulkan, and other GPU backends, plus Docker Compose and systemd service configuration.	llm inference llama-cpp build deployment	2026-08-28

llama.cpp — Build and Deployment Setup

See Overview for server usage and API reference.

---

Building llama.cpp

Prerequisites

• Git
• CMake (3.12+)
• C++ compiler (GCC 9+, Clang 12+, MSVC 2019+)

CPU-only Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build
cmake --build build --config Release -j 8

The llama-server binary will be at ./build/bin/llama-server.

Faster compilation:

• Use Ninja: cmake -B build -G Ninja
• Install ccache for incremental builds

Debug build:

cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build

CUDA Build (NVIDIA GPU)

# Requires CUDA toolkit installed
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

Non-native build (works on all CUDA GPUs):

cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build build --config Release

Override compute capabilities:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89"

Metal Build (Apple Silicon)

Metal is enabled by default on macOS. No special flags needed:

cmake -B build
cmake --build build --config Release

Disable GPU offloading at runtime with --n-gpu-layers 0.

Other GPU Backends

Backend	Flag	Platform
HIP/ROCm (AMD GPU)	-DGGML_HIP=ON	Linux
Vulkan	-DGGML_VULKAN=ON	All
SYCL (Intel GPU)	-DGGML_SYCL=ON	Linux
CANN (Ascend NPU)	-DGGML_CANN=ON	Linux
OpenVINO	-DGGML_OPENVINO=ON	Intel
ZenDNN (AMD CPU)	-DGGML_ZENDNN=ON	Linux
WebGPU	-DGGML_WEBGPU=ON	All (WASM)

SSL/TLS Support

# Requires OpenSSL dev libraries
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server

---

Key Command-Line Options

Model Loading

Flag	Description	Default
-m, --model FNAME	Path to GGUF model file	required
-hf, --hf-repo USER/MODEL	Download from Hugging Face	—
-mmproj FILE	Multimodal projector file	—
-lora FNAME	LoRA adapter (repeatable)	—
-ngl, --gpu-layers N	Layers to offload to GPU	auto

Context & Performance

Flag	Description	Default
-c, --ctx-size N	Prompt context size	0 (from model)
-t, --threads N	CPU threads	-1 (all)
-b, --batch-size N	Max logical batch size	2048
--flash-attn	Flash Attention	auto
--mlock	Keep model in RAM (no swap)	false
--mmap	Memory-map model file	true
--fit	Auto-fit to GPU memory	on

Server Configuration

Flag	Description	Default
--host HOST	Bind address	127.0.0.1
--port PORT	Port number	8080
-np, --parallel N	Concurrent slots	-1 (auto)
--api-key KEY	API key (repeatable)	none
--metrics	Prometheus metrics	disabled
--jinja	Jinja template engine	enabled

Sampling Parameters

Flag	Description	Default
-s, --seed N	RNG seed	-1 (random)
--temp, --temperature N	Temperature	0.80
--top-k N	Top-k sampling	40
--top-p N	Top-p sampling	0.95
--min-p N	Min-p sampling	0.05
--repeat-penalty N	Repetition penalty	1.1

---

GPU Offloading

# Offload all layers to GPU
llama-server -m model.gguf --n-gpu-layers 99

# Auto-detect (default)
llama-server -m model.gguf --n-gpu-layers auto

# Disable GPU (CPU-only)
llama-server -m model.gguf --n-gpu-layers 0

# Multi-GPU (layer mode)
llama-server -m model.gguf --device 0,1 --split-mode layer

# Multi-GPU (row split)
llama-server -m model.gguf --device 0,1 --split-mode row

---

Context Length

# Use model's native context
llama-server -m model.gguf -c 0

# Set specific context
llama-server -m model.gguf -c 8192

# RoPE scaling to extend beyond trained size
llama-server -m model.gguf -c 65536 --rope-scale 2

# YaRN scaling
llama-server -m model.gguf -c 65536 --rope-scaling yarn --yarn-orig-ctx 8192

Important: Larger context = more VRAM/RAM usage (KV cache grows linearly). Reduce with quantization:

llama-server -m model.gguf -c 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0

---

Speculative Decoding

# Draft model based
llama-server -m main_model.gguf -md draft_model.gguf

# N-gram based (no draft model needed)
llama-server -m model.gguf --spec-type ngram-mod --spec-ngram-mod-n-max 64

---

Docker Compose Example

services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 8192
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_PORT: 8080
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

---

Running as systemd Service

# /etc/systemd/system/llama-server.service
[Unit]
Description=llama.cpp Server
After=network.target

[Service]
Type=simple
User=dank
ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \
  -m /home/dank/models/my_model.gguf -c 131072 -ngl 12 \
  --host 0.0.0.0 --port 8080
Restart=on-failure
WorkingDirectory=/home/dank/GitPackages/llama.cpp

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server

---

Cross-References

• See Overview for API reference and usage examples.
• See Ornith Setup for model-specific configuration.