Files

T

Frank 39385636be docs(llama): delete fabricated ornith-llama-cpp-setup tree (608 lines)

Night fact-check falsified the tree's subject: no `ornith` CLI exists
(npm 404), github.com/rhitching/ornith is 404, and the model IDs the
tree cites are wrong — the real family is ornith-ai/Ornith-1.0/1.5
(9B, 35B-A3B, 397B), built on Qwen3.5/Gemma4 per its model card, not
the 'Ornith-9B-Thinking-1.5T based on Qwen3-8B' the docs assert.
Nothing in the tree describes this fleet: no ornith binary on PATH
(audit 2026-10-18 said the same). Real facts it echoed (PrismML fork
on frank-zealot) already live in /system/.

Also: tools/index.md de-duplicated (same files were listed twice, once
as wikilinks once as links, live-sync missorted) and links normalized;
llama-cpp setup/index xrefs retargeted.

Refs #29

2026-10-19 03:17:49 -07:00

5.9 KiB

Raw Blame History

type, title, description, tags, verified

type

title

description

llama.cpp — Build and Deployment Setup

See Overview for server usage and API reference.

Building llama.cpp

Prerequisites

Git
CMake (3.12+)
C++ compiler (GCC 9+, Clang 12+, MSVC 2019+)

CPU-only Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build
cmake --build build --config Release -j 8

The llama-server binary will be at ./build/bin/llama-server.

Faster compilation:

Use Ninja: cmake -B build -G Ninja
Install ccache for incremental builds

Debug build:

cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build

CUDA Build (NVIDIA GPU)

# Requires CUDA toolkit installed
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

Non-native build (works on all CUDA GPUs):

cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
cmake --build build --config Release

Override compute capabilities:

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86;89"

Metal Build (Apple Silicon)

Metal is enabled by default on macOS. No special flags needed:

cmake -B build
cmake --build build --config Release

Disable GPU offloading at runtime with --n-gpu-layers 0.

Other GPU Backends

Backend	Flag	Platform
HIP/ROCm (AMD GPU)	`-DGGML_HIP=ON`	Linux
Vulkan	`-DGGML_VULKAN=ON`	All
SYCL (Intel GPU)	`-DGGML_SYCL=ON`	Linux
CANN (Ascend NPU)	`-DGGML_CANN=ON`	Linux
OpenVINO	`-DGGML_OPENVINO=ON`	Intel
ZenDNN (AMD CPU)	`-DGGML_ZENDNN=ON`	Linux
WebGPU	`-DGGML_WEBGPU=ON`	All (WASM)

SSL/TLS Support

# Requires OpenSSL dev libraries
cmake -B build -DLLAMA_OPENSSL=ON
cmake --build build --config Release -t llama-server

Key Command-Line Options

Model Loading

Flag	Description	Default
`-m, --model FNAME`	Path to GGUF model file	required
`-hf, --hf-repo USER/MODEL`	Download from Hugging Face	—
`-mmproj FILE`	Multimodal projector file	—
`-lora FNAME`	LoRA adapter (repeatable)	—
`-ngl, --gpu-layers N`	Layers to offload to GPU	auto

Context & Performance

Flag	Description	Default
`-c, --ctx-size N`	Prompt context size	0 (from model)
`-t, --threads N`	CPU threads	-1 (all)
`-b, --batch-size N`	Max logical batch size	2048
`--flash-attn`	Flash Attention	auto
`--mlock`	Keep model in RAM (no swap)	false
`--mmap`	Memory-map model file	true
`--fit`	Auto-fit to GPU memory	on

Server Configuration

Flag	Description	Default
`--host HOST`	Bind address	127.0.0.1
`--port PORT`	Port number	8080
`-np, --parallel N`	Concurrent slots	-1 (auto)
`--api-key KEY`	API key (repeatable)	none
`--metrics`	Prometheus metrics	disabled
`--jinja`	Jinja template engine	enabled

Sampling Parameters

Flag	Description	Default
`-s, --seed N`	RNG seed	-1 (random)
`--temp, --temperature N`	Temperature	0.80
`--top-k N`	Top-k sampling	40
`--top-p N`	Top-p sampling	0.95
`--min-p N`	Min-p sampling	0.05
`--repeat-penalty N`	Repetition penalty	1.1

GPU Offloading

# Offload all layers to GPU
llama-server -m model.gguf --n-gpu-layers 99

# Auto-detect (default)
llama-server -m model.gguf --n-gpu-layers auto

# Disable GPU (CPU-only)
llama-server -m model.gguf --n-gpu-layers 0

# Multi-GPU (layer mode)
llama-server -m model.gguf --device 0,1 --split-mode layer

# Multi-GPU (row split)
llama-server -m model.gguf --device 0,1 --split-mode row

Context Length

# Use model's native context
llama-server -m model.gguf -c 0

# Set specific context
llama-server -m model.gguf -c 8192

# RoPE scaling to extend beyond trained size
llama-server -m model.gguf -c 65536 --rope-scale 2

# YaRN scaling
llama-server -m model.gguf -c 65536 --rope-scaling yarn --yarn-orig-ctx 8192

Important: Larger context = more VRAM/RAM usage (KV cache grows linearly). Reduce with quantization:

llama-server -m model.gguf -c 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0

Speculative Decoding

# Draft model based
llama-server -m main_model.gguf -md draft_model.gguf

# N-gram based (no draft model needed)
llama-server -m model.gguf --spec-type ngram-mod --spec-ngram-mod-n-max 64

Docker Compose Example

services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 8192
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_PORT: 8080
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Running as systemd Service

# /etc/systemd/system/llama-server.service
[Unit]
Description=llama.cpp Server
After=network.target

[Service]
Type=simple
User=dank
ExecStart=/home/dank/GitPackages/llama.cpp/build/bin/llama-server \
  -m /home/dank/models/my_model.gguf -c 131072 -ngl 12 \
  --host 0.0.0.0 --port 8080
Restart=on-failure
WorkingDirectory=/home/dank/GitPackages/llama.cpp

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable llama-server
sudo systemctl start llama-server

Cross-References

See Overview for API reference and usage examples.

5.9 KiB Raw Blame History