---
name: llamacpp-mtp-vision-endpoint-bringup
description: "Bring up a new llama.cpp (mainline) OpenAI endpoint on a single CUDA GPU with MTP speculative decoding and vision — GGUF header preflight by HTTP range, hybrid-architecture KV arithmetic, the -mtp / --spec-type draft-mtp contract, --parallel vs ctx math, and the verification battery that proves MTP, tools and vision actually work."
---

# llama.cpp MTP + vision endpoint bring-up (single CUDA GPU)

Use when standing up or swapping a mainline llama.cpp endpoint for a GGUF that
may ship an MTP (NextN) head and a vision projector. Verified end to end on
Dankclaw 2026-09-16 (RTX 3090 24 GiB, Qwen3.8-27B-GSQ-RCO IQ3_S `-mtp` +
BF16 mmproj, 131072 ctx / 1 slot, port 7771).

## 1. Preflight the GGUF WITHOUT downloading it

GGUF metadata lives in the header, and HuggingFace honours ranged GETs, so a
few MB answers every sizing question before a 13 GB transfer:

```python
import urllib.request
req = urllib.request.Request(url, headers={"Range": "bytes=0-8000000", "User-Agent": "curl/8"})
buf = urllib.request.urlopen(req, timeout=180).read()
```

Parse: magic `GGUF`, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs
(string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte —
the type hand-written parsers forget). Walk arrays by byte size even when you
only want a summary: a naive parser desynchronises on `tokenizer.ggml.tokens`
and lands gigabytes off.

Read these keys: `general.architecture`, `<arch>.block_count`,
`<arch>.context_length`, `<arch>.nextn_predict_layers`, `<arch>.head_count_kv`,
`<arch>.key_length` / `value_length`, `<arch>.full_attention_interval`,
`<arch>.ssm.state_size` / `group_count` / `time_step_rank`, and for the
projector file `clip.projector_type` (e.g. `qwen3vl_merger`) plus
`clip.has_vision_encoder`.

## 2. Decide the file from `nextn_predict_layers`

**MTP is a property of the FILE, not of the flags.** If the release ships an
`-mtp` build, that build carries the head as `blk.<n>.nextn.*` (a few hundred
MB larger); the plain build has no MTP tensors at all.

With the `-mtp` file, llama.cpp needs only `--spec-type draft-mtp`: it sets
`load_mtp` and the draft context runs on the MAIN model's weights
(`shares_model`, `LLAMA_CONTEXT_TYPE_MTP`). Do NOT add `--model-draft` — that
loads the whole file a second time (12 GiB twice on a 24 GiB card). A separate
`-md` is only correct for standalone MTP-only sidecars.

Confirm the flag exists in the deployed build before writing the script:
`grep -rn "common_speculative_all_types_str" common/speculative.cpp` and
`llama-server --help | grep -A4 spec-type`.

## 3. KV arithmetic (do this before choosing ctx)

Only **full-attention** layers take KV. For a hybrid arch (Qwen3-Next/3.8 style
`16 x (3 x Gated DeltaNet + 1 x Gated Attention)`):

```
attn_layers = block_count / full_attention_interval
bytes/token = attn_layers x n_head_kv x head_dim x 2 (K,V) x bytes_per_elem
              f16 = 2 B | q8_0 = 1.0625 B | q4_0 = 0.5625 B
```

The linear-attention layers hold constant-size per-sequence state instead
(~0.16 GiB per sequence at 48 layers / 128 d_state / 48 v-heads in f32), and
the MTP draft context costs its own `n_ctx x (1 layer x n_head_kv x head_dim x
2 x 2 B)` ≈ 4 KiB/token for a 1-layer head.

VRAM budget: weights + mmproj + MTP draft KV + DeltaNet state + ~1.5 GiB of
CUDA context and compute buffers. On a 24 GiB card that leaves ~9.5 GiB for KV,
which at 64 KiB/token is ~155k tokens total.

**`--parallel N` DIVIDES `--ctx-size` between slots** — it does not multiply.
`-np 1 -c 131072` is one 131k conversation; `-np 4 -c 131072` is four 32k ones.

## 4. Flags that matter

- `-ngl 999`, `-fa on`, `-ctk/-ctv f16` for the usual shape.
- Quantized KV (`q8_0`, `q4_0`) requires a build with
  `-DGGML_CUDA_FA_ALL_QUANTS=ON`; without it CUDA prefill collapses ~12x. f16
  needs nothing, so prefer it unless the context demand is real.
- `--jinja` — pin it explicitly even when the build defaults it on: the GGUF's
own template supplies `reasoning_effort`, `preserve_thinking` and the
tool-call grammar, and a future default flip must not silently remove tools.
- `--metrics` if anything scrapes `/metrics`, `--alias` to set the served id,
  `--temp 0.0` for a greedy-by-default fleet contract,
  `--reasoning-format deepseek --reasoning-budget -1` to project thinking into
  `reasoning_content`.

## 5. Serve-script and unit invariants

Fail loudly before loading VRAM: check the binary, the model file and the
mmproj exist, and `exit 1` with a named message otherwise — a missing weight
must not become a `Restart=on-failure` crash loop that reads like a runtime
fault. For a native CUDA binary the unit is a ROOT systemd unit
(`[Install] WantedBy=multi-user.target`, `Restart=on-failure`,
`RestartSec=10`, `LimitMEMLOCK=infinity`); a container would be a Quadlet.

## 6. Verification battery (prove each claim, do not infer)

1. `/health` 200, and `/v1/models` reports the intended alias.
2. A chat completion returns `content` AND `reasoning_content`.
3. A request carrying `tools` returns `tool_calls` — a RUNTIME property: the
   ik_llama.cpp runtime answered `500 tools param requires --jinja flag`.
4. `GET /metrics` after a generation: `llamacpp:spec_decode_num_draft_tokens_total`
   and `..._accepted_tokens_total` both > 0, acceptance 0.6-0.95 ⇒ MTP engaged,
   not merely configured. The slot timing lines in the journal say the same.
5. One image request (`data:` URL or bare base64) ⇒ an answer that reflects the
   image; confirm the journal printed a multimodal projector load.
6. `nvidia-smi` memory used is close to the modelled total, and `n_slots` /
   `n_ctx_slot` in the load log match the intended shape.

## Traps that cost real time

- **A template can reject the client's thinking level.** Qwen3.8 accepts only
  `xhigh|medium|low` for `reasoning_effort` and raises otherwise — HTTP 500
  naming Jinja, not the model. Map every harness level onto the accepted set
  (e.g. `compat.reasoningEffortMap: {minimal: low, low: low, medium: medium,
  high: xhigh, max: xhigh}`) and probe each level with curl before routing
  roles at the endpoint.
- **Thinking counts against `max_tokens`.** A budget that runs out mid-thought
  returns `finish_reason: "length"` with empty `content` — not a broken
  endpoint, not a vision failure. Raise the budget.
- **Never point `--model-draft` at the `-mtp` file** (step 2).
- **Any `-ot` / `-ncmoe`-style override can silently disable mmap** for the
  whole load, turning a mapped model into anonymous memory.
- On a host that publishes ports through **pasta**, the replacement must not
  start until the predecessor released the port: poll
  `ss -tan "sport = :<port>"` (listener, accepted AND TIME_WAIT) or the
  container dies with `Address already in use`.
- On a shared host, read the serving engine's own boot ledger for device
  budget vs need: a pool that silently clamped still answers a green `/health`.