command-center/agent/managed-skills/llamacpp-mtp-vision-endpoint-bringup/SKILL.md

name, description

name	description
llamacpp-mtp-vision-endpoint-bringup	Bring up a new llama.cpp (mainline) OpenAI endpoint on a single CUDA GPU with MTP speculative decoding and vision — GGUF header preflight by HTTP range, hybrid-architecture KV arithmetic, the -mtp / --spec-type draft-mtp contract, --parallel vs ctx math, and the verification battery that proves MTP, tools and vision actually work.

llama.cpp MTP + vision endpoint bring-up (single CUDA GPU)

Use when standing up or swapping a mainline llama.cpp endpoint for a GGUF that may ship an MTP (NextN) head and a vision projector. Verified end to end on Dankclaw 2026-09-16 (RTX 3090 24 GiB, Qwen3.8-27B-GSQ-RCO IQ3_S -mtp + BF16 mmproj, 131072 ctx / 1 slot, port 7771).

1. Preflight the GGUF WITHOUT downloading it

GGUF metadata lives in the header, and HuggingFace honours ranged GETs, so a few MB answers every sizing question before a 13 GB transfer:

import urllib.request
req = urllib.request.Request(url, headers={"Range": "bytes=0-8000000", "User-Agent": "curl/8"})
buf = urllib.request.urlopen(req, timeout=180).read()

Parse: magic GGUF, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers forget). Walk arrays by byte size even when you only want a summary: a naive parser desynchronises on tokenizer.ggml.tokens and lands gigabytes off.

Read these keys: general.architecture, <arch>.block_count, <arch>.context_length, <arch>.nextn_predict_layers, <arch>.head_count_kv, <arch>.key_length / value_length, <arch>.full_attention_interval, <arch>.ssm.state_size / group_count / time_step_rank, and for the projector file clip.projector_type (e.g. qwen3vl_merger) plus clip.has_vision_encoder.

2. Decide the file from nextn_predict_layers

MTP is a property of the FILE, not of the flags. If the release ships an -mtp build, that build carries the head as blk.<n>.nextn.* (a few hundred MB larger); the plain build has no MTP tensors at all.

With the -mtp file, llama.cpp needs only --spec-type draft-mtp: it sets load_mtp and the draft context runs on the MAIN model's weights (shares_model, LLAMA_CONTEXT_TYPE_MTP). Do NOT add --model-draft — that loads the whole file a second time (12 GiB twice on a 24 GiB card). A separate -md is only correct for standalone MTP-only sidecars.

Confirm the flag exists in the deployed build before writing the script: grep -rn "common_speculative_all_types_str" common/speculative.cpp and llama-server --help | grep -A4 spec-type.

3. KV arithmetic (do this before choosing ctx)

Only full-attention layers take KV. For a hybrid arch (Qwen3-Next/3.8 style 16 x (3 x Gated DeltaNet + 1 x Gated Attention)):

attn_layers = block_count / full_attention_interval
bytes/token = attn_layers x n_head_kv x head_dim x 2 (K,V) x bytes_per_elem
              f16 = 2 B | q8_0 = 1.0625 B | q4_0 = 0.5625 B

The linear-attention layers hold constant-size per-sequence state instead (~0.16 GiB per sequence at 48 layers / 128 d_state / 48 v-heads in f32), and the MTP draft context costs its own n_ctx x (1 layer x n_head_kv x head_dim x 2 x 2 B) ≈ 4 KiB/token for a 1-layer head.

VRAM budget: weights + mmproj + MTP draft KV + DeltaNet state + ~1.5 GiB of CUDA context and compute buffers. On a 24 GiB card that leaves ~9.5 GiB for KV, which at 64 KiB/token is ~155k tokens total.

--parallel N DIVIDES --ctx-size between slots — it does not multiply. -np 1 -c 131072 is one 131k conversation; -np 4 -c 131072 is four 32k ones.

4. Flags that matter

• -ngl 999, -fa on, -ctk/-ctv f16 for the usual shape.
• Quantized KV (q8_0, q4_0) requires a build with -DGGML_CUDA_FA_ALL_QUANTS=ON; without it CUDA prefill collapses ~12x. f16 needs nothing, so prefer it unless the context demand is real.
• --jinja — pin it explicitly even when the build defaults it on: the GGUF's own template supplies reasoning_effort, preserve_thinking and the tool-call grammar, and a future default flip must not silently remove tools.
• --metrics if anything scrapes /metrics, --alias to set the served id, --temp 0.0 for a greedy-by-default fleet contract, --reasoning-format deepseek --reasoning-budget -1 to project thinking into reasoning_content.

5. Serve-script and unit invariants

Fail loudly before loading VRAM: check the binary, the model file and the mmproj exist, and exit 1 with a named message otherwise — a missing weight must not become a Restart=on-failure crash loop that reads like a runtime fault. For a native CUDA binary the unit is a ROOT systemd unit ([Install] WantedBy=multi-user.target, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity); a container would be a Quadlet.

6. Verification battery (prove each claim, do not infer)

1. /health 200, and /v1/models reports the intended alias.
2. A chat completion returns content AND reasoning_content.
3. A request carrying tools returns tool_calls — a RUNTIME property: the ik_llama.cpp runtime answered 500 tools param requires --jinja flag.
4. GET /metrics after a generation: llamacpp:spec_decode_num_draft_tokens_total and ..._accepted_tokens_total both > 0, acceptance 0.6-0.95 ⇒ MTP engaged, not merely configured. The slot timing lines in the journal say the same.
5. One image request (data: URL or bare base64) ⇒ an answer that reflects the image; confirm the journal printed a multimodal projector load.
6. nvidia-smi memory used is close to the modelled total, and n_slots / n_ctx_slot in the load log match the intended shape.

Traps that cost real time

• A template can reject the client's thinking level. Qwen3.8 accepts only xhigh|medium|low for reasoning_effort and raises otherwise — HTTP 500 naming Jinja, not the model. Map every harness level onto the accepted set (e.g. compat.reasoningEffortMap: {minimal: low, low: low, medium: medium, high: xhigh, max: xhigh}) and probe each level with curl before routing roles at the endpoint.
• Thinking counts against max_tokens. A budget that runs out mid-thought returns finish_reason: "length" with empty content — not a broken endpoint, not a vision failure. Raise the budget.
• Never point --model-draft at the -mtp file (step 2).
• Any -ot / -ncmoe-style override can silently disable mmap for the whole load, turning a mapped model into anonymous memory.
• On a host that publishes ports through pasta, the replacement must not start until the predecessor released the port: poll ss -tan "sport = :<port>" (listener, accepted AND TIME_WAIT) or the container dies with Address already in use.
• On a shared host, read the serving engine's own boot ledger for device budget vs need: a pool that silently clamped still answers a green /health.