Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
133 lines
6.8 KiB
Markdown
133 lines
6.8 KiB
Markdown
---
|
|
name: llamacpp-mtp-vision-endpoint-bringup
|
|
description: "Bring up a new llama.cpp (mainline) OpenAI endpoint on a single CUDA GPU with MTP speculative decoding and vision — GGUF header preflight by HTTP range, hybrid-architecture KV arithmetic, the -mtp / --spec-type draft-mtp contract, --parallel vs ctx math, and the verification battery that proves MTP, tools and vision actually work."
|
|
---
|
|
|
|
# llama.cpp MTP + vision endpoint bring-up (single CUDA GPU)
|
|
|
|
Use when standing up or swapping a mainline llama.cpp endpoint for a GGUF that
|
|
may ship an MTP (NextN) head and a vision projector. Verified end to end on
|
|
Dankclaw 2026-09-16 (RTX 3090 24 GiB, Qwen3.8-27B-GSQ-RCO IQ3_S `-mtp` +
|
|
BF16 mmproj, 131072 ctx / 1 slot, port 7771).
|
|
|
|
## 1. Preflight the GGUF WITHOUT downloading it
|
|
|
|
GGUF metadata lives in the header, and HuggingFace honours ranged GETs, so a
|
|
few MB answers every sizing question before a 13 GB transfer:
|
|
|
|
```python
|
|
import urllib.request
|
|
req = urllib.request.Request(url, headers={"Range": "bytes=0-8000000", "User-Agent": "curl/8"})
|
|
buf = urllib.request.urlopen(req, timeout=180).read()
|
|
```
|
|
|
|
Parse: magic `GGUF`, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs
|
|
(string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte —
|
|
the type hand-written parsers forget). Walk arrays by byte size even when you
|
|
only want a summary: a naive parser desynchronises on `tokenizer.ggml.tokens`
|
|
and lands gigabytes off.
|
|
|
|
Read these keys: `general.architecture`, `<arch>.block_count`,
|
|
`<arch>.context_length`, `<arch>.nextn_predict_layers`, `<arch>.head_count_kv`,
|
|
`<arch>.key_length` / `value_length`, `<arch>.full_attention_interval`,
|
|
`<arch>.ssm.state_size` / `group_count` / `time_step_rank`, and for the
|
|
projector file `clip.projector_type` (e.g. `qwen3vl_merger`) plus
|
|
`clip.has_vision_encoder`.
|
|
|
|
## 2. Decide the file from `nextn_predict_layers`
|
|
|
|
**MTP is a property of the FILE, not of the flags.** If the release ships an
|
|
`-mtp` build, that build carries the head as `blk.<n>.nextn.*` (a few hundred
|
|
MB larger); the plain build has no MTP tensors at all.
|
|
|
|
With the `-mtp` file, llama.cpp needs only `--spec-type draft-mtp`: it sets
|
|
`load_mtp` and the draft context runs on the MAIN model's weights
|
|
(`shares_model`, `LLAMA_CONTEXT_TYPE_MTP`). Do NOT add `--model-draft` — that
|
|
loads the whole file a second time (12 GiB twice on a 24 GiB card). A separate
|
|
`-md` is only correct for standalone MTP-only sidecars.
|
|
|
|
Confirm the flag exists in the deployed build before writing the script:
|
|
`grep -rn "common_speculative_all_types_str" common/speculative.cpp` and
|
|
`llama-server --help | grep -A4 spec-type`.
|
|
|
|
## 3. KV arithmetic (do this before choosing ctx)
|
|
|
|
Only **full-attention** layers take KV. For a hybrid arch (Qwen3-Next/3.8 style
|
|
`16 x (3 x Gated DeltaNet + 1 x Gated Attention)`):
|
|
|
|
```
|
|
attn_layers = block_count / full_attention_interval
|
|
bytes/token = attn_layers x n_head_kv x head_dim x 2 (K,V) x bytes_per_elem
|
|
f16 = 2 B | q8_0 = 1.0625 B | q4_0 = 0.5625 B
|
|
```
|
|
|
|
The linear-attention layers hold constant-size per-sequence state instead
|
|
(~0.16 GiB per sequence at 48 layers / 128 d_state / 48 v-heads in f32), and
|
|
the MTP draft context costs its own `n_ctx x (1 layer x n_head_kv x head_dim x
|
|
2 x 2 B)` ≈ 4 KiB/token for a 1-layer head.
|
|
|
|
VRAM budget: weights + mmproj + MTP draft KV + DeltaNet state + ~1.5 GiB of
|
|
CUDA context and compute buffers. On a 24 GiB card that leaves ~9.5 GiB for KV,
|
|
which at 64 KiB/token is ~155k tokens total.
|
|
|
|
**`--parallel N` DIVIDES `--ctx-size` between slots** — it does not multiply.
|
|
`-np 1 -c 131072` is one 131k conversation; `-np 4 -c 131072` is four 32k ones.
|
|
|
|
## 4. Flags that matter
|
|
|
|
- `-ngl 999`, `-fa on`, `-ctk/-ctv f16` for the usual shape.
|
|
- Quantized KV (`q8_0`, `q4_0`) requires a build with
|
|
`-DGGML_CUDA_FA_ALL_QUANTS=ON`; without it CUDA prefill collapses ~12x. f16
|
|
needs nothing, so prefer it unless the context demand is real.
|
|
- `--jinja` — pin it explicitly even when the build defaults it on: the GGUF's
|
|
own template supplies `reasoning_effort`, `preserve_thinking` and the
|
|
tool-call grammar, and a future default flip must not silently remove tools.
|
|
- `--metrics` if anything scrapes `/metrics`, `--alias` to set the served id,
|
|
`--temp 0.0` for a greedy-by-default fleet contract,
|
|
`--reasoning-format deepseek --reasoning-budget -1` to project thinking into
|
|
`reasoning_content`.
|
|
|
|
## 5. Serve-script and unit invariants
|
|
|
|
Fail loudly before loading VRAM: check the binary, the model file and the
|
|
mmproj exist, and `exit 1` with a named message otherwise — a missing weight
|
|
must not become a `Restart=on-failure` crash loop that reads like a runtime
|
|
fault. For a native CUDA binary the unit is a ROOT systemd unit
|
|
(`[Install] WantedBy=multi-user.target`, `Restart=on-failure`,
|
|
`RestartSec=10`, `LimitMEMLOCK=infinity`); a container would be a Quadlet.
|
|
|
|
## 6. Verification battery (prove each claim, do not infer)
|
|
|
|
1. `/health` 200, and `/v1/models` reports the intended alias.
|
|
2. A chat completion returns `content` AND `reasoning_content`.
|
|
3. A request carrying `tools` returns `tool_calls` — a RUNTIME property: the
|
|
ik_llama.cpp runtime answered `500 tools param requires --jinja flag`.
|
|
4. `GET /metrics` after a generation: `llamacpp:spec_decode_num_draft_tokens_total`
|
|
and `..._accepted_tokens_total` both > 0, acceptance 0.6-0.95 ⇒ MTP engaged,
|
|
not merely configured. The slot timing lines in the journal say the same.
|
|
5. One image request (`data:` URL or bare base64) ⇒ an answer that reflects the
|
|
image; confirm the journal printed a multimodal projector load.
|
|
6. `nvidia-smi` memory used is close to the modelled total, and `n_slots` /
|
|
`n_ctx_slot` in the load log match the intended shape.
|
|
|
|
## Traps that cost real time
|
|
|
|
- **A template can reject the client's thinking level.** Qwen3.8 accepts only
|
|
`xhigh|medium|low` for `reasoning_effort` and raises otherwise — HTTP 500
|
|
naming Jinja, not the model. Map every harness level onto the accepted set
|
|
(e.g. `compat.reasoningEffortMap: {minimal: low, low: low, medium: medium,
|
|
high: xhigh, max: xhigh}`) and probe each level with curl before routing
|
|
roles at the endpoint.
|
|
- **Thinking counts against `max_tokens`.** A budget that runs out mid-thought
|
|
returns `finish_reason: "length"` with empty `content` — not a broken
|
|
endpoint, not a vision failure. Raise the budget.
|
|
- **Never point `--model-draft` at the `-mtp` file** (step 2).
|
|
- **Any `-ot` / `-ncmoe`-style override can silently disable mmap** for the
|
|
whole load, turning a mapped model into anonymous memory.
|
|
- On a host that publishes ports through **pasta**, the replacement must not
|
|
start until the predecessor released the port: poll
|
|
`ss -tan "sport = :<port>"` (listener, accepted AND TIME_WAIT) or the
|
|
container dies with `Address already in use`.
|
|
- On a shared host, read the serving engine's own boot ledger for device
|
|
budget vs need: a pool that silently clamped still answers a green `/health`.
|