36 lines
3.7 KiB
Markdown
36 lines
3.7 KiB
Markdown
---
|
|
name: model-runtime-support-survey
|
|
description: "Survey how every open-source runtime serves a given LLM (vLLM, llama.cpp, SGLang, KTransformers, overlays, vendor GGUFs) before writing a serving/feasibility plan; harvest quant recipes, measured rates, and mapped landmines from PR comments and overlay repos."
|
|
---
|
|
|
|
# Model Runtime Support Survey
|
|
|
|
Answer "how do other engines run model X, and what did they pay to learn it?" before committing a bring-up plan. Every kernel spec, quant recipe, and landmine found upstream is one you don't pay for twice. Proven on GLM-5.3-Flash (2026-10-14): surfaced a production 8xR9700 vLLM overlay, MTP 2.3x decode numbers, and 4 landmines (fp8-KV scale calibration, prefix-cache 0-hits, KDA autotune nondeterminism, chat-template notation).
|
|
|
|
## Harvest order
|
|
|
|
1. **Official model card** (`huggingface.co/<org>/<model>`) — the "Serve Locally" section lists officially supported runtimes; sampling/template contract lives here (e.g. `reasoning_effort` defaults, `clear_thinking`).
|
|
2. **Per-runtime PRs** (vLLM, llama.cpp, SGLang) — fetch via the GitHub-API-backed read; **the comments, not the body, are the payload**: field reports (prefix-cache hit censuses, ROCm/AMD enablement branches, extension rebuild recipes, crash repros, free tuning levers via `--override-kv`, nondeterminism root-causes). Get the PR status (merged/open/closed) — it sets what "master supports" means.
|
|
3. **Overlay/serving repos on HF** (search model name + your GPU family, e.g. `rocm-r9700`): read the tree (`huggingface.co/api/models/<repo>/tree/main`), then `raw/main/README.md`, `raw/main/OVERLAY_NOTES.md`, `raw/main/run-*.sh`. Launch scripts carry the laws in comments (KV sizing justifications, env ladders, dead knobs).
|
|
4. **Vendor GGUF guides** (Unsloth docs): append `.md` to the docs URL for clean markdown; gives quant KLD/top-1 tables, template fixes, hardware floors. Check whether their GGUF carries MTP/nextn tensors — that's often the only public source of draft-head weights.
|
|
5. **KTransformers tutorials** (`raw.githubusercontent.com/kvcache-ai/ktransformers/main/doc/en/kt-kernel/<Model>-Tutorial.md`) + **SGLang cookbook** (`docs.sglang.io/.../<Model>.md`): CPU-offload expert shape validation, 1M-ctx recipes.
|
|
|
|
## Harvest checklist (per runtime)
|
|
|
|
- Measured decode/prefill + hardware shape (cards, TP) — anchor, not decoration; per-card effective GB/s is the cross-engine comparison unit.
|
|
- Quant recipe: bits-per-weight split by module class (experts/trunk/embed) — convergent recipes cross-validate your planned conversion.
|
|
- Spec-decode: acceptance length, per-position accept, when it pays (CPU-offload) vs loses (VRAM-resident).
|
|
- Landmines: KV-scale calibration reading uninitialized linear-attn state; prefix-cache semantics on hybrid KDA/DSA models; kernel autotune nondeterminism (KLD across launches); fp8 flavour bugs (FNUZ vs e4m3fn scale); template notation engines can't parse; import gates per gfx/arch.
|
|
- Harness contract: reasoning-effort defaults (16k+ hidden thinking tokens), sampling params, tool/reasoning parser names.
|
|
|
|
## Output
|
|
|
|
A runtime matrix table (engine | arch status | hardware | artifact | measured) + a numbered "laws we inherit" list; fold every law into the plan's phases and risks with § cross-refs.
|
|
|
|
## Gotchas
|
|
|
|
- Read PR **comments** — the highest-value data (crash repros, censuses) is comment-only.
|
|
- HF fetches: `api/models/<repo>/tree/main` for listing, `raw/main/<path>` for files; the HTML card view renders README inline too.
|
|
- `browser.open` targets the harness's headless browser, not the user's — refresh user-facing pages with `xdg-open`.
|
|
- Companion skill for the artifact side: `radiance-model-feasibility-scout` (GGUF header census).
|