Frank
04f4e2c6d4
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
203 lines
12 KiB
Markdown
203 lines
12 KiB
Markdown
---
|
||
type: reference
|
||
title: Tcclaviger Fork Ops
|
||
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
|
||
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# Tcclaviger Fork Ops
|
||
|
||
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
|
||
first-class expert-offload planner) served `frank-archon` at shape G until
|
||
the 2026-10-05 FreeToken cutover; it remains the rollback target
|
||
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
|
||
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
|
||
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
|
||
[[LLM Bench Windows and Measurement]].
|
||
|
||
## The ad-hoc sweep harness (dankserver)
|
||
|
||
Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
|
||
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
|
||
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
|
||
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
|
||
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
|
||
into every fresh container — `podman cp` files die on container recreate.
|
||
|
||
- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
|
||
`mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
|
||
ttft)`; chunk counting is INVALID under spec decode.
|
||
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
|
||
`out NAME` ([[Remote Job Runner]]).
|
||
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
|
||
(else repetition-halt or early EOS), finish_reason, and inspect the text —
|
||
`bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
|
||
loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
|
||
Acceptance swings 48–61% run-to-run — noise; decode is the metric.
|
||
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
|
||
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
|
||
`Application startup complete` — it is on STDERR; an unmerged grep can
|
||
never match and burns the whole wait timeout.
|
||
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
|
||
are CPU-side); VRAM climbing is the load indicator.
|
||
- Config edit discipline: the serve script is patched by scp'd python
|
||
scripts with asserts (anchor must exist, addition must not); `bash -n`
|
||
after every patch.
|
||
|
||
## Measured knob verdicts (2026-09-29 campaign)
|
||
|
||
Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
|
||
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
|
||
|
||
| Knob | Verdict | Evidence |
|
||
|---|---|---|
|
||
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
|
||
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
|
||
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
|
||
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
|
||
|
||
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
|
||
batch-shaped compute knobs move ~nothing.
|
||
Measured with no spec: **12.11 tok/s FLAT from 304 to 16,095 context tokens** — KV
|
||
attention is not the cost, so a context-slope hunt on this stack is null by construction.
|
||
|
||
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
|
||
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
|
||
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
|
||
depth.
|
||
|
||
Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
|
||
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
|
||
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
|
||
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
|
||
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
|
||
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
|
||
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
|
||
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
|
||
temporal reuse dominates, so never infer miss rates from resident-slot
|
||
fractions.
|
||
|
||
## Sweep hygiene (fork-specific preflight)
|
||
|
||
1. **Knob landed:** for each `-e` env knob, confirm in-container
|
||
`podman exec … env` (or the engine log's confirmed-values line). A knob
|
||
the fork silently ignores produces a "parity" arm that is really the
|
||
default — the v1 gather sweep lost all three arms to exactly this
|
||
(GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
|
||
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
|
||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms
|
||
with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
|
||
(no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
|
||
verdict.
|
||
3. **Spec arms must prove engagement:** engine log
|
||
`SpecDecoding metrics: Mean acceptance length …` and
|
||
`curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
|
||
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
|
||
refused boot dies in one poll instead of 20 min of health waits. Capture
|
||
>8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
|
||
refusal) lives higher up.
|
||
5. Report per-depth tables, never one number: g32×64's win grows with depth
|
||
(54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
|
||
rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
|
||
rows across arms only.
|
||
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
|
||
noise).
|
||
|
||
Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
|
||
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
|
||
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
|
||
measure-stage bug costs decode tables but not boot-log findings.
|
||
|
||
**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
|
||
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
|
||
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
|
||
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
|
||
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
|
||
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
|
||
tokens). Acceptance is real: mean length 2.4–2.8, per-position
|
||
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
|
||
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
|
||
|
||
## Serving a model larger than VRAM (dankserver recipe)
|
||
|
||
Dankserver's card count, per-card VRAM and host RAM are in [[Dankserver System Specs]];
|
||
`torch.cuda.device_count()` returns 3 on this box and summing `total_memory` overstates it
|
||
badly (the Raphael iGPU is in the list — reading discipline: [[Llama Cpp Serving Sizing]]).
|
||
|
||
Everything else about running this image — the quant-kernel gate that decides
|
||
which checkpoints can serve at all, the entrypoint contract, the known-good
|
||
invocation, the `/dev/dri` and memlock traps, the util and max-model-len arena
|
||
traps, the `--ipc=host` / `-d` stdout traps and the gated-fetch discipline — is
|
||
written once on [[RDNA4 Container Serving]], and the plan arithmetic behind the
|
||
arena/KV traps is [[RDNA4 MoE Expert Offload VRAM Budget]]'s. Three findings
|
||
stay here because this harness produced them:
|
||
|
||
- **The KV reserve undercounts hybrid-attention models ~2×** (the planner's
|
||
B/token figure against ~18,000 real: the Mamba state is a fixed per-sequence
|
||
cost). Fix by pinning the pool: `--kv-cache-memory-bytes`.
|
||
- **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
|
||
structurally orthogonal to `--expert-precise-memory` (verified in-container
|
||
2026-10-05: precise-memory only swaps the GPU overhead term,
|
||
expert_offload_mem only sets the host bound; they rejoin only inside
|
||
`place_experts`, and the plan rewrites `expert_offload_mem` back to the cap
|
||
the plan met, so plan and allocation cannot drift).
|
||
- **Check whether the stack supports the model natively before porting an
|
||
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
|
||
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
|
||
through two image tags to bring an overlay up was entirely wasted work.
|
||
|
||
The checkpoint this recipe was verified on is
|
||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131 shards,
|
||
`Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10, multimodal,
|
||
native 262,144) — an [[frank-archon]] era payload.
|
||
|
||
## Compile-cache persistence (survives `podman rm -f`)
|
||
|
||
Serve scripts that recreate the container discard the writable layer, so
|
||
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
|
||
cold, of which 173 s was one graph).
|
||
|
||
1. **Prove it and locate it:** fresh container vs live
|
||
(`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
|
||
`podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
|
||
f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
|
||
`/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
|
||
`/root/.cache/comgr`.
|
||
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
|
||
both trees, safe whenever the image ships no such directory (step 1
|
||
proves it; otherwise you would be hiding baked state). Wire as an opt-in
|
||
tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
|
||
`${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
|
||
checkout.
|
||
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
|
||
N s`, warm `Directly load the compiled graph(s) for compile range (…)
|
||
from the cache` with `torch.compile took` in single-digit seconds.
|
||
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
|
||
ONE recompile, not a permanent tax; do not claim a proportional boot cut
|
||
(time to `/health` is dominated by weight loading: 490 s warm vs 188 s
|
||
cold compile); stale entries are misses, not wrong answers (version +
|
||
config hashed); if the serve script is seam-gated, assert the mkdir and
|
||
the exact mount target.
|
||
|
||
## Plan-line verification checklist (after any fork deploy)
|
||
|
||
The journal one-liner that reads the plan lines (digit-anchored `grep -oE`,
|
||
because plain grep truncates those columns to `17.`) and the post-deploy
|
||
verification order — plan vs captured graph memory, `GPU KV cache size` vs the
|
||
ask, per-card `rocm-smi` high-water, `NRestarts` = 0, `VLLM_CUDAGRAPH_*` present
|
||
in the container env, then a real completion from a PEER because
|
||
`curl localhost:7770/health` inside an ssh session on the host can return `000`
|
||
and look like an outage — are step 3 of
|
||
[[RDNA4 MoE Expert Offload VRAM Budget]] §7. Run that; do not re-derive it
|
||
here.
|
||
|
||
## Related
|
||
|
||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
|
||
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
|
||
- [[Model Swap and Rollback]] — where the fork sits in the swap history
|
||
- [[Remote Job Runner]] — the job.sh harness referenced above
|