Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
203 lines
12 KiB
Markdown
203 lines
12 KiB
Markdown
---
|
||
type: reference
|
||
title: Tcclaviger Fork Ops
|
||
description: "Operating the RDNA4 vLLM fork: the sweep harness, measured knob verdicts, sweep hygiene, compile-cache persistence."
|
||
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# Tcclaviger Fork Ops
|
||
|
||
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
|
||
first-class expert-offload planner) served `frank-archon` at shape G until
|
||
the 2026-10-05 FreeToken cutover; it remains the rollback target
|
||
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
|
||
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
|
||
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
|
||
[[LLM Bench Windows and Measurement]].
|
||
|
||
## The ad-hoc sweep harness (dankserver)
|
||
|
||
Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
|
||
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
|
||
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
|
||
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
|
||
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
|
||
into every fresh container — `podman cp` files die on container recreate.
|
||
|
||
- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
|
||
`mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
|
||
ttft)`; chunk counting is INVALID under spec decode.
|
||
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
|
||
`out NAME` ([[Remote Job Runner]]).
|
||
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
|
||
(else repetition-halt or early EOS), finish_reason, and inspect the text —
|
||
`bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
|
||
loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
|
||
Acceptance swings 48–61% run-to-run — noise; decode is the metric.
|
||
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
|
||
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
|
||
`Application startup complete` — it is on STDERR; an unmerged grep can
|
||
never match and burns the whole wait timeout.
|
||
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
|
||
are CPU-side); VRAM climbing is the load indicator.
|
||
- Config edit discipline: the serve script is patched by scp'd python
|
||
scripts with asserts (anchor must exist, addition must not); `bash -n`
|
||
after every patch.
|
||
|
||
## Measured knob verdicts (2026-09-29 campaign)
|
||
|
||
Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
|
||
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
|
||
|
||
| Knob | Verdict | Evidence |
|
||
|---|---|---|
|
||
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
|
||
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
|
||
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
|
||
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
|
||
|
||
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
|
||
batch-shaped compute knobs move ~nothing.
|
||
Measured with no spec: **12.11 tok/s FLAT from 304 to 16,095 context tokens** — KV
|
||
attention is not the cost, so a context-slope hunt on this stack is null by construction.
|
||
|
||
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
|
||
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
|
||
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
|
||
depth.
|
||
|
||
Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
|
||
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
|
||
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
|
||
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
|
||
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
|
||
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
|
||
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
|
||
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
|
||
temporal reuse dominates, so never infer miss rates from resident-slot
|
||
fractions.
|
||
|
||
## Sweep hygiene (fork-specific preflight)
|
||
|
||
1. **Knob landed:** for each `-e` env knob, confirm in-container
|
||
`podman exec … env` (or the engine log's confirmed-values line). A knob
|
||
the fork silently ignores produces a "parity" arm that is really the
|
||
default — the v1 gather sweep lost all three arms to exactly this
|
||
(GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
|
||
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
|
||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms
|
||
with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
|
||
(no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
|
||
verdict.
|
||
3. **Spec arms must prove engagement:** engine log
|
||
`SpecDecoding metrics: Mean acceptance length …` and
|
||
`curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
|
||
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
|
||
refused boot dies in one poll instead of 20 min of health waits. Capture
|
||
>8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
|
||
refusal) lives higher up.
|
||
5. Report per-depth tables, never one number: g32×64's win grows with depth
|
||
(54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
|
||
rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
|
||
rows across arms only.
|
||
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
|
||
noise).
|
||
|
||
Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
|
||
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
|
||
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
|
||
measure-stage bug costs decode tables but not boot-log findings.
|
||
|
||
**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
|
||
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
|
||
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
|
||
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
|
||
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
|
||
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
|
||
tokens). Acceptance is real: mean length 2.4–2.8, per-position
|
||
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
|
||
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
|
||
|
||
## Serving a model larger than VRAM (dankserver recipe)
|
||
|
||
Dankserver's card count, per-card VRAM and host RAM are in [[Dankserver System Specs]];
|
||
`torch.cuda.device_count()` returns 3 on this box and summing `total_memory` overstates it
|
||
badly (the Raphael iGPU is in the list — reading discipline: [[Llama Cpp Serving Sizing]]).
|
||
|
||
Everything else about running this image — the quant-kernel gate that decides
|
||
which checkpoints can serve at all, the entrypoint contract, the known-good
|
||
invocation, the `/dev/dri` and memlock traps, the util and max-model-len arena
|
||
traps, the `--ipc=host` / `-d` stdout traps and the gated-fetch discipline — is
|
||
written once on [[RDNA4 Container Serving]], and the plan arithmetic behind the
|
||
arena/KV traps is [[RDNA4 MoE Expert Offload VRAM Budget]]'s. Three findings
|
||
stay here because this harness produced them:
|
||
|
||
- **The KV reserve undercounts hybrid-attention models ~2×** (the planner's
|
||
B/token figure against ~18,000 real: the Mamba state is a fixed per-sequence
|
||
cost). Fix by pinning the pool: `--kv-cache-memory-bytes`.
|
||
- **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
|
||
structurally orthogonal to `--expert-precise-memory` (verified in-container
|
||
2026-10-05: precise-memory only swaps the GPU overhead term,
|
||
expert_offload_mem only sets the host bound; they rejoin only inside
|
||
`place_experts`, and the plan rewrites `expert_offload_mem` back to the cap
|
||
the plan met, so plan and allocation cannot drift).
|
||
- **Check whether the stack supports the model natively before porting an
|
||
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
|
||
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
|
||
through two image tags to bring an overlay up was entirely wasted work.
|
||
|
||
The checkpoint this recipe was verified on is
|
||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131 shards,
|
||
`Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10, multimodal,
|
||
native 262,144) — an [[frank-archon]] era payload.
|
||
|
||
## Compile-cache persistence (survives `podman rm -f`)
|
||
|
||
Serve scripts that recreate the container discard the writable layer, so
|
||
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
|
||
cold, of which 173 s was one graph).
|
||
|
||
1. **Prove it and locate it:** fresh container vs live
|
||
(`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
|
||
`podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
|
||
f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
|
||
`/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
|
||
`/root/.cache/comgr`.
|
||
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
|
||
both trees, safe whenever the image ships no such directory (step 1
|
||
proves it; otherwise you would be hiding baked state). Wire as an opt-in
|
||
tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
|
||
`${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
|
||
checkout.
|
||
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
|
||
N s`, warm `Directly load the compiled graph(s) for compile range (…)
|
||
from the cache` with `torch.compile took` in single-digit seconds.
|
||
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
|
||
ONE recompile, not a permanent tax; do not claim a proportional boot cut
|
||
(time to `/health` is dominated by weight loading: 490 s warm vs 188 s
|
||
cold compile); stale entries are misses, not wrong answers (version +
|
||
config hashed); if the serve script is seam-gated, assert the mkdir and
|
||
the exact mount target.
|
||
|
||
## Plan-line verification checklist (after any fork deploy)
|
||
|
||
The journal one-liner that reads the plan lines (digit-anchored `grep -oE`,
|
||
because plain grep truncates those columns to `17.`) and the post-deploy
|
||
verification order — plan vs captured graph memory, `GPU KV cache size` vs the
|
||
ask, per-card `rocm-smi` high-water, `NRestarts` = 0, `VLLM_CUDAGRAPH_*` present
|
||
in the container env, then a real completion from a PEER because
|
||
`curl localhost:7770/health` inside an ssh session on the host can return `000`
|
||
and look like an outage — are step 3 of
|
||
[[RDNA4 MoE Expert Offload VRAM Budget]] §7. Run that; do not re-derive it
|
||
here.
|
||
|
||
## Related
|
||
|
||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
|
||
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
|
||
- [[Model Swap and Rollback]] — where the fork sits in the swap history
|
||
- [[Remote Job Runner]] — the job.sh harness referenced above
|