---
type: reference
title: Tcclaviger Fork Ops
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
timestamp: 2026-10-05T00:00:00-07:00
---

# Tcclaviger Fork Ops

The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
first-class expert-offload planner) served `frank-archon` at shape G until
the 2026-10-05 FreeToken cutover; it remains the rollback target
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
[[LLM Bench Windows and Measurement]].

## The ad-hoc sweep harness (dankserver)

Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
into every fresh container — `podman cp` files die on container recreate.

- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
  `mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
  ttft)`; chunk counting is INVALID under spec decode.
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
  `out NAME` ([[Remote Job Runner]]).
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
  (else repetition-halt or early EOS), finish_reason, and inspect the text —
  `bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
  loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
  Acceptance swings 48–61% run-to-run — noise; decode is the metric.
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
  `Application startup complete` — it is on STDERR; an unmerged grep can
  never match and burns the whole wait timeout.
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
  are CPU-side); VRAM climbing is the load indicator.
- Config edit discipline: the serve script is patched by scp'd python
  scripts with asserts (anchor must exist, addition must not); `bash -n`
  after every patch.

## Measured knob verdicts (2026-09-29 campaign)

Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.

| Knob | Verdict | Evidence |
|---|---|---|
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |

Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
batch-shaped compute knobs move ~nothing.
Measured with no spec: **12.11 tok/s FLAT from 304 to 16,095 context tokens** — KV
attention is not the cost, so a context-slope hunt on this stack is null by construction.

The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
depth.

Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
temporal reuse dominates, so never infer miss rates from resident-slot
fractions.

## Sweep hygiene (fork-specific preflight)

1. **Knob landed:** for each `-e` env knob, confirm in-container
   `podman exec … env` (or the engine log's confirmed-values line). A knob
   the fork silently ignores produces a "parity" arm that is really the
   default — the v1 gather sweep lost all three arms to exactly this
   (GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
   expert-prefetch setting. The 2026-09-29 campaign measured losing arms
   with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
   (no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
   verdict.
3. **Spec arms must prove engagement:** engine log
   `SpecDecoding metrics: Mean acceptance length …` and
   `curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
   refused boot dies in one poll instead of 20 min of health waits. Capture
   >8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
   refusal) lives higher up.
5. Report per-depth tables, never one number: g32×64's win grows with depth
   (54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
   rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
   rows across arms only.
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
   noise).

Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
measure-stage bug costs decode tables but not boot-log findings.

**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
tokens). Acceptance is real: mean length 2.4–2.8, per-position
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.

## Serving a model larger than VRAM (dankserver recipe)

Dankserver's card count, per-card VRAM and host RAM are in [[Dankserver System Specs]];
`torch.cuda.device_count()` returns 3 on this box and summing `total_memory` overstates it
badly (the Raphael iGPU is in the list — reading discipline: [[Llama Cpp Serving Sizing]]).

Everything else about running this image — the quant-kernel gate that decides
which checkpoints can serve at all, the entrypoint contract, the known-good
invocation, the `/dev/dri` and memlock traps, the util and max-model-len arena
traps, the `--ipc=host` / `-d` stdout traps and the gated-fetch discipline — is
written once on [[RDNA4 Container Serving]], and the plan arithmetic behind the
arena/KV traps is [[RDNA4 MoE Expert Offload VRAM Budget]]'s. Three findings
stay here because this harness produced them:

- **The KV reserve undercounts hybrid-attention models ~2×** (the planner's
  B/token figure against ~18,000 real: the Mamba state is a fixed per-sequence
  cost). Fix by pinning the pool: `--kv-cache-memory-bytes`.
- **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
  structurally orthogonal to `--expert-precise-memory` (verified in-container
  2026-10-05: precise-memory only swaps the GPU overhead term,
  expert_offload_mem only sets the host bound; they rejoin only inside
  `place_experts`, and the plan rewrites `expert_offload_mem` back to the cap
  the plan met, so plan and allocation cannot drift).
- **Check whether the stack supports the model natively before porting an
  overlay.** The image already registered `Glm5NextForConditionalGeneration`,
  `Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
  through two image tags to bring an overlay up was entirely wasted work.

The checkpoint this recipe was verified on is
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131 shards,
`Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10, multimodal,
native 262,144) — an [[frank-archon]] era payload.

## Compile-cache persistence (survives `podman rm -f`)

Serve scripts that recreate the container discard the writable layer, so
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
cold, of which 173 s was one graph).

1. **Prove it and locate it:** fresh container vs live
   (`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
   `podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
   f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
   `/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
   `/root/.cache/comgr`.
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
   both trees, safe whenever the image ships no such directory (step 1
   proves it; otherwise you would be hiding baked state). Wire as an opt-in
   tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
   `${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
   checkout.
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
   N s`, warm `Directly load the compiled graph(s) for compile range (…)
   from the cache` with `torch.compile took` in single-digit seconds.
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
   ONE recompile, not a permanent tax; do not claim a proportional boot cut
   (time to `/health` is dominated by weight loading: 490 s warm vs 188 s
   cold compile); stale entries are misses, not wrong answers (version +
   config hashed); if the serve script is seam-gated, assert the mkdir and
   the exact mount target.

## Plan-line verification checklist (after any fork deploy)

The journal one-liner that reads the plan lines (digit-anchored `grep -oE`,
because plain grep truncates those columns to `17.`) and the post-deploy
verification order — plan vs captured graph memory, `GPU KV cache size` vs the
ask, per-card `rocm-smi` high-water, `NRestarts` = 0, `VLLM_CUDAGRAPH_*` present
in the container env, then a real completion from a PEER because
`curl localhost:7770/health` inside an ssh session on the host can return `000`
and look like an outage — are step 3 of
[[RDNA4 MoE Expert Offload VRAM Budget]] §7. Run that; do not re-derive it
here.

## Related

- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
- [[Model Swap and Rollback]] — where the fork sits in the swap history
- [[Remote Job Runner]] — the job.sh harness referenced above