frank-primary: qwen cutover docs + deployer identity + seam re-pin
- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4 (37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index. - runbook: RETIRED verdict — DFlash2 drafter served and drafted but acceptance pinned 1.00-1.38 on every text class including verbatim repetition (mis-wired aux capture, not a weak drafter); 5x slower than baseline at ~1.2 acceptance. - log.md: 2026-10-01 rollback entry. - deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen; glm-probe joins the legacy sweep list. - frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool dialect, automatic offload. The old 'MTP buys zero' law carried the prefetch tax and is retired.
This commit is contained in:
Frank
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill, bounded by unfused pointwise glue (the expert gather is 1.05 % of device time; the trunk is uncompiled) | 7770 |
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
|
||||
@@ -423,115 +423,39 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-primary — tcclaviger vLLM, GLM-5.3-Flash W4A16 int4 (Dankserver, ACTIVE)
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
|
||||
|
||||
The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`
|
||||
— the same digest-pinned image the 2026-09-29 qwen tuning campaign measured)
|
||||
serving **`Intel/GLM-5.3-Flash-W4A16-AutoRound`**: AutoRound **symmetric int4,
|
||||
group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors,
|
||||
113,074 tensors, ~170 GiB — on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
|
||||
**131,072 ctx × 2 seqs**, bf16 KV, util 0.98, NBATCH 2048, prefix caching,
|
||||
**no MTP**, **no prefetch**. Cut over 2026-09-30 from the qwen FP8 + MTP4 shape,
|
||||
which itself replaced the R9V radiance fork on 2026-09-29.
|
||||
The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||||
digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
|
||||
(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
|
||||
**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
|
||||
**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
|
||||
GLM/DFlash2 experiment was declared failed (same day).
|
||||
|
||||
**The payload is chosen for the offload path, not for the quant.** 170 GiB of
|
||||
int4 experts cannot fit 64 GiB of VRAM, so expert offload is mandatory — and
|
||||
the fork reaches its W4A16 kernel (`R4dW4a16MoEMethod`, libr4d's grouped W4A16
|
||||
GEMM, 6.234 MiB per expert-rank slot) **only** when the checkpoint's
|
||||
`quant_method` is `auto-round`, which the fork overrides to `inc`. A
|
||||
compressed-tensors W4A16 checkpoint of the same model resolves to
|
||||
`TritonWNA16Experts`, which **has no offload recipe at all**, and refuses to
|
||||
serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over
|
||||
12,096 host-backed experts = **73.64 GiB pinned host rows per rank**.
|
||||
Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
|
||||
~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
|
||||
35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
|
||||
BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
|
||||
throughput one. V-only q8 KV is NOT possible via config: stock
|
||||
`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
|
||||
|
||||
**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k
|
||||
prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met. The first
|
||||
explanation for that — "the expert gather is the bound at ~8.2 GB/s" — was
|
||||
FALSIFIED by a torch-profiler kernel-time window** (496 decode tokens, rank0
|
||||
trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms per
|
||||
token, of which **44 % is pointwise glue** — `vectorized_elementwise_kernel`
|
||||
alone is 16.07 s over **708,500 launches (1,428 per token)**, plus 3.11 s of
|
||||
`elementwise_kernel_manual_unroll` — against **1.05 % for
|
||||
`r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K GEMM 6 %, the
|
||||
router/top-k 9 %, the mHC kernels 6.5 %. The real wall is that **the fork
|
||||
compiles its own Qwen trunk (`@support_torch_compile` in
|
||||
`models/qwen4_exp/amd/model.py`) and not its GLM trunk**, so every pointwise op
|
||||
is a separate eager kernel. Two consequences: spec decode **cannot** help this
|
||||
shape (pointwise cost scales with the token count, which is exactly why MTP4
|
||||
measured flat), and neither arena size nor hit rate is a lever (the gather is
|
||||
1 %).
|
||||
The engine auto-sets the attention block size (mamba-page floor); the
|
||||
expert plan's registry prices KV per token; the expert arena (6,574 slots
|
||||
x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
|
||||
for KV headroom collapses the offload hit rate.
|
||||
|
||||
**Fork work is staged, not deployed.** Two WIP changelists, both apply-verified
|
||||
against the pinned image (`patch -p1` reproduces each artifact's recorded sha256):
|
||||
`scripts/patches/glm53-compile-0{1..5}*.patch` (decorate the GLM trunk; freeze the ten
|
||||
libr4d capability probes that stat/`.so`-probe *inside* the traced forward; make
|
||||
`fullgraph=True` opt-out; take GLM off the fused-norm JIT kernel; make the indexer's
|
||||
in-forward workspace acquisition compiler-opaque) and
|
||||
`glm53-dflash-01-aux-hidden-states.patch` (the EAGLE3 interface, below). Boots
|
||||
established that `functools.lru_cache` does **not** protect a probe (dynamo re-traces
|
||||
the wrapped body even warm), that `suppress_errors` does **not** cover graph breaks
|
||||
(only `fullgraph=False` does), and that the GLM trunk can never satisfy `fullgraph`
|
||||
(its indexer metadata syncs to host, `glm5next/amd/sparse_indexer.py:263,419`). With
|
||||
breaks genuinely tolerated the trunk **traces** (`Dynamo bytecode transform time:
|
||||
1.08 s`) and the single remaining blocker is
|
||||
`KeyError: 'in_proj_qkvbfg_a'` — raised in a callee dynamo inlines into
|
||||
`Glm5NextDecoderLayer.forward`, **not** at any of dynamo's four `nn_modules` lookups and
|
||||
**not** a literal string lookup, so it is currently a dynamo-side name-resolution
|
||||
failure rather than a demonstrated fork bug (an earlier framing here said "a latent bug
|
||||
the compile exposes"; that was retracted). Watch the launcher/reset gotchas: the scratch
|
||||
launcher once silently dropped `VLLM_COMPILE_FULLGRAPH=0` (making four boots run
|
||||
`fullgraph=True` where every break is fatal), and a `sitecustomize.py` on `PYTHONPATH`
|
||||
became a fork bomb — see the runbook.
|
||||
Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
|
||||
`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
|
||||
frank-primary-glm`. The serve script drives `podman run` directly, gates
|
||||
the digest-pinned image, the payload's `config.json` and its exact
|
||||
131-shard count and the GPU device nodes, clears stale container
|
||||
identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
|
||||
same, waits out :7770, relinks/enables/restarts the unit and health-gates
|
||||
with the MainPID check. Boot cost ~7 min warm; health returns 503 while
|
||||
loading.
|
||||
|
||||
**No new HIP kernels are possible** — the image ships `r4d.so` as a binary with
|
||||
no sources. The 9 % router is therefore **binary-blocked**: its fused path is
|
||||
gated `num_experts in {128, 256, 512}` (`router/r4d_route_router.py`) and only
|
||||
`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has **288**
|
||||
experts. Only Python/config/inductor changes (or runtime-JIT Triton) are
|
||||
available on this fork.
|
||||
|
||||
**DFlash2 for GLM is a target-side port, not a drafter port.**
|
||||
`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
|
||||
and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
|
||||
`target_layer_ids [5, 14, 24, 33, 42]` of 45), and the fork **already maps it**
|
||||
(`registry.py:636` → `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
|
||||
`"dflash"`). What was missing is the target side — `glm5next` implemented neither
|
||||
`set_aux_hidden_state_layers()` nor the per-layer aux capture that
|
||||
`deepseek_v2.py` (also MLA+MoE) demonstrates — and that is now written:
|
||||
`scripts/patches/glm53-dflash-01-aux-hidden-states.patch` implements exactly that
|
||||
contract (`EagleModelMixin` on the trunk, capture of the single-stream state entering
|
||||
each requested layer, the conditional `(hidden_states, aux_hidden_states)` tuple
|
||||
return, and the two delegating methods on `Glm5NextForCausalLM`). The engine's
|
||||
refusal was literally `RuntimeError: Model does not support EAGLE3 interface`, so
|
||||
this patch is the prerequisite for driving the drafter at all. Budget no speedup:
|
||||
per the profile above, a verification pass pays pointwise cost proportional to its
|
||||
token count.
|
||||
|
||||
**MTP works here and is still rejected.** The head loads
|
||||
(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted
|
||||
length 3.76–4.90, per-position 0.92/0.85/0.69/0.58, 69–97 %), but decode was
|
||||
unchanged (9.6–11.2 vs 10.0 tok/s) while prefill **collapsed 918 → 255 tok/s**:
|
||||
a verification forward gathers the *union* of the batch's experts, so spec
|
||||
decode amortizes compute, not a residency-bound gather. It also costs 2.6 GiB
|
||||
of arena. **fp8 KV is illegal on this model** — it disables the sparse MLA
|
||||
backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2,
|
||||
so the MTP draft step demands an op that cannot exist on gfx12
|
||||
(`no attribute 'rocm_aiter_mla_decode_fwd'`).
|
||||
|
||||
**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
|
||||
`User=dank`, `LimitMEMLOCK=infinity` — the 73.64 GiB/rank expert pin needs it),
|
||||
`ExecStop=podman stop -t 120 frank-primary-glm`. The serve script drives
|
||||
`podman run` **directly** (the image entrypoint execs `vllm serve`), gates the
|
||||
digest-pinned image, the payload's `config.json` and its exact 34-shard count
|
||||
and the GPU device nodes, clears the stale container identities, then **blocks
|
||||
on `podman wait`** so a crash is a nonzero ExecStart. The deployer's
|
||||
`deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts
|
||||
the unit and health-gates with the MainPID check (the false-green stray law).
|
||||
Boot cost ~7 min warm (~15 min cold); health returns 503 while loading.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
Rollback: commit `2339fae` (qwen FP8 + MTP4) — restore `scripts/` from it and
|
||||
deploy; the qwen payload and the R9V state dirs stay on disk.
|
||||
Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
|
||||
## Key Lessons Learned
|
||||
Durable operational lessons from the fleet's history. Full detail in the
|
||||
@@ -880,7 +804,7 @@ Active:
|
||||
|---------|---------|
|
||||
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-primary (tcclaviger vLLM, GLM-5.3-Flash W4A16 int4, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -888,6 +812,7 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
|
||||
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
|
||||
|
||||
@@ -1217,8 +1217,8 @@ deploy_llm_small() {
|
||||
# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
|
||||
# sha256 verification was the fetch script's job at download time, never
|
||||
# re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-glm
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-qwen
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe"
|
||||
|
||||
deploy_llm_primary() {
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
@@ -1270,9 +1270,9 @@ deploy_llm_primary() {
|
||||
# A stopped unit can leave its container running: systemd's cgroup kill
|
||||
# races conmon's --rm cleanup, and the orphan holds the port and GPUs.
|
||||
# Stop the unit, then remove the containers BEFORE waiting on the port.
|
||||
# The legacy identities (the retired R9V container and the ad hoc
|
||||
# experiment container qwen-fp8, the ad hoc glm-probe) hold the same GPUs
|
||||
# and die here too.
|
||||
# The legacy identities (the retired R9V container, the ad hoc
|
||||
# experiment container qwen-fp8, the retired glm-probe) hold the same
|
||||
# GPUs and die here too.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
local ctr
|
||||
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
|
||||
|
||||
@@ -18,6 +18,22 @@ timestamp: 2026-09-30
|
||||
|
||||
# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
||||
|
||||
> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
|
||||
> **failure**: after five patches the drafter served and drafted (12.8 tok/s
|
||||
> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
|
||||
> **acceptance pinned at 1.00-1.38 on every text class including verbatim
|
||||
> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
|
||||
> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
|
||||
> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
|
||||
> untested alternative capture semantics: post-layer output state
|
||||
> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
|
||||
> (170 GiB) + drafter (2.2 GiB) staged to
|
||||
> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
|
||||
> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
|
||||
> generic unifier/tailpin/grouping fixes, apply-verified against the image.
|
||||
|
||||
|
||||
> **Question this page answers.** Can the fleet serve the 320B-class
|
||||
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
|
||||
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
|
||||
|
||||
@@ -1,5 +1,24 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-01
|
||||
- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
|
||||
GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
|
||||
served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
|
||||
captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
|
||||
interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
|
||||
draft+hidden groups, registry method-signal), but **acceptance pinned at
|
||||
1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
|
||||
aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
|
||||
8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
|
||||
(5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
|
||||
not originate from this engine (APIServer access log: every completion
|
||||
POST was host-local; his workstation appears once, a /v1/models GET).
|
||||
Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
|
||||
(200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
|
||||
(2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
|
||||
KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
|
||||
unifier/tailpin/grouping/registry fixes, apply-verified).
|
||||
|
||||
## 2026-09-30
|
||||
- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
|
||||
the 35 tok/s target, and the shortfall is measured rather than inferred.**
|
||||
|
||||
@@ -1,22 +1,26 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
|
||||
#
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-30
|
||||
# cutover to GLM-5.3-Flash W4A16 int4, which replaced the qwen FP8 + MTP4
|
||||
# shape): the serve script drives `podman run` DIRECTLY (the image entrypoint
|
||||
# execs `vllm serve`), with every engine flag qualified by boot probe on
|
||||
# 2026-09-30 and carried in the repo env file. The seam pins what the flags
|
||||
# alone do not say: the qualification law (131,072 x 2 seqs at bf16 KV, util
|
||||
# 0.98 is what boots; the window is forced up by the fork's KV reserve model,
|
||||
# not by capacity), the offload law (170 GiB of experts cannot fit 64 GiB of
|
||||
# VRAM — the engine MUST run with --enable-expert-offload), the self-healing
|
||||
# law (the run returns before the engine is up, so the serve script must wait
|
||||
# on the container for Restart=on-failure to mean anything), the memlock law
|
||||
# that makes the 73.64 GiB/rank expert pin legal, the deployer's image/payload
|
||||
# gates with the false-green MainPID check, and the three measured-absent
|
||||
# flags: expert prefetch (36% precision, costs ~40-50% decode), MTP spec
|
||||
# decode (buys zero decode and collapses prefill on an offload-bound shape),
|
||||
# and fp8 KV (disables the only MLA decode path gfx12 has).
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
|
||||
# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which
|
||||
# reversed the failed GLM/DFlash2 experiment): the serve script drives
|
||||
# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
|
||||
# engine flag qualified by boot probe and carried in the repo env file. The
|
||||
# seam pins what the flags alone do not say: the KV qualification law
|
||||
# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB
|
||||
# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV
|
||||
# qualification (e4m3 for BOTH K and V — the stock flag quantizes them
|
||||
# together), the automatic-offload law (the fork plans the tiered-expert
|
||||
# offload itself; no flag exists to set), the self-healing law (the run
|
||||
# returns before the engine is up, so the serve script must wait on the
|
||||
# container for Restart=on-failure to mean anything), the memlock law that
|
||||
# makes the pinned host rows legal, the deployer's image/payload gates with
|
||||
# the false-green MainPID check, and the measured laws: expert prefetch
|
||||
# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved
|
||||
# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance
|
||||
# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading
|
||||
# carried the prefetch tax), and NO tool dialect (the GLM era added
|
||||
# --enable-auto-tool-choice; the 2339fae trio does not serve tools).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -56,8 +60,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# outlives the unit and holds both GPUs. The container name must match
|
||||
# the env file's identity.
|
||||
grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
|
||||
grep -q 'frank-primary-glm' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-glm$' "$ENV_FILE"
|
||||
grep -q 'frank-primary-qwen' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve shape is direct podman run, not a launcher or shim" {
|
||||
@@ -72,77 +76,75 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-09-30 boot probe: 131,072 x 2 seqs at bf16 KV, util 0.98 boots and
|
||||
# serves (measured "GPU KV cache size: 138,561 tokens"). 0.99 fails the
|
||||
# free-memory wall. The window is not a capacity choice: the engine's own
|
||||
# reserve model (one max_model_len request) needs a large L for this
|
||||
# model's per-sequence KDA state, and below ~92k the boot REFUSES with
|
||||
# "needed ... larger than the available KV cache memory". NBATCH pins the
|
||||
# prefill chunk that was measured.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=131072$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
|
||||
# 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES
|
||||
# 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and
|
||||
# reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is
|
||||
# PER RANK — sizing it to the global requirement OOMs the worker
|
||||
# (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4
|
||||
# full-length 200k contexts do NOT fit: four clients share the ~730k
|
||||
# token pool and each runs below 200k when all are deep.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_NBATCH=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
|
||||
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_NBATCH"' "$SERVE"
|
||||
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE"
|
||||
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
||||
grep -q -- '--enable-prefix-caching' "$SERVE"
|
||||
grep -q -- '--trust-remote-code' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the endpoint advertises the GLM tool dialect" {
|
||||
# Without --enable-auto-tool-choice the engine rejects tool_choice "auto"
|
||||
# with a 400 ("requires --enable-auto-tool-choice and --tool-call-parser"),
|
||||
# which breaks every tool-using agent on the endpoint.
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOLPARSER"' "$SERVE"
|
||||
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_TOOLPARSER=glm47$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE"
|
||||
@test "the endpoint serves NO tool dialect — tool agents route elsewhere" {
|
||||
# The 2339fae trio does NOT wire --enable-auto-tool-choice /
|
||||
# --tool-call-parser (the GLM era added glm47 dialect flags; the rollback
|
||||
# dropped them). tool_choice "auto" gets a 400 here — tool-using agents
|
||||
# must route at frank-small or frank-large until a parser is qualified
|
||||
# for this arm (a behavior change beyond the rollback, not a silent fix).
|
||||
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
! grep -q -- '--tool-call-parser' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expandable segments is ON — the cache arena leaves no slack" {
|
||||
# The expert-offload plan fills VRAM to the modelled headroom, so the
|
||||
# profile/graph-capture phase has ~0.2 GiB to work with and the default
|
||||
# allocator's fragmentation fails it: measured 2026-09-30 on the deployed
|
||||
# unit, "Tried to allocate 662.00 MiB ... 240.00 MiB is free" from the
|
||||
# engine core while the identical shape booted with this env set.
|
||||
grep -q 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' "$SERVE"
|
||||
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
|
||||
# ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
|
||||
# expert planner moves the overflow to pinned host rows by itself (the
|
||||
# arena 6,574 slots x 2.344 MiB plus host rows). There is no
|
||||
# --enable-expert-offload flag in this arm; the legality of the pin is
|
||||
# the unit's unbounded LimitMEMLOCK.
|
||||
! grep -q -- '--enable-expert-offload' "$SERVE"
|
||||
! grep -q -- '--expert-precise-memory' "$SERVE"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expert offload is ON — the payload cannot fit VRAM without it" {
|
||||
# 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag
|
||||
# the engine has no home for the expert tensors at all; the arena
|
||||
# (14.51 GiB = 2,383 slots) plus 73.64 GiB/rank of pinned host rows is the
|
||||
# only shape that boots. --expert-precise-memory is the qualified sizing.
|
||||
grep -q -- '--enable-expert-offload' "$SERVE"
|
||||
grep -q -- '--expert-precise-memory' "$SERVE"
|
||||
@test "MTP4 spec decode is PRESENT BY MEASUREMENT" {
|
||||
# 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57
|
||||
# tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec
|
||||
# baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured
|
||||
# 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses
|
||||
# (17.4% acceptance). The earlier "MTP buys zero decode" reading carried
|
||||
# the prefetch tax and is retired with it. ngram-mod never booted on this
|
||||
# model (fork bug: PLE inputs prepared only on the MTP path).
|
||||
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
|
||||
grep -q -- '--speculative-config' "$SERVE"
|
||||
}
|
||||
|
||||
@test "MTP spec decode is absent BY MEASUREMENT, not by omission" {
|
||||
# 2026-09-30: the MTP head loads and accepts well (mean accepted length
|
||||
# 3.76-4.90), but decode was unchanged (9.6-11.2 vs 10.0 tok/s) while
|
||||
# prefill collapsed 918 -> 255 tok/s — the verification forward gathers the
|
||||
# union of the batch's experts, so spec decode does not amortize a
|
||||
# residency-bound gather. It also costs 2.6 GiB of arena.
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
! grep -q 'speculative' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "fp8 KV is absent — it would disable the only MLA decode path on gfx12" {
|
||||
# _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend requires a
|
||||
# non-fp8 KV cache, and that rope-free Triton path is the only MLA decode
|
||||
# available on gfx12: aiter's MLA decode op is registered under
|
||||
# @if_aiter_supported, gated on CDNA>2. With fp8 KV the MTP draft step dies
|
||||
# with "_OpNamespace 'vllm' object has no attribute
|
||||
# 'rocm_aiter_mla_decode_fwd'". KV dtype therefore stays auto/bf16.
|
||||
! grep -q -- '--kv-cache-dtype' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_KV_DTYPE=' "$ENV_FILE"
|
||||
@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" {
|
||||
# --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes
|
||||
# them together; a V-only quant needs a fork patch). The GLM-era MLA
|
||||
# concern (sparse-triton MLA decode path refusing fp8 KV) does not apply
|
||||
# to this arm: qwen's attention path accepted the dtype and a completion
|
||||
# probe qualified it 2026-09-29.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expert prefetch is absent BY LAW, not by omission" {
|
||||
@@ -199,10 +201,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
@test "the model id matches the payload's served alias wiring" {
|
||||
# The payload dir and its shard count are the fetch script's output
|
||||
# layout; the fetch script must write where the env points.
|
||||
grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE"
|
||||
grep -q 'GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD"
|
||||
grep -q 'Intel/GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
|
||||
grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
|
||||
}
|
||||
|
||||
@test "no bare HIP-reserved env var is defined" {
|
||||
@@ -218,7 +220,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# storage or the payload is missing/wrong-sharded — then clears stale
|
||||
# containers (including the legacy identities), waits out the port,
|
||||
# relinks/enables/restarts the root unit and health-gates it.
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-glm' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
|
||||
grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
|
||||
|
||||
Reference in New Issue
Block a user