From 26f6f82c988a4612cece170da65ff70f6a4616c0 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 30 Sep 2026 20:24:49 -0700 Subject: [PATCH] frank-primary: qwen cutover docs + deployer identity + seam re-pin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4 (37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index. - runbook: RETIRED verdict — DFlash2 drafter served and drafted but acceptance pinned 1.00-1.38 on every text class including verbatim repetition (mis-wired aux capture, not a weak drafter); 5x slower than baseline at ~1.2 acceptance. - log.md: 2026-10-01 rollback entry. - deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen; glm-probe joins the legacy sweep list. - frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool dialect, automatic offload. The old 'MTP buys zero' law carried the prefetch tax and is retired. --- AGENTS.md | 137 ++++----------- scripts/deploy-remote.sh | 10 +- .../LLM/frank-primary-glm53-flash-w4a16.md | 16 ++ serverAppConfigs-docs/log.md | 19 +++ tests/frank_primary_unit.bats | 156 +++++++++--------- 5 files changed, 150 insertions(+), 188 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b555f43..f194b5e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill, bounded by unfused pointwise glue (the expert gather is 1.05 % of device time; the trunk is uncompiled) | 7770 | +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -423,115 +423,39 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-primary — tcclaviger vLLM, GLM-5.3-Flash W4A16 int4 (Dankserver, ACTIVE) +### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE) -The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…` -— the same digest-pinned image the 2026-09-29 qwen tuning campaign measured) -serving **`Intel/GLM-5.3-Flash-W4A16-AutoRound`**: AutoRound **symmetric int4, -group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, -113,074 tensors, ~170 GiB — on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**, -**131,072 ctx × 2 seqs**, bf16 KV, util 0.98, NBATCH 2048, prefix caching, -**no MTP**, **no prefetch**. Cut over 2026-09-30 from the qwen FP8 + MTP4 shape, -which itself replaced the R9V radiance fork on 2026-09-29. +The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`, +digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** +(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**, +**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs, +**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the +GLM/DFlash2 experiment was declared failed (same day). -**The payload is chosen for the offload path, not for the quant.** 170 GiB of -int4 experts cannot fit 64 GiB of VRAM, so expert offload is mandatory — and -the fork reaches its W4A16 kernel (`R4dW4a16MoEMethod`, libr4d's grouped W4A16 -GEMM, 6.234 MiB per expert-rank slot) **only** when the checkpoint's -`quant_method` is `auto-round`, which the fork overrides to `inc`. A -compressed-tensors W4A16 checkpoint of the same model resolves to -`TritonWNA16Experts`, which **has no offload recipe at all**, and refuses to -serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over -12,096 host-backed experts = **73.64 GiB pinned host rows per rank**. +Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance +~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the +35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums +BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a +throughput one. V-only q8 KV is NOT possible via config: stock +`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch. -**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k -prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met. The first -explanation for that — "the expert gather is the bound at ~8.2 GB/s" — was -FALSIFIED by a torch-profiler kernel-time window** (496 decode tokens, rank0 -trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms per -token, of which **44 % is pointwise glue** — `vectorized_elementwise_kernel` -alone is 16.07 s over **708,500 launches (1,428 per token)**, plus 3.11 s of -`elementwise_kernel_manual_unroll` — against **1.05 % for -`r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K GEMM 6 %, the -router/top-k 9 %, the mHC kernels 6.5 %. The real wall is that **the fork -compiles its own Qwen trunk (`@support_torch_compile` in -`models/qwen4_exp/amd/model.py`) and not its GLM trunk**, so every pointwise op -is a separate eager kernel. Two consequences: spec decode **cannot** help this -shape (pointwise cost scales with the token count, which is exactly why MTP4 -measured flat), and neither arena size nor hit rate is a lever (the gather is -1 %). +The engine auto-sets the attention block size (mamba-page floor); the +expert plan's registry prices KV per token; the expert arena (6,574 slots +x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it +for KV headroom collapses the offload hit rate. -**Fork work is staged, not deployed.** Two WIP changelists, both apply-verified -against the pinned image (`patch -p1` reproduces each artifact's recorded sha256): -`scripts/patches/glm53-compile-0{1..5}*.patch` (decorate the GLM trunk; freeze the ten -libr4d capability probes that stat/`.so`-probe *inside* the traced forward; make -`fullgraph=True` opt-out; take GLM off the fused-norm JIT kernel; make the indexer's -in-forward workspace acquisition compiler-opaque) and -`glm53-dflash-01-aux-hidden-states.patch` (the EAGLE3 interface, below). Boots -established that `functools.lru_cache` does **not** protect a probe (dynamo re-traces -the wrapped body even warm), that `suppress_errors` does **not** cover graph breaks -(only `fullgraph=False` does), and that the GLM trunk can never satisfy `fullgraph` -(its indexer metadata syncs to host, `glm5next/amd/sparse_indexer.py:263,419`). With -breaks genuinely tolerated the trunk **traces** (`Dynamo bytecode transform time: -1.08 s`) and the single remaining blocker is -`KeyError: 'in_proj_qkvbfg_a'` — raised in a callee dynamo inlines into -`Glm5NextDecoderLayer.forward`, **not** at any of dynamo's four `nn_modules` lookups and -**not** a literal string lookup, so it is currently a dynamo-side name-resolution -failure rather than a demonstrated fork bug (an earlier framing here said "a latent bug -the compile exposes"; that was retracted). Watch the launcher/reset gotchas: the scratch -launcher once silently dropped `VLLM_COMPILE_FULLGRAPH=0` (making four boots run -`fullgraph=True` where every break is fatal), and a `sitecustomize.py` on `PYTHONPATH` -became a fork bomb — see the runbook. +Unit and self-healing: root systemd unit (`scripts/frank-primary.service`, +`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120 +frank-primary-glm`. The serve script drives `podman run` directly, gates +the digest-pinned image, the payload's `config.json` and its exact +131-shard count and the GPU device nodes, clears stale container +identities, then blocks on `podman wait`. `deploy_llm_primary` gates the +same, waits out :7770, relinks/enables/restarts the unit and health-gates +with the MainPID check. Boot cost ~7 min warm; health returns 503 while +loading. -**No new HIP kernels are possible** — the image ships `r4d.so` as a binary with -no sources. The 9 % router is therefore **binary-blocked**: its fused path is -gated `num_experts in {128, 256, 512}` (`router/r4d_route_router.py`) and only -`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has **288** -experts. Only Python/config/inductor changes (or runtime-JIT Triton) are -available on this fork. - -**DFlash2 for GLM is a target-side port, not a drafter port.** -`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]` -and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8, -`target_layer_ids [5, 14, 24, 33, 42]` of 45), and the fork **already maps it** -(`registry.py:636` → `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method -`"dflash"`). What was missing is the target side — `glm5next` implemented neither -`set_aux_hidden_state_layers()` nor the per-layer aux capture that -`deepseek_v2.py` (also MLA+MoE) demonstrates — and that is now written: -`scripts/patches/glm53-dflash-01-aux-hidden-states.patch` implements exactly that -contract (`EagleModelMixin` on the trunk, capture of the single-stream state entering -each requested layer, the conditional `(hidden_states, aux_hidden_states)` tuple -return, and the two delegating methods on `Glm5NextForCausalLM`). The engine's -refusal was literally `RuntimeError: Model does not support EAGLE3 interface`, so -this patch is the prerequisite for driving the drafter at all. Budget no speedup: -per the profile above, a verification pass pays pointwise cost proportional to its -token count. - -**MTP works here and is still rejected.** The head loads -(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted -length 3.76–4.90, per-position 0.92/0.85/0.69/0.58, 69–97 %), but decode was -unchanged (9.6–11.2 vs 10.0 tok/s) while prefill **collapsed 918 → 255 tok/s**: -a verification forward gathers the *union* of the batch's experts, so spec -decode amortizes compute, not a residency-bound gather. It also costs 2.6 GiB -of arena. **fp8 KV is illegal on this model** — it disables the sparse MLA -backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2, -so the MTP draft step demands an op that cannot exist on gfx12 -(`no attribute 'rocm_aiter_mla_decode_fwd'`). - -**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`, -`User=dank`, `LimitMEMLOCK=infinity` — the 73.64 GiB/rank expert pin needs it), -`ExecStop=podman stop -t 120 frank-primary-glm`. The serve script drives -`podman run` **directly** (the image entrypoint execs `vllm serve`), gates the -digest-pinned image, the payload's `config.json` and its exact 34-shard count -and the GPU device nodes, clears the stale container identities, then **blocks -on `podman wait`** so a crash is a nonzero ExecStart. The deployer's -`deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts -the unit and health-gates with the MainPID check (the false-green stray law). -Boot cost ~7 min warm (~15 min cold); health returns 503 while loading. - -Runbook: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`. -Rollback: commit `2339fae` (qwen FP8 + MTP4) — restore `scripts/` from it and -deploy; the qwen payload and the R9V state dirs stay on disk. +Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`. +Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`. ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the @@ -880,7 +804,7 @@ Active: |---------|---------| | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | -| frank-primary (tcclaviger vLLM, GLM-5.3-Flash W4A16 int4, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md` | +| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -888,6 +812,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| +|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`| | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` | | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` | | llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 2a8c8ea..1825e46 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1217,8 +1217,8 @@ deploy_llm_small() { # (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file # sha256 verification was the fetch script's job at download time, never # re-hashed here. -LLM_PRIMARY_CONTAINER=frank-primary-glm -LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe" +LLM_PRIMARY_CONTAINER=frank-primary-qwen +LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8 glm-probe" deploy_llm_primary() { # Refuse rather than restart-loop: a unit whose image or payload is @@ -1270,9 +1270,9 @@ deploy_llm_primary() { # A stopped unit can leave its container running: systemd's cgroup kill # races conmon's --rm cleanup, and the orphan holds the port and GPUs. # Stop the unit, then remove the containers BEFORE waiting on the port. - # The legacy identities (the retired R9V container and the ad hoc - # experiment container qwen-fp8, the ad hoc glm-probe) hold the same GPUs - # and die here too. + # The legacy identities (the retired R9V container, the ad hoc + # experiment container qwen-fp8, the retired glm-probe) hold the same + # GPUs and die here too. run_privileged systemctl stop frank-primary.service 2>/dev/null || true local ctr for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index fb43186..0d53e9e 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -18,6 +18,22 @@ timestamp: 2026-09-30 # frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) +> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8 +> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a +> **failure**: after five patches the drafter served and drafted (12.8 tok/s +> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but +> **acceptance pinned at 1.00-1.38 on every text class including verbatim +> repetition** — the signature of a mis-wired aux-hidden-state capture, not a +> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode +> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The +> untested alternative capture semantics: post-layer output state +> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload +> (170 GiB) + drafter (2.2 GiB) staged to +> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval. +> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are +> generic unifier/tailpin/grouping fixes, apply-verified against the image. + + > **Question this page answers.** Can the fleet serve the 320B-class > `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it? > **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index f246d28..8ff6c7f 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,24 @@ # Bundle Change Log +## 2026-10-01 +- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4; + GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and + served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks + captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper + interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with + draft+hidden groups, registry method-signal), but **acceptance pinned at + 1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired + aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the + 8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline + (5x slower). Frank's independently reported 18 tok/s / ~370 prefill did + not originate from this engine (APIServer access log: every completion + POST was host-local; his workstation appears once, a /v1/models GET). + Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio + (200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter + (2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`. + KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic + unifier/tailpin/grouping/registry fixes, apply-verified). + ## 2026-09-30 - **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at the 35 tok/s target, and the shortfall is measured rather than inferred.** diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 21a9bfc..82defd9 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -1,22 +1,26 @@ #!/usr/bin/env bats # Seam: frank-primary — the Dankserver LLM endpoint on :7770. # -# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-30 -# cutover to GLM-5.3-Flash W4A16 int4, which replaced the qwen FP8 + MTP4 -# shape): the serve script drives `podman run` DIRECTLY (the image entrypoint -# execs `vllm serve`), with every engine flag qualified by boot probe on -# 2026-09-30 and carried in the repo env file. The seam pins what the flags -# alone do not say: the qualification law (131,072 x 2 seqs at bf16 KV, util -# 0.98 is what boots; the window is forced up by the fork's KV reserve model, -# not by capacity), the offload law (170 GiB of experts cannot fit 64 GiB of -# VRAM — the engine MUST run with --enable-expert-offload), the self-healing -# law (the run returns before the engine is up, so the serve script must wait -# on the container for Restart=on-failure to mean anything), the memlock law -# that makes the 73.64 GiB/rank expert pin legal, the deployer's image/payload -# gates with the false-green MainPID check, and the three measured-absent -# flags: expert prefetch (36% precision, costs ~40-50% decode), MTP spec -# decode (buys zero decode and collapses prefill on an offload-bound shape), -# and fp8 KV (disables the only MLA decode path gfx12 has). +# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the +# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which +# reversed the failed GLM/DFlash2 experiment): the serve script drives +# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every +# engine flag qualified by boot probe and carried in the repo env file. The +# seam pins what the flags alone do not say: the KV qualification law +# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB +# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV +# qualification (e4m3 for BOTH K and V — the stock flag quantizes them +# together), the automatic-offload law (the fork plans the tiered-expert +# offload itself; no flag exists to set), the self-healing law (the run +# returns before the engine is up, so the serve script must wait on the +# container for Restart=on-failure to mean anything), the memlock law that +# makes the pinned host rows legal, the deployer's image/payload gates with +# the false-green MainPID check, and the measured laws: expert prefetch +# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved +# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance +# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading +# carried the prefetch tax), and NO tool dialect (the GLM era added +# --enable-auto-tool-choice; the 2339fae trio does not serve tools). load helpers @@ -56,8 +60,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # outlives the unit and holds both GPUs. The container name must match # the env file's identity. grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT" - grep -q 'frank-primary-glm' "$UNIT" - grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-glm$' "$ENV_FILE" + grep -q 'frank-primary-qwen' "$UNIT" + grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE" } @test "the serve shape is direct podman run, not a launcher or shim" { @@ -72,77 +76,75 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" } @test "the qualified engine shape is in the env file" { - # 2026-09-30 boot probe: 131,072 x 2 seqs at bf16 KV, util 0.98 boots and - # serves (measured "GPU KV cache size: 138,561 tokens"). 0.99 fails the - # free-memory wall. The window is not a capacity choice: the engine's own - # reserve model (one max_model_len request) needs a large L for this - # model's per-sequence KDA state, and below ~92k the boot REFUSES with - # "needed ... larger than the available KV cache memory". NBATCH pins the - # prefill chunk that was measured. - grep -q '^FRANK_PRIMARY_MAXLEN=131072$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE" + # 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES + # 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and + # reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is + # PER RANK — sizing it to the global requirement OOMs the worker + # (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4 + # full-length 200k contexts do NOT fit: four clients share the ~730k + # token pool and each runs below 200k when all are deep. + grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_NBATCH=2048$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE" } @test "the serve script wires the engine flags from the env" { grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE" grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE" - grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_NBATCH"' "$SERVE" + grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE" + grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE" + grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE" grep -q -- '--tensor-parallel-size 2' "$SERVE" grep -q -- '--enable-prefix-caching' "$SERVE" grep -q -- '--trust-remote-code' "$SERVE" } -@test "the endpoint advertises the GLM tool dialect" { - # Without --enable-auto-tool-choice the engine rejects tool_choice "auto" - # with a 400 ("requires --enable-auto-tool-choice and --tool-call-parser"), - # which breaks every tool-using agent on the endpoint. - grep -q -- '--enable-auto-tool-choice' "$SERVE" - grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOLPARSER"' "$SERVE" - grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER"' "$SERVE" - grep -q '^FRANK_PRIMARY_TOOLPARSER=glm47$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE" +@test "the endpoint serves NO tool dialect — tool agents route elsewhere" { + # The 2339fae trio does NOT wire --enable-auto-tool-choice / + # --tool-call-parser (the GLM era added glm47 dialect flags; the rollback + # dropped them). tool_choice "auto" gets a 400 here — tool-using agents + # must route at frank-small or frank-large until a parser is qualified + # for this arm (a behavior change beyond the rollback, not a silent fix). + ! grep -q -- '--enable-auto-tool-choice' "$SERVE" + ! grep -q -- '--tool-call-parser' "$SERVE" } -@test "expandable segments is ON — the cache arena leaves no slack" { - # The expert-offload plan fills VRAM to the modelled headroom, so the - # profile/graph-capture phase has ~0.2 GiB to work with and the default - # allocator's fragmentation fails it: measured 2026-09-30 on the deployed - # unit, "Tried to allocate 662.00 MiB ... 240.00 MiB is free" from the - # engine core while the identical shape booted with this env set. - grep -q 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' "$SERVE" +@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" { + # ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered- + # expert planner moves the overflow to pinned host rows by itself (the + # arena 6,574 slots x 2.344 MiB plus host rows). There is no + # --enable-expert-offload flag in this arm; the legality of the pin is + # the unit's unbounded LimitMEMLOCK. + ! grep -q -- '--enable-expert-offload' "$SERVE" + ! grep -q -- '--expert-precise-memory' "$SERVE" + grep -q '^LimitMEMLOCK=infinity$' "$UNIT" + grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE" } -@test "expert offload is ON — the payload cannot fit VRAM without it" { - # 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag - # the engine has no home for the expert tensors at all; the arena - # (14.51 GiB = 2,383 slots) plus 73.64 GiB/rank of pinned host rows is the - # only shape that boots. --expert-precise-memory is the qualified sizing. - grep -q -- '--enable-expert-offload' "$SERVE" - grep -q -- '--expert-precise-memory' "$SERVE" +@test "MTP4 spec decode is PRESENT BY MEASUREMENT" { + # 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57 + # tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec + # baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured + # 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses + # (17.4% acceptance). The earlier "MTP buys zero decode" reading carried + # the prefetch tax and is retired with it. ngram-mod never booted on this + # model (fork bug: PLE inputs prepared only on the MTP path). + grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE" + grep -q -- '--speculative-config' "$SERVE" } -@test "MTP spec decode is absent BY MEASUREMENT, not by omission" { - # 2026-09-30: the MTP head loads and accepts well (mean accepted length - # 3.76-4.90), but decode was unchanged (9.6-11.2 vs 10.0 tok/s) while - # prefill collapsed 918 -> 255 tok/s — the verification forward gathers the - # union of the batch's experts, so spec decode does not amortize a - # residency-bound gather. It also costs 2.6 GiB of arena. - ! grep -q -- '--speculative-config' "$SERVE" - ! grep -q 'speculative' "$ENV_FILE" -} - -@test "fp8 KV is absent — it would disable the only MLA decode path on gfx12" { - # _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend requires a - # non-fp8 KV cache, and that rope-free Triton path is the only MLA decode - # available on gfx12: aiter's MLA decode op is registered under - # @if_aiter_supported, gated on CDNA>2. With fp8 KV the MTP draft step dies - # with "_OpNamespace 'vllm' object has no attribute - # 'rocm_aiter_mla_decode_fwd'". KV dtype therefore stays auto/bf16. - ! grep -q -- '--kv-cache-dtype' "$SERVE" - ! grep -q '^FRANK_PRIMARY_KV_DTYPE=' "$ENV_FILE" +@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" { + # --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes + # them together; a V-only quant needs a fork patch). The GLM-era MLA + # concern (sparse-triton MLA decode path refusing fp8 KV) does not apply + # to this arm: qwen's attention path accepted the dtype and a completion + # probe qualified it 2026-09-29. + grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" + grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE" } @test "expert prefetch is absent BY LAW, not by omission" { @@ -199,10 +201,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" @test "the model id matches the payload's served alias wiring" { # The payload dir and its shard count are the fetch script's output # layout; the fetch script must write where the env points. - grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE" - grep -q 'GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD" - grep -q 'Intel/GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD" + grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE" + grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD" + grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD" } @test "no bare HIP-reserved env var is defined" { @@ -218,7 +220,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # storage or the payload is missing/wrong-sharded — then clears stale # containers (including the legacy identities), waits out the port, # relinks/enables/restarts the root unit and health-gates it. - grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-glm' "$REMOTE_DEPLOY" + grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY" grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY" grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY" grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"