dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: qwen cutover docs + deployer identity + seam re-pin

Browse Source 
- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4
  (37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index.
- runbook: RETIRED verdict — DFlash2 drafter served and drafted but
  acceptance pinned 1.00-1.38 on every text class including verbatim
  repetition (mis-wired aux capture, not a weak drafter); 5x slower than
  baseline at ~1.2 acceptance.
- log.md: 2026-10-01 rollback entry.
- deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen;
  glm-probe joins the legacy sweep list.
- frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification
  law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV
  PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool
  dialect, automatic offload. The old 'MTP buys zero' law carried the
  prefetch tax and is retired.
This commit is contained in:
Frank committed 2026-09-30 20:24:49 -07:00
1 parent d91379cff2
commit 26f6f82c98
5 files changed
+150 -188
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+31 -106
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill, bounded by unfused pointwise glue (the expert gather is 1.05 % of device time; the trunk is uncompiled) | 7770 |
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload behind a 15.05 GiB / 6,574-slot VRAM LRU arena, MTP4, prefetch OFF, 200000 ctx × 3 seqs, fp8 KV; measured 37.4-40.6 tok/s decode / ~54 % acceptance (the fleet's only endpoint meeting the 35 target). Rolled back 2026-10-01 from the failed GLM/DFlash2 experiment (commit `2339fae` trio) | 7770 |
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -423,115 +423,39 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-primary — tcclaviger vLLM, GLM-5.3-Flash W4A16 int4 (Dankserver, ACTIVE)
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`
— the same digest-pinned image the 2026-09-29 qwen tuning campaign measured)
serving **`Intel/GLM-5.3-Flash-W4A16-AutoRound`**: AutoRound **symmetric int4,
group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors,
113,074 tensors, ~170 GiB — on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
**131,072 ctx × 2 seqs**, bf16 KV, util 0.98, NBATCH 2048, prefix caching,
**no MTP**, **no prefetch**. Cut over 2026-09-30 from the qwen FP8 + MTP4 shape,
which itself replaced the R9V radiance fork on 2026-09-29.
The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
**MTP4** speculative decoding, **expert prefetch OFF**, 200,000 ctx x 3 seqs,
**fp8 KV**. Deployed 2026-10-01 by rollback from commit `2339fae` after the
GLM/DFlash2 experiment was declared failed (same day).
**The payload is chosen for the offload path, not for the quant.** 170 GiB of
int4 experts cannot fit 64 GiB of VRAM, so expert offload is mandatory — and
the fork reaches its W4A16 kernel (`R4dW4a16MoEMethod`, libr4d's grouped W4A16
GEMM, 6.234 MiB per expert-rank slot) **only** when the checkpoint's
`quant_method` is `auto-round`, which the fork overrides to `inc`. A
compressed-tensors W4A16 checkpoint of the same model resolves to
`TritonWNA16Experts`, which **has no offload recipe at all**, and refuses to
serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over
12,096 host-backed experts = **73.64 GiB pinned host rows per rank**.
Measured 2026-09-29 tuning campaign: **decode 37.4-40.6 tok/s, acceptance
~54 %** (MTP4 + prefetch off — the fleet's only endpoint that meets the
35 tok/s target). 256k ctx arm measured 34.1-36.5; 2-way concurrency sums
BELOW parity (25.49 vs 26.8 single) — slots are a capacity feature, not a
throughput one. V-only q8 KV is NOT possible via config: stock
`--kv-cache-dtype` is K+V together; a V-only quant needs a fork patch.
**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k
prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met. The first
explanation for that — "the expert gather is the bound at ~8.2 GB/s" — was
FALSIFIED by a torch-profiler kernel-time window** (496 decode tokens, rank0
trace parsed for device-activity time): 46.18 s of kernel time = ~93 ms per
token, of which **44 % is pointwise glue** — `vectorized_elementwise_kernel`
alone is 16.07 s over **708,500 launches (1,428 per token)**, plus 3.11 s of
`elementwise_kernel_manual_unroll` — against **1.05 % for
`r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K GEMM 6 %, the
router/top-k 9 %, the mHC kernels 6.5 %. The real wall is that **the fork
compiles its own Qwen trunk (`@support_torch_compile` in
`models/qwen4_exp/amd/model.py`) and not its GLM trunk**, so every pointwise op
is a separate eager kernel. Two consequences: spec decode **cannot** help this
shape (pointwise cost scales with the token count, which is exactly why MTP4
measured flat), and neither arena size nor hit rate is a lever (the gather is
1 %).
The engine auto-sets the attention block size (mamba-page floor); the
expert plan's registry prices KV per token; the expert arena (6,574 slots
x 2.344 MiB = 15.05 GiB/rank) is the decode-critical region — cutting it
for KV headroom collapses the offload hit rate.
**Fork work is staged, not deployed.** Two WIP changelists, both apply-verified
against the pinned image (`patch -p1` reproduces each artifact's recorded sha256):
`scripts/patches/glm53-compile-0{1..5}*.patch` (decorate the GLM trunk; freeze the ten
libr4d capability probes that stat/`.so`-probe *inside* the traced forward; make
`fullgraph=True` opt-out; take GLM off the fused-norm JIT kernel; make the indexer's
in-forward workspace acquisition compiler-opaque) and
`glm53-dflash-01-aux-hidden-states.patch` (the EAGLE3 interface, below). Boots
established that `functools.lru_cache` does **not** protect a probe (dynamo re-traces
the wrapped body even warm), that `suppress_errors` does **not** cover graph breaks
(only `fullgraph=False` does), and that the GLM trunk can never satisfy `fullgraph`
(its indexer metadata syncs to host, `glm5next/amd/sparse_indexer.py:263,419`). With
breaks genuinely tolerated the trunk **traces** (`Dynamo bytecode transform time:
1.08 s`) and the single remaining blocker is
`KeyError: 'in_proj_qkvbfg_a'` — raised in a callee dynamo inlines into
`Glm5NextDecoderLayer.forward`, **not** at any of dynamo's four `nn_modules` lookups and
**not** a literal string lookup, so it is currently a dynamo-side name-resolution
failure rather than a demonstrated fork bug (an earlier framing here said "a latent bug
the compile exposes"; that was retracted). Watch the launcher/reset gotchas: the scratch
launcher once silently dropped `VLLM_COMPILE_FULLGRAPH=0` (making four boots run
`fullgraph=True` where every break is fatal), and a `sitecustomize.py` on `PYTHONPATH`
became a fork bomb — see the runbook.
Unit and self-healing: root systemd unit (`scripts/frank-primary.service`,
`User=dank`, `LimitMEMLOCK=infinity`), `ExecStop=podman stop -t 120
frank-primary-glm`. The serve script drives `podman run` directly, gates
the digest-pinned image, the payload's `config.json` and its exact
131-shard count and the GPU device nodes, clears stale container
identities, then blocks on `podman wait`. `deploy_llm_primary` gates the
same, waits out :7770, relinks/enables/restarts the unit and health-gates
with the MainPID check. Boot cost ~7 min warm; health returns 503 while
loading.
**No new HIP kernels are possible** — the image ships `r4d.so` as a binary with
no sources. The 9 % router is therefore **binary-blocked**: its fused path is
gated `num_experts in {128, 256, 512}` (`router/r4d_route_router.py`) and only
`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has **288**
experts. Only Python/config/inductor changes (or runtime-JIT Triton) are
available on this fork.
**DFlash2 for GLM is a target-side port, not a drafter port.**
`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
`target_layer_ids [5, 14, 24, 33, 42]` of 45), and the fork **already maps it**
(`registry.py:636` → `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
`"dflash"`). What was missing is the target side — `glm5next` implemented neither
`set_aux_hidden_state_layers()` nor the per-layer aux capture that
`deepseek_v2.py` (also MLA+MoE) demonstrates — and that is now written:
`scripts/patches/glm53-dflash-01-aux-hidden-states.patch` implements exactly that
contract (`EagleModelMixin` on the trunk, capture of the single-stream state entering
each requested layer, the conditional `(hidden_states, aux_hidden_states)` tuple
return, and the two delegating methods on `Glm5NextForCausalLM`). The engine's
refusal was literally `RuntimeError: Model does not support EAGLE3 interface`, so
this patch is the prerequisite for driving the drafter at all. Budget no speedup:
per the profile above, a verification pass pays pointwise cost proportional to its
token count.
**MTP works here and is still rejected.** The head loads
(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted
length 3.76–4.90, per-position 0.92/0.85/0.69/0.58, 69–97 %), but decode was
unchanged (9.6–11.2 vs 10.0 tok/s) while prefill **collapsed 918 → 255 tok/s**:
a verification forward gathers the *union* of the batch's experts, so spec
decode amortizes compute, not a residency-bound gather. It also costs 2.6 GiB
of arena. **fp8 KV is illegal on this model** — it disables the sparse MLA
backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2,
so the MTP draft step demands an op that cannot exist on gfx12
(`no attribute 'rocm_aiter_mla_decode_fwd'`).
**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
`User=dank`, `LimitMEMLOCK=infinity` — the 73.64 GiB/rank expert pin needs it),
`ExecStop=podman stop -t 120 frank-primary-glm`. The serve script drives
`podman run` **directly** (the image entrypoint execs `vllm serve`), gates the
digest-pinned image, the payload's `config.json` and its exact 34-shard count
and the GPU device nodes, clears the stale container identities, then **blocks
on `podman wait`** so a crash is a nonzero ExecStart. The deployer's
`deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts
the unit and health-gates with the MainPID check (the false-green stray law).
Boot cost ~7 min warm (~15 min cold); health returns 503 while loading.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
Rollback: commit `2339fae` (qwen FP8 + MTP4) — restore `scripts/` from it and
deploy; the qwen payload and the R9V state dirs stay on disk.
Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -880,7 +804,7 @@ Active:
|---------|---------|
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-primary (tcclaviger vLLM, GLM-5.3-Flash W4A16 int4, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md` |
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -888,6 +812,7 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |