dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary cutover: GLM-5.3-Flash W4A16 int4 (Intel AutoRound)

Browse Source 
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).

Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.

Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.

Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).

Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.

Rollback: 2339fae (qwen FP8 + MTP4).
This commit is contained in:
Frank committed 2026-09-30 04:22:11 -07:00
1 parent 2339faeaa4
commit f2bcf84e88
12 files changed
+668 -224
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+52 -76
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 |
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill | 7770 |
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -245,7 +245,7 @@ merely wedged kernel usually still answers ARP.
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
- Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700):
- Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
2. Bench.
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
@@ -423,86 +423,62 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE)
### frank-primary — tcclaviger vLLM, GLM-5.3-Flash W4A16 int4 (Dankserver, ACTIVE)
The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned
prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving
the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards)
with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI
PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at
**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE
n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0
template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork
`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut
over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from
llama.cpp b11030 the same day) at Frank's direction.
The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…`
— the same digest-pinned image the 2026-09-29 qwen tuning campaign measured)
serving **`Intel/GLM-5.3-Flash-W4A16-AutoRound`**: AutoRound **symmetric int4,
group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors,
113,074 tensors, ~170 GiB — on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
**131,072 ctx × 2 seqs**, bf16 KV, util 0.98, NBATCH 2048, prefix caching,
**no MTP**, **no prefetch**. Cut over 2026-09-30 from the qwen FP8 + MTP4 shape,
which itself replaced the R9V radiance fork on 2026-09-29.
**The tuned shape lives on-host, not in the repo.** The q8 profile's setup
run qualified the `R9V_*` config into
`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env
file `scripts/frank-primary.env` carries ONLY production overrides
(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`,
`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`,
`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter
(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload
quantizes BOTH fused hyper-connection shards, so the shard-uniformity
check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the
overlay's env.always injects the uncensored 160-slot cache geometry, which
the q8 80-slot tiered cache rejects). Never copy tuned values from the
state into the repo — it would drift from the qualification.
**The payload is chosen for the offload path, not for the quant.** 170 GiB of
int4 experts cannot fit 64 GiB of VRAM, so expert offload is mandatory — and
the fork reaches its W4A16 kernel (`R4dW4a16MoEMethod`, libr4d's grouped W4A16
GEMM, 6.234 MiB per expert-rank slot) **only** when the checkpoint's
`quant_method` is `auto-round`, which the fork overrides to `inc`. A
compressed-tensors W4A16 checkpoint of the same model resolves to
`TritonWNA16Experts`, which **has no offload recipe at all**, and refuses to
serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over
12,096 host-backed experts = **73.64 GiB pinned host rows per rank**.
**Spec decoding is MTP4 only.** The fork's `speculative_config` is a
single dict — one method per engine, NO chaining — so the llama.cpp
`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired
(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along
architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative
single method, not a chainable one.
**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k
prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met, and this is
a measured wall rather than a tuning gap:** each decode step must move 2.045 GiB
of routed experts per rank (42 layers × 8 experts × 6.234 MiB), live telemetry
(`R4D_LRU_TELEMETRY=1`) puts the expert-cache hit rate at **0.53–0.66**, and the
fork's gather read path delivers **~8.2 GB/s against a link measured at
22.74 GB/s** — the same structural gap the qwen campaign found. Growing the
arena 13.08 → 14.51 GiB changed decode by *nothing* (10.03 → 9.88 tok/s), so
residency stopped being the binding term; 0.89 hit rate would be needed for 35.
**Serving = the fork's own launcher through a docker→podman shim.** The R9V
tooling is docker-pinned; the shim at
`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries
three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys
the image by its OCI manifest digest while podman keys by config digest but
preserves the manifest digest in `RepoDigests` — the shim answers
`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run
refs to `repo@sha256:X`; (2) podman inspect answers
`HostConfig.LogConfig.Config: null` where docker returns the log-opts map
(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs
and normalizes inspect output; (3) podman lacks docker's `SecurityOptions`
/`ServerVersion` info fields — the shim answers them.
**MTP works here and is still rejected.** The head loads
(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted
length 3.76–4.90, per-position 0.92/0.85/0.69/0.58, 69–97 %), but decode was
unchanged (9.6–11.2 vs 10.0 tok/s) while prefill **collapsed 918 → 255 tok/s**:
a verification forward gathers the *union* of the batch's experts, so spec
decode amortizes compute, not a residency-bound gather. It also costs 2.6 GiB
of arena. **fp8 KV is illegal on this model** — it disables the sparse MLA
backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2,
so the MTP draft step demands an op that cannot exist on gfx12
(`no attribute 'rocm_aiter_mla_decode_fwd'`).
**Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`,
`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is
54.4 GB, and a root unit grants the pin budget via PID1 (a rootless
quadlet would be capped by the user manager's limit).
The serve script (`scripts/serve-frank-primary.sh`) gates the setup state
(ready + paths), the pinned image (RepoDigests), the PLE byte count
(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes,
clears a stale container, then execs the fork's `scripts/launch.sh` through
the shim and **blocks on `docker wait`** — the launcher `--detach`es, so
without the wait a crashed engine is a green unit; `docker wait` cannot stop
the container child, hence `ExecStop=podman stop -t 120
frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/
PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/
restarts the unit and health-gates with the MainPID check (the false-green
stray law).
`User=dank`, `LimitMEMLOCK=infinity` — the 73.64 GiB/rank expert pin needs it),
`ExecStop=podman stop -t 120 frank-primary-glm`. The serve script drives
`podman run` **directly** (the image entrypoint execs `vllm serve`), gates the
digest-pinned image, the payload's `config.json` and its exact 34-shard count
and the GPU device nodes, clears the stale container identities, then **blocks
on `podman wait`** so a crash is a nonzero ExecStart. The deployer's
`deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts
the unit and health-gates with the MainPID check (the false-green stray law).
Boot cost ~7 min warm (~15 min cold); health returns 503 while loading.
Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20
payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up
battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance
length 3.19–5.00, and the PLE pinned proof (`mapping_bytes =
mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool
round-trip + template probe re-verified on :7770 after the deploy. No
throughput bench was run (Frank declined); the probe-era 11 tok/s decode
was residency-bound (halved expert headroom at double the bytes/expert),
not the production number.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`.
Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
and the template keys, redeploy (state + `restore-config-20260927.env` +
payload intact on disk) — see
[[frank-primary-r9v-radiance-qwen38-uncensored]].
Runbook: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
Rollback: commit `2339fae` (qwen FP8 + MTP4) — restore `scripts/` from it and
deploy; the qwen payload and the R9V state dirs stay on disk.
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -851,7 +827,7 @@ Active:
|---------|---------|
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` |
| frank-primary (tcclaviger vLLM, GLM-5.3-Flash W4A16 int4, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |