Comrade Chief Designer
bd8c72ff47
A torch-profiler kernel-time window (496 decode tokens, rank0 trace parsed for
device-activity time) on the deployed GLM-5.3-Flash shape falsifies this page's
own "the expert gather is the bound" explanation:
46.18 s of kernel time / 496 tokens = ~93 ms per token (actual step ~104 ms)
34.8 % vectorized_elementwise_kernel 708,500 launches (1,428/token)
10.0 % r4d_gemm_moe_w4a16_nt_b16_m64 (the MoE GEMM)
6.7 % elementwise_kernel_manual_unroll
6.4 % wvSplitK_hf_sml_ (split-K GEMM)
8.9 % router: sbtopk + warpMergeSortTopK + bitonicSortKVInPlace
6.5 % r4d_mhc_* (hyper-connections)
1.05 % r4d_moe_lru_gather_kernel <-- the retracted suspect
The earlier arithmetic was circular (it assumed the gather consumed the whole
step). The real bound is unfused pointwise glue: the fork compiles its own Qwen
trunk (support_torch_compile in models/qwen4_exp/amd/model.py) and not its GLM
trunk, so the engine boots with "provides neither a compiled submodule nor
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY." That also explains why
MTP4 measured flat — pointwise cost scales with the token count, so spec decode
amortizes nothing here — and it retires arena/hit-rate tuning (the gather is 1 %).
Staged as documented patch artifacts (not wired into any serve script; the
engine does not yet boot with them) — scripts/patches/glm53-compile-0{1,2,3}*.patch:
-01 models/glm5next/common/model.py decorate Glm5NextModel (R4D_GLM_COMPILE)
-02 model_executor/kernels/r4d_lib.py freeze 10 capability probes that do
os.path.exists/.so lookups inside the traced forward. functools.lru_cache
does NOT protect them: dynamo re-traces the wrapped body even when warm
(verified in-image); a plain module-global read compiles.
-03 compilation/wrapper.py make fullgraph=True opt-out, because
suppress_errors does NOT cover graph breaks and this trunk can never
satisfy fullgraph (indexer metadata syncs to host — .item() at
glm5next/amd/sparse_indexer.py:263,419).
With all three the trunk traces (Dynamo bytecode transform time 1.08 s) and dies
on KeyError: in_proj_qkvbfg_a in the mHC/KDA path — a latent bug the compile
exposes, and the remaining blocker.
Also records two structural constraints: no new HIP kernels are possible (r4d.so
ships without sources), which binary-blocks the fused router at GLM's 288
experts; and DFlash2 for GLM is a TARGET-side port, not a drafter port
(incoai/GLM-5.3-Flash-DFlash2 is architectures ["DFlash2DraftModel"] +
model_type qwen3, already mapped at registry.py:636; glm5next lacks only the
aux-hidden-state plumbing deepseek_v2.py demonstrates).
Runbook frontmatter, intro, mermaid and wall section, AGENTS.md and the work log
corrected to match the measurement. tests/run.sh: 362 seams pass.
353 lines
20 KiB
Markdown
353 lines
20 KiB
Markdown
---
|
|
type: runbook
|
|
title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
|
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
|
|
tags:
|
|
- dankserver
|
|
- r9700
|
|
- gfx1201
|
|
- glm5.3
|
|
- w4a16
|
|
- int4
|
|
- expert-offload
|
|
- expert-cache
|
|
- autoround
|
|
- vllm
|
|
timestamp: 2026-09-30
|
|
---
|
|
|
|
# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
|
|
|
|
> **Question this page answers.** Can the fleet serve the 320B-class
|
|
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
|
|
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
|
|
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
|
|
> reachable** on this payload as the fork ships it. The bound is **not** the
|
|
> model, the quant, the link, or the expert gather (measured: 1.05 % of device
|
|
> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
|
|
> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
|
|
> because the fork compiles its own Qwen trunk and not its GLM one. See
|
|
> [The measured profile](#the-measured-profile-and-the-real-wall).
|
|
|
|
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
|
|
hardware — its conclusion was drawn from throughput arithmetic and is superseded
|
|
by this page's kernel-time measurement),
|
|
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
|
|
[[VLLM Index]], [[LLM/index|LLM fleet]].
|
|
|
|
## The deployed shape
|
|
|
|
| term | value |
|
|
| --- | --- |
|
|
| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
|
|
| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
|
|
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
|
|
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
|
|
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
|
|
| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
|
|
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
|
|
|
|
```mermaid
|
|
graph LR
|
|
A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
|
|
B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
|
|
C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
|
|
D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
|
|
D --> F[73.64 GiB pinned host rows / rank]
|
|
E --> G[decode step]
|
|
F -->|host misses<br/>1.05% of device time| G
|
|
```
|
|
|
|
## Why this payload and not another
|
|
|
|
The W4A16 fast path on this fork is reachable **only** through the INC
|
|
(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
|
|
for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
|
|
their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
|
|
offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
|
|
cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
|
|
|
|
The chain that makes Intel's checkpoint work, with the exact evidence:
|
|
|
|
```
|
|
[expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round"
|
|
[inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
|
|
[r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
|
|
[expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
|
|
[expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
|
|
host 73.64 GiB pinned, host rows prepacked on the GPU
|
|
```
|
|
|
|
`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
|
|
format-agnostic but requires the experts class to expose the hot/cold split.
|
|
|
|
## The numbers, and the wall
|
|
|
|
Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
|
|
|
|
| depth | prefill | decode |
|
|
| --- | --- | --- |
|
|
| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
|
|
| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
|
|
| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
|
|
|
|
Decode is flat across context and across every arena change — the signature of a
|
|
per-step cost that is not KV- or residency-driven at the margin.
|
|
|
|
### The measured profile, and the real wall
|
|
|
|
**This section replaces an earlier explanation, which the measurement falsified.**
|
|
The first reading of this page bounded the step with throughput arithmetic: 42 MoE
|
|
layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
|
|
missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
|
|
the gather is the wall". That arithmetic was circular — it *assumed* the gather
|
|
consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
|
|
|
|
A torch-profiler window (`--profiler-config`, `POST /start_profile` /
|
|
`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
|
|
device-activity time) gives the true split. Total device kernel time **46.18 s /
|
|
496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
|
|
the profiler attached):
|
|
|
|
| ms | % | launches | kernel |
|
|
| --- | --- | --- | --- |
|
|
| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
|
|
| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
|
|
| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
|
|
| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
|
|
| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
|
|
| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
|
|
| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
|
|
| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
|
|
| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
|
|
|
|
So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
|
|
LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
|
|
PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
|
|
conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
|
|
passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
|
|
token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
|
|
wall clock, so this is real device work, not launch-gap overhead.
|
|
|
|
Two consequences that kill three whole families of planned work:
|
|
|
|
- **Spec decode cannot help this shape, and the MTP measurement was not an
|
|
accident.** Pointwise cost scales with the token count, so a draft batch that
|
|
verifies 4 tokens pays ~4x the glue for the same per-token output — which is
|
|
exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
|
|
the gather was never the bound.
|
|
- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
|
|
(arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
|
|
|
|
## Fork work: the compile changelist (WIP, does not boot yet)
|
|
|
|
The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
|
|
carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
|
|
every boot:
|
|
|
|
```
|
|
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
|
|
provides neither a compiled submodule nor breakable CUDA graphs.
|
|
Overriding to FULL_DECODE_ONLY.
|
|
```
|
|
|
|
Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
|
|
Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
|
|
its own evidence header, base/patched sha256 and the next blocker:
|
|
|
|
| patch | target | fixes |
|
|
| --- | --- | --- |
|
|
| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
|
|
| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
|
|
| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
|
|
|
|
What the boots established, in order (each one ~7 min):
|
|
|
|
1. **01 alone dies** on the first cold probe call inside the traced region:
|
|
`Dynamo does not know how to trace the builtin 'posix.stat'` —
|
|
`r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
|
|
**`functools.lru_cache` does not protect these probes**: dynamo re-traces the
|
|
wrapped body to build its guard even when the cache is warm (it warns about
|
|
exactly this), verified in-image — a *warm* lru_cache-calling function still
|
|
fails `fullgraph=True`, while a plain module-global read compiles.
|
|
2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
|
|
four processes) and die on
|
|
`RuntimeError: Graph breaks are not supported with aot compile`, because this
|
|
build runs AOT compile by default and AOT refuses breaks.
|
|
3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
|
|
trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies in
|
|
`profile_run` with `KeyError: 'in_proj_qkvbfg_a'` at
|
|
`glm5next/common/model.py:456` (the KDA layer's merged projection,
|
|
`kda.py:217`) — a dict/registry lookup keyed by attribute name that the eager
|
|
path never reaches, i.e. **a latent bug the compile exposes, not a dynamo
|
|
limit**.
|
|
|
|
That last row is the state of the work: the remaining blocker is a real bug in the
|
|
fork's mHC/KDA wiring, and it must be fixed (plus whatever follows it) before the
|
|
changelist can boot. Two verified facts make the payoff concrete: `suppress_errors`
|
|
does **not** cover graph breaks (only `fullgraph=False` does), and the GLM trunk can
|
|
never satisfy `fullgraph` because its indexer metadata syncs to host
|
|
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`).
|
|
|
|
### Two structural constraints on any further fork work
|
|
|
|
- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
|
|
sources. The obvious second lever — the 9 % router — is gated
|
|
`num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
|
|
contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
|
|
experts, so it falls back to the generic sort-based top-k and **cannot be moved to
|
|
the fused router without rebuilding libr4d**. Only Python/config/inductor changes
|
|
(and runtime-JIT Triton) are available.
|
|
- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
|
|
is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
|
|
window; it only removes the only MLA decode path gfx12 has (trap 1 above).
|
|
|
|
### DFlash2 for GLM — what the drafter actually is
|
|
|
|
`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
|
|
port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
|
|
`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
|
|
head_dim 128, sliding attention (window 2048), `is_causal: false`,
|
|
`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
|
|
top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
|
|
`num_target_layers: 45`**.
|
|
|
|
The fork already maps that architecture:
|
|
`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
|
|
and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
|
|
**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
|
|
layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
|
|
`set_aux_hidden_state_layers()` / `get_eagle3_aux_hidden_state_layers()` plus
|
|
capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
|
|
implements neither.
|
|
|
|
Expected benefit is **not** assumed: DFlash2 amortizes a draft block (8 tokens) over
|
|
one verification pass, and per the profile above a verification pass costs pointwise
|
|
work proportional to tokens — the same reason MTP4 was flat. Measure it; do not
|
|
budget on it.
|
|
|
|
Note also that the KV-cap patch's predecessor arithmetic in
|
|
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
|
|
"gather-bound" conclusion came from the same throughput arithmetic that this page's
|
|
profiler window has now falsified.
|
|
|
|
## Measured arms (all 2026-09-30, one boot each)
|
|
|
|
| arm | decode | prefill | verdict |
|
|
| --- | --- | --- | --- |
|
|
| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
|
|
| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
|
|
| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
|
|
| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
|
|
|
|
### Why MTP is rejected here (it is not that it fails)
|
|
|
|
MTP **works** on GLM in this fork: the head loads, the architecture resolves to
|
|
`Glm5NextMTPModel`, and acceptance is excellent —
|
|
|
|
```
|
|
SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
|
|
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
|
|
Avg Draft acceptance rate: 69-97%
|
|
```
|
|
|
|
— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
|
|
prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
|
|
verification forward gathers the **union** of the batch's experts, so spec
|
|
decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
|
|
of arena. On a compute-bound (fully-resident) shape this would invert; here it
|
|
does not.
|
|
|
|
## Served surface (verified 2026-09-30 against the live engine)
|
|
|
|
| capability | state | evidence |
|
|
| --- | --- | --- |
|
|
| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
|
|
| completion | yes | `bench_glm.py` runs above |
|
|
| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
|
|
| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
|
|
| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
|
|
|
|
Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
|
|
requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
|
|
breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
|
|
newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
|
|
|
|
## Fork traps found (for the engine authors)
|
|
|
|
1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model.** The
|
|
`ROCM_AITER_MLA_SPARSE` backend's `_use_rocm_sparse_triton()` requires a
|
|
non-fp8 KV cache, and that rope-free Triton path is the **only** MLA decode
|
|
available on gfx12: aiter's MLA decode op is registered under
|
|
`@if_aiter_supported`, which is gated on **CDNA > 2**
|
|
(`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
|
|
the MTP draft step demands an op that cannot exist here:
|
|
`RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
|
|
attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
|
|
(`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
|
|
CDNA check, not the env var.)
|
|
2. **The plan's KV reserve under-models this model, and the cap enforces the
|
|
mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
|
|
carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
|
|
the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
|
|
`--max-model-len` becomes unbootable:
|
|
`ValueError: To serve at least one request with the model's max seq len
|
|
(131072), (2.07 GiB KV cache is needed, which is larger than the available KV
|
|
cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
|
|
tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
|
|
fixed) — where the reserve would eat a third of the arena.
|
|
`scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
|
|
request raise the cap; **it is not part of the deployed shape**, which boots
|
|
on the plan's own reserve.
|
|
3. **A window is forced, not chosen.** The production `max_model_len 131072` is
|
|
what the reserve model requires for this payload (below ~92k the boot
|
|
refuses); it is not a capacity decision.
|
|
4. **Expert prefetch remains poison** — unchanged from the qwen campaign
|
|
(36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
|
|
|
|
## Operations
|
|
|
|
```bash
|
|
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
|
|
# dirs are untouched on disk):
|
|
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
|
|
./scripts/deploy.sh dankserver --service frank-primary
|
|
|
|
# payload inventory (34 shards + 2 extras, weight-map index):
|
|
bash scripts/download-frank-primary.sh --verify-only
|
|
|
|
# live telemetry (needs the profiler window; with_stack=false is mandatory —
|
|
# with stack the stop path hangs and the trace buffer grows to tens of GB):
|
|
podman logs frank-primary-glm 2>&1 | grep "expert_lru.py"
|
|
|
|
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
|
|
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
|
|
```
|
|
|
|
Boot gate: the engine refuses a legal-looking shape in two measured ways — the
|
|
KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
|
|
64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
|
|
what killed the first two boots at util 0.95.
|
|
|
|
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
|
|
shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
|
|
the default allocator's fragmentation fails the profile/graph-capture phase
|
|
(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
|
|
MiB is free").
|
|
|
|
The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
|
|
walks every guarded vhost — including `http://192.168.0.112:8989/` on
|
|
FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
|
|
timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
|
|
healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
|
|
`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
|
|
and the WoL packet did not wake it (macOS "Wake for network access" needs
|
|
toggling on the mini itself).
|
|
|
|
## Rollback targets
|
|
|
|
| state | where |
|
|
| --- | --- |
|
|
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
|
|
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
|