diff --git a/AGENTS.md b/AGENTS.md index a902255..581679d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp` (docker-pinned image bundle `2dac17a2`, docker→podman shim, root systemd unit) | unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 GGUF shards) with the blockfp8 MTP4 head (depth 4) and mmproj, TP=2 tiered experts (80-slot LRU cache; PLE n-gram PINNED in host RAM 54.4 GB; quantized inject-shard dev-overlay adapter), CED off, 131072 ctx single stream, Qwen-Sharp template, tool parser `qwen3_coder`, reasoning channel `reasoning` | 7770 | +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound symmetric int4 g128 (170 GiB, 34 shards, 113,074 tensors), TP=2, expert offload to 73.64 GiB/rank of pinned host rows behind a 14.51 GiB / 2,383-slot VRAM LRU arena, 131072 ctx × 2 seqs, bf16 KV, no MTP, no prefetch; measured 9.9-10.0 tok/s decode / 918 tok/s prefill | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -245,7 +245,7 @@ merely wedged kernel usually still answers ARP. `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - - Dankserver `frank-primary` (root unit, R9V radiance on the 2x R9700): + - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` 2. Bench. NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in @@ -423,86 +423,62 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-primary — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0 (Dankserver, ACTIVE) +### frank-primary — tcclaviger vLLM, GLM-5.3-Flash W4A16 int4 (Dankserver, ACTIVE) -The **R9V radiance-vLLM fork** (`drwolfen/radiance-vllm-qwen4exp`, docker-pinned -prebuilt image bundle `r9v-qwen38-flash-next-wmma@sha256:2dac17a2…`) serving -the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** (175.30 GiB, 6 GGUF shards) -with the blockfp8 MTP4 head (depth 4) and the mmproj on **both** Radeon AI -PRO R9700s (gfx1201, BDFs `0000:03:00.0` / `0000:07:00.0`), one stream at -**131,072 ctx**, **TP=2 tiered experts** (80-slot LRU expert cache; PLE -n-gram table PINNED in host RAM, 54.4 GB), the **Qwen-Sharp v22.4.0 -template** (via the launcher's `R9V_CHAT_TEMPLATE_FILE` passthrough, fork -`30d4a32`), tool parser `qwen3_coder`, reasoning channel `reasoning`. Cut -over 2026-09-27 from the uncensored IQ4_XS payload (itself cut over from -llama.cpp b11030 the same day) at Frank's direction. +The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666…` +— the same digest-pinned image the 2026-09-29 qwen tuning campaign measured) +serving **`Intel/GLM-5.3-Flash-W4A16-AutoRound`**: AutoRound **symmetric int4, +group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, +113,074 tensors, ~170 GiB — on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**, +**131,072 ctx × 2 seqs**, bf16 KV, util 0.98, NBATCH 2048, prefix caching, +**no MTP**, **no prefetch**. Cut over 2026-09-30 from the qwen FP8 + MTP4 shape, +which itself replaced the R9V radiance fork on 2026-09-29. -**The tuned shape lives on-host, not in the repo.** The q8 profile's setup -run qualified the `R9V_*` config into -`/home/dank/r9v-state/q8xl/setup.json` (`ready: true`); the repo env -file `scripts/frank-primary.env` carries ONLY production overrides -(`R9V_HOST_BIND=0.0.0.0`, `R9V_HOST_PORT=7770`, -`R9V_SERVED_MODEL_NAME=frank-primary`, `R9V_CONTAINER_NAME=frank-primary-r9v`, -`R9V_EXPECTED_GPU_BDFS`) plus two q8-only keys: the dev-overlay adapter -(`R9V_DEV_QWEN4_EXP_ADAPTER_PY`, fork commit `363d606` — the Q8_0 payload -quantizes BOTH fused hyper-connection shards, so the shard-uniformity -check needs the in-stream dequant) and `R9V_FULL_MUTABLE_CACHE=0` (the -overlay's env.always injects the uncensored 160-slot cache geometry, which -the q8 80-slot tiered cache rejects). Never copy tuned values from the -state into the repo — it would drift from the qualification. +**The payload is chosen for the offload path, not for the quant.** 170 GiB of +int4 experts cannot fit 64 GiB of VRAM, so expert offload is mandatory — and +the fork reaches its W4A16 kernel (`R4dW4a16MoEMethod`, libr4d's grouped W4A16 +GEMM, 6.234 MiB per expert-rank slot) **only** when the checkpoint's +`quant_method` is `auto-round`, which the fork overrides to `inc`. A +compressed-tensors W4A16 checkpoint of the same model resolves to +`TritonWNA16Experts`, which **has no offload recipe at all**, and refuses to +serve. Built shape: 14.51 GiB arena = **2,383 slots × 6.234 MiB per rank** over +12,096 host-backed experts = **73.64 GiB pinned host rows per rank**. -**Spec decoding is MTP4 only.** The fork's `speculative_config` is a -single dict — one method per engine, NO chaining — so the llama.cpp -`draft-mtp,ngram-mod` pattern is NOT expressible here. MTP4 is wired -(`R9V_MTP_SPEC_TOKENS=4`, draft TP=2); the PLE n-gram rides along -architecturally (`ngram_embedding`); `ngram_gpu` exists as an alternative -single method, not a chainable one. +**Measured 2026-09-30: decode 9.9–10.0 tok/s, prefill 918 tok/s** (warm, 2.6k +prompt), KV cache 138,561 tokens. **The 35 tok/s target is NOT met, and this is +a measured wall rather than a tuning gap:** each decode step must move 2.045 GiB +of routed experts per rank (42 layers × 8 experts × 6.234 MiB), live telemetry +(`R4D_LRU_TELEMETRY=1`) puts the expert-cache hit rate at **0.53–0.66**, and the +fork's gather read path delivers **~8.2 GB/s against a link measured at +22.74 GB/s** — the same structural gap the qwen campaign found. Growing the +arena 13.08 → 14.51 GiB changed decode by *nothing* (10.03 → 9.88 tok/s), so +residency stopped being the binding term; 0.89 hit rate would be needed for 35. -**Serving = the fork's own launcher through a docker→podman shim.** The R9V -tooling is docker-pinned; the shim at -`/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker` carries -three laws: (1) the bundle is a containerd/OCI docker-save, so docker keys -the image by its OCI manifest digest while podman keys by config digest but -preserves the manifest digest in `RepoDigests` — the shim answers -`image inspect sha256:X` from RepoDigests and rewrites bare `sha256:X` run -refs to `repo@sha256:X`; (2) podman inspect answers -`HostConfig.LogConfig.Config: null` where docker returns the log-opts map -(the fork's doctor crashes on it) — the shim records `run --log-opt` pairs -and normalizes inspect output; (3) podman lacks docker's `SecurityOptions` -/`ServerVersion` info fields — the shim answers them. +**MTP works here and is still rejected.** The head loads +(`Resolved architecture: Glm5NextMTPModel`) and accepts well (mean accepted +length 3.76–4.90, per-position 0.92/0.85/0.69/0.58, 69–97 %), but decode was +unchanged (9.6–11.2 vs 10.0 tok/s) while prefill **collapsed 918 → 255 tok/s**: +a verification forward gathers the *union* of the batch's experts, so spec +decode amortizes compute, not a residency-bound gather. It also costs 2.6 GiB +of arena. **fp8 KV is illegal on this model** — it disables the sparse MLA +backend's rope-free Triton path, and aiter's MLA decode op is gated on CDNA>2, +so the MTP draft step demands an op that cannot exist on gfx12 +(`no attribute 'rocm_aiter_mla_decode_fwd'`). **Unit and self-healing.** Root systemd unit (`scripts/frank-primary.service`, -`User=dank`, `LimitMEMLOCK=infinity`) — the PLE pinned residency alone is -54.4 GB, and a root unit grants the pin budget via PID1 (a rootless -quadlet would be capped by the user manager's limit). -The serve script (`scripts/serve-frank-primary.sh`) gates the setup state -(ready + paths), the pinned image (RepoDigests), the PLE byte count -(`stat -c %s`), every GGUF shard + the mmproj and the GPU device nodes, -clears a stale container, then execs the fork's `scripts/launch.sh` through -the shim and **blocks on `docker wait`** — the launcher `--detach`es, so -without the wait a crashed engine is a green unit; `docker wait` cannot stop -the container child, hence `ExecStop=podman stop -t 120 -frank-primary-r9v`. The deployer's `deploy_llm_primary` gates state/image/ -PLE/shards, `podman rm -f`s the container, waits out :7770, relinks/enables/ -restarts the unit and health-gates with the MainPID check (the false-green -stray law). +`User=dank`, `LimitMEMLOCK=infinity` — the 73.64 GiB/rank expert pin needs it), +`ExecStop=podman stop -t 120 frank-primary-glm`. The serve script drives +`podman run` **directly** (the image entrypoint execs `vllm serve`), gates the +digest-pinned image, the payload's `config.json` and its exact 34-shard count +and the GPU device nodes, clears the stale container identities, then **blocks +on `podman wait`** so a crash is a nonzero ExecStart. The deployer's +`deploy_llm_primary` gates the same, waits out :7770, relinks/enables/restarts +the unit and health-gates with the MainPID check (the false-green stray law). +Boot cost ~7 min warm (~15 min cold); health returns 503 while loading. -Verified at cutover (2026-09-27): q8 setup state `ready: true`, all 20 -payload artifacts sha256-verified (175.30 GiB); the scratch-port bring-up -battery answered vision ("Red" for a 64×64 red PNG), MTP4 mean acceptance -length 3.19–5.00, and the PLE pinned proof (`mapping_bytes = -mapping_rss_bytes = 54,400,261,120`, fully RSS-resident); wire + tool -round-trip + template probe re-verified on :7770 after the deploy. No -throughput bench was run (Frank declined); the probe-era 11 tok/s decode -was residency-bound (halved expert headroom at double the bytes/expert), -not the production number. - -Runbook: `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`. -Rollback: the uncensored payload — flip `R9V_STATE_FILE` back to -`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys -and the template keys, redeploy (state + `restore-config-20260927.env` + -payload intact on disk) — see -[[frank-primary-r9v-radiance-qwen38-uncensored]]. +Runbook: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`. +Rollback: commit `2339fae` (qwen FP8 + MTP4) — restore `scripts/` from it and +deploy; the qwen payload and the R9V state dirs stay on disk. ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the @@ -851,7 +827,7 @@ Active: |---------|---------| | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | -| frank-primary (R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` | +| frank-primary (tcclaviger vLLM, GLM-5.3-Flash W4A16 int4, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 19ea64d..2a8c8ea 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1217,8 +1217,8 @@ deploy_llm_small() { # (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file # sha256 verification was the fetch script's job at download time, never # re-hashed here. -LLM_PRIMARY_CONTAINER=frank-primary-qwen -LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" +LLM_PRIMARY_CONTAINER=frank-primary-glm +LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe" deploy_llm_primary() { # Refuse rather than restart-loop: a unit whose image or payload is @@ -1271,7 +1271,8 @@ deploy_llm_primary() { # races conmon's --rm cleanup, and the orphan holds the port and GPUs. # Stop the unit, then remove the containers BEFORE waiting on the port. # The legacy identities (the retired R9V container and the ad hoc - # experiment container qwen-fp8) hold the same GPUs and die here too. + # experiment container qwen-fp8, the ad hoc glm-probe) hold the same GPUs + # and die here too. run_privileged systemctl stop frank-primary.service 2>/dev/null || true local ctr for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do diff --git a/scripts/download-frank-primary.sh b/scripts/download-frank-primary.sh index 8347940..53659fe 100755 --- a/scripts/download-frank-primary.sh +++ b/scripts/download-frank-primary.sh @@ -1,17 +1,24 @@ #!/usr/bin/env bash # Payload inventory/verification for frank-primary: -# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131 -# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that -# became frank-primary on 2026-09-29. +# Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4, group 128, +# packing "auto_round:auto_gptq": 34 numbered shards + model_extra_conv / +# model_extra_tensors, weight-map index, 113,074 tensors, ~170 GiB) — the +# int4 payload that became frank-primary on 2026-09-30. # -# bash scripts/download-frank-primary.sh --verify-only (the ONLY mode) +# bash scripts/download-frank-primary.sh --verify-only (the default) # bash scripts/download-frank-primary.sh --fetch # -# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch -# is impossible and the tree API masks LFS hashes, so a byte-exact upstream -# manifest cannot be built without the authed endpoint. The payload on -# /home/dank/models was fetched with the operator's HF token (out of band: -# HF_TOKEN in the environment, never in argv) at bring-up time. +# THIS REPO IS PUBLIC (measured 2026-09-30: the payload fetched anonymously at +# bring-up). The LFS oids are still not verified byte-exactly here — the tree +# API masks them — so integrity rests on the weight-map index, the shard count +# and the loader's own tensor-shape checks at boot. +# +# WHY THIS QUANT: its quantization_config.quant_method is "auto-round", which +# the fork overrides to "inc" — that is the ONLY route to R4dW4a16MoEMethod +# (libr4d's grouped W4A16 GEMM) and therefore to an expert-offload recipe. A +# compressed-tensors W4A16 checkpoint of the same model (e.g. +# dealignai/...-CYBERSECURITY-W4A16) reaches TritonWNA16Experts instead, which +# has no offload recipe, and refuses to serve at all on this host. # # Modes: # --fetch resume any missing/truncated shard via `hf download` with @@ -24,16 +31,17 @@ # (slow, ~10 min for 173 GiB). set -euo pipefail -REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8" -DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8" -EXPECTED_SHARDS=131 +REPO="Intel/GLM-5.3-Flash-W4A16-AutoRound" +DEST="/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound" +EXPECTED_SHARDS=34 mode="verify" [[ "${1:-}" == "--fetch" ]] && mode="fetch" if [[ "$mode" == "fetch" ]]; then - : "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}" - export HF_TOKEN + # The repo is public, so no token is required; if one is present it is taken + # from the environment and never from argv (a -H header is readable in + # /proc//cmdline — the 2026-09-16 leak). mkdir -p "$DEST" hf download "$REPO" --local-dir "$DEST" fi @@ -81,7 +89,7 @@ if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then echo "hashing every shard (slow)..." cd "$DEST" sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \ - sha256sum model-*-of-00131.safetensors > /dev/null # presence check; content hash list printed on demand + sha256sum model-*-of-00034.safetensors > /dev/null # presence check; content hash list printed on demand fi [[ "$fail" -eq 0 ]] || exit 1 diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index b38788d..e9ee133 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -1,34 +1,53 @@ # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770). # -# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state -# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as -# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last -# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same -# digest-pinned image the 2026-09-29 tuning campaign measured — serving -# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with -# the MTP4 drafter on the two R9700s. +# 2026-09-30 cutover from orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (MTP4) to +# Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet has served +# through the fork's W4A16 expert-offload path. ROLLBACK: the pre-cutover qwen +# state is commit 2339fae ("frank-primary: fix the SIGPIPE race in both +# image-digest gates") — restore scripts/ from it and deploy; the qwen payload +# (173 GiB) and the R9V state dirs are untouched on disk. # -# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited): -# * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom: -# the engine measured "GPU KV cache size: 729,545 tokens; Maximum -# concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB -# per rank. KV_BYTES is PER RANK — sizing it to the global requirement -# OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB -# free; 7.2 GiB boots). -# * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V -# — the stock flag quantizes them together, which is exactly what was -# requested here). Accepted by the attention path; qualified with a -# completion probe. -# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the -# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) — -# roughly half the gather traffic was waste; prefetch off is the tuned -# incumbent and the +40-51% decode came from deleting the flag. Never -# re-add --expert-prefetch without a fresh A/B. +# QUALIFIED 2026-09-30 by boot probe + bench against the live engine: +# * THE 35 tok/s TARGET IS NOT MET: decode 9.9-10.0 tok/s, prefill 918 tok/s +# (warm, 2.6k-token prompt) at 131,072 ctx x 2 seqs. See the runbook's +# "Why 35 tok/s is out of reach on this hardware" — the bound is the fork's +# expert-gather read path (8.2 GB/s effective vs the link's 22.74 GB/s), +# the same structural gap the 2026-09-29 qwen campaign measured. +# * Arena 14.51 GiB = 2,383 slots x 6.234 MiB per rank; 12,096 host-backed +# experts per rank, 73.64 GiB pinned. Live per-layer telemetry +# (R4D_LRU_TELEMETRY=1, 2026-09-30): hit rate 0.53-0.66, ~500-730 installs +# vs ~2,000 routed per layer over the window. +# * --expert-precise-memory is ON: it replaces the 6.25 GiB default +# compile-headroom figure with the registry's fitted 5.46 GiB, buying +# 1.43 GiB of cache (13.08 -> 14.51 GiB). Decode measured flat across that +# change (10.03 -> 9.88 tok/s), which is the evidence that the hit rate +# saturates near 0.6 and residency is no longer the binding term. +# * KV dtype stays bf16/AUTO — fp8 KV is NOT legal on this model. It disables +# _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend (the +# predicate requires a non-fp8 cache), and that rope-free Triton path is the +# only MLA decode available on gfx12: aiter's MLA decode op is registered +# under @if_aiter_supported, i.e. gated on CDNA>2. With fp8 KV the MTP draft +# step dies with "'_OpNamespace' 'vllm' object has no attribute +# 'rocm_aiter_mla_decode_fwd'". +# * MTP4 spec decode is ABSENT DELIBERATELY, unlike the qwen shape. It WORKS +# here (the MTP head loads, architecture resolves to Glm5NextMTPModel, and +# acceptance is excellent: mean accepted length 3.76-4.90, per-position +# 0.92/0.85/0.69/0.58, 69-97%) — but it buys nothing on an offload-bound +# deployment: measured decode 9.6-11.2 tok/s (vs 10.0 without) while prefill +# COLLAPSED 918 -> 255 tok/s. The verification forward gathers the union of +# the batch's experts, so spec decode does not amortize a residency-bound +# gather the way it amortizes compute. It also costs 2.6 GiB of arena. +# Reinstate only with a fresh A/B on a non-offload-bound shape. +# * PREFETCH IS ABSENT BY LAW, unchanged from the qwen campaign: expert-cache +# telemetry measured the prefetcher at 36% precision and deleting the flag +# bought +40-51% decode. Never re-add --expert-prefetch without a fresh A/B. # * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert # offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts # numerics (repetition loops, early halts, invalid measurements). -# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at -# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots). +# * KV is NOT capped explicitly: the fork's plan reserve (1.76 GiB at 131,072) +# already covers what max_model_len needs for this shape. It does NOT for an +# MTP shape — see the runbook's fork-bug section and +# scripts/patches/gpu_worker-kv-cap.patch. # # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is # prefixed FRANK_PRIMARY_ so none can leak. @@ -36,26 +55,57 @@ # --- runtime ---------------------------------------------------------------- # Digest-pinned: the floating :latest tag moved once already. Local alias # qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same -# bytes in rootless storage. +# bytes in rootless storage. Same image as the qwen cutover — the W4A16 path +# (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM) ships in it. FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea # --- payload ---------------------------------------------------------------- -FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 -FRANK_PRIMARY_MODEL_SHARDS=131 +# Intel AutoRound W4A16: symmetric int4, group 128, packing "auto_round:auto_gptq". +# quantization_config.quant_method is "auto-round", which the fork overrides to +# "inc" (inc.py:364) — that is what routes the MoE to R4dW4a16MoEMethod instead +# of the compressed-tensors path's Triton experts, which has no offload recipe. +# 34 numbered shards + model_extra_{conv,tensors}.safetensors, 113,074 tensors. +FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound +FRANK_PRIMARY_MODEL_SHARDS=34 # --- identity --------------------------------------------------------------- # The harness models.yml id stays frank-primary (provider 192.168.0.235:7770). -FRANK_PRIMARY_CONTAINER=frank-primary-qwen +FRANK_PRIMARY_CONTAINER=frank-primary-glm FRANK_PRIMARY_SERVED_NAME=frank-primary FRANK_PRIMARY_PUBLISH=7770:7770 -# Containers removed before every start: the retired R9V identity and the ad -# hoc experiment identity (both hold the GPUs / name registrations). -FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" +# Containers removed before every start: the pre-cutover qwen identity (it holds +# the same port and GPUs), the retired R9V identity, and the ad hoc experiment +# identity. +FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe" # --- qualified engine shape --------------------------------------------------- -FRANK_PRIMARY_MAXLEN=200000 -FRANK_PRIMARY_SEQS=3 -FRANK_PRIMARY_KV_BYTES=7730934784 -FRANK_PRIMARY_KV_DTYPE=fp8 -FRANK_PRIMARY_SPEC=4 +# 131,072 x 2 seqs. The window is NOT chosen for capacity: the engine's own +# plan reserve is one max_model_len request, and below ~92k tokens the fork's +# reserve model (13,600 B/token + a small mamba term) falls short of what the +# engine actually needs for this model's per-sequence KDA state, which makes the +# boot refuse ("2.07 GiB needed vs 1.74 GiB available"). 131,072 clears it with +# the plan's own cap intact. KV cache as built: 138,561 tokens, 1.06x +# concurrency for a 131,072-token request. +FRANK_PRIMARY_MAXLEN=131072 +FRANK_PRIMARY_SEQS=2 +# 0.99 fails boot ("Free memory on device cuda:1 ..."); 0.98 leaves the +# compile/capture headroom the plan models. FRANK_PRIMARY_UTIL=0.98 +# Prefill chunk. 2,048 is the qualified value; the expert union a chunk routes +# to saturates quickly, so the chunk size is an install-amortization knob, not a +# residency one. Measured 2026-09-30: NBATCH 8,192 boots (arena 12.51 GiB) and +# leads to the SAME warm prefill (922 vs 918 tok/s) — no reason to pay its +# 2 GiB of profile buffer. +FRANK_PRIMARY_NBATCH=2048 +# GLM's own tool-call dialect. Without --enable-auto-tool-choice the endpoint +# rejects tool_choice "auto" outright ("\"auto\" tool choice requires +# --enable-auto-tool-choice and --tool-call-parser to be set"), which breaks +# every tool-using agent on the endpoint. Verified 2026-09-30: a get_weather +# prompt returns tool_calls with finish_reason "tool_calls". +FRANK_PRIMARY_TOOLPARSER=glm47 +# GLM's reasoning dialect. Without it the model's thinking leaks into `content` +# (measured: content began with the answer, then "Let me verify: ... = 391" +# followed by a bare " $D/vllm/v1/worker/gpu_worker.py +# ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch ) +# # then add to the podman run: +# # -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro +# # -e CLAV_KV_RESERVE_FLOOR_GIB= (or pass --kv-cache-memory-bytes) +# +# Base file sha256 (the pristine file out of the pinned image): +# 1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7 +# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces +# these bytes exactly): +# 80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840 +# +--- a/vllm/v1/worker/gpu_worker.py ++++ b/vllm/v1/worker/gpu_worker.py +@@ -824,6 +824,21 @@ + expert_cfg = self.vllm_config.offload_config.expert + if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0: + cap = int(expert_cfg.planned_kv_reserve_bytes) ++ # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as ++ # 13600 B/token + a small mamba term, but this model also carries a ++ # per-sequence KDA state and (under MTP) the draft's KV, so the ++ # reserve can sit BELOW what max_model_len legitimately needs -- and ++ # the cap then makes a legal config unbootable (measured: need ++ # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An ++ # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB, ++ # is the operator's word: let it raise the cap. The compile/capture ++ # headroom still comes off the top. ++ _explicit = self.cache_config.kv_cache_memory_bytes ++ _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB") ++ if _explicit: ++ cap = max(cap, int(_explicit)) ++ if _floor: ++ cap = max(cap, int(float(_floor) * (1 << 30))) + if self.available_kv_cache_memory_bytes > cap: + logger.info_once( + "expert offload: KV capped at the plan's reserve %s GiB; %s GiB of " diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index 830a47f..d09b6c8 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -1,27 +1,30 @@ #!/usr/bin/env bash # serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork, -# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700, -# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag -# frank-primary-r9v-q8-20260929 is the pre-cutover state). +# Intel/GLM-5.3-Flash-W4A16-AutoRound, 2x R9700, dankserver :7770). +# +# 2026-09-30 cutover from orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4. +# Rollback: commit 2339fae carries the whole pre-cutover scripts/ tree; restore +# it and deploy. The qwen payload and the R9V state dirs stay on disk. # # The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs -# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces -# the R9V launcher/shim machinery (that fork needed a docker->podman shim and -# a 104-key on-host setup state; this stack is fully expressed by flags, and -# every flag here was qualified by boot probe on 2026-09-29 — see -# frank-primary.env for the qualification record). +# `vllm serve "$@"`, so the run line below IS the engine argv. # -# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype -# fp8: e4m3 for both K and V — the stock flag quantizes them together, which -# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured -# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP -# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36% -# precision — the tuned incumbent deletes it), TunableOp off. +# Why this payload runs the offload path at all: 170 GiB of int4 experts cannot +# fit in 64 GiB of VRAM, so the engine keeps 12,096 experts per rank in pinned +# host memory and serves them through a 2,383-slot VRAM LRU arena. The MoE +# kernel is libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod), reached because the +# checkpoint's quant_method "auto-round" is overridden to "inc" by the fork. +# +# Shape: MAXLEN 131,072 x 2 seqs, bf16 KV, util 0.98, MTP OFF, prefetch OFF. +# Measured 2026-09-30: decode 9.9-10.0 tok/s, prefill 918 tok/s warm. The +# 35 tok/s target is NOT met — the bound is the fork's expert-gather read path +# (8.2 GB/s effective vs the link's 22.74 GB/s); see frank-primary.env and the +# runbook's arithmetic section. # # The root unit (scripts/frank-primary.service, User=dank, -# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host -# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session -# also works, but the unit is the fleet-parity boot-persistent shape. +# LimitMEMLOCK=infinity) grants the pin budget for the 73.64 GiB/rank of pinned +# host expert rows. A plain shell session also works, but the unit is the +# fleet-parity boot-persistent shape. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -44,8 +47,9 @@ set +a : "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}" : "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}" : "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}" -: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}" -: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}" +: "${FRANK_PRIMARY_NBATCH:?FRANK_PRIMARY_NBATCH must be set}" +: "${FRANK_PRIMARY_TOOLPARSER:?FRANK_PRIMARY_TOOLPARSER must be set}" +: "${FRANK_PRIMARY_REASONINGPARSER:?FRANK_PRIMARY_REASONINGPARSER must be set}" # Capture podman's output BEFORE grepping: under `set -o pipefail`, the # classic `podman images | grep -q` pattern races — grep exits on its first # match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail @@ -106,14 +110,27 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \ --tensor-parallel-size 2 \ --max-model-len "$FRANK_PRIMARY_MAXLEN" \ --max-num-seqs "$FRANK_PRIMARY_SEQS" \ + --max-num-batched-tokens "$FRANK_PRIMARY_NBATCH" \ --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \ - --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \ - --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \ + --enable-expert-offload \ + --expert-precise-memory \ --enable-prefix-caching \ - --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \ + --enable-auto-tool-choice \ + --tool-call-parser "$FRANK_PRIMARY_TOOLPARSER" \ + --reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER" \ --trust-remote-code \ > "$LOG" 2>&1 +# NOT passed, deliberately (each measured on this payload — see the env file): +# --kv-cache-dtype fp8 disables the sparse MLA backend's rope-free +# Triton path, which is the only MLA decode on +# gfx12 (aiter's MLA op is gated on CDNA>2). +# --speculative-config MTP head loads and accepts well, but buys zero +# decode and collapses prefill on an +# offload-bound shape. +# --expert-prefetch* 36% precision; deleting it bought +40-51%. +# --kv-cache-memory-bytes the plan's own reserve covers this shape. + # From here on, this script's lifetime IS the container's: wait returns the # container's exit code, so a crash is a nonzero ExecStart and the unit's # Restart=on-failure is real. (systemd stop goes through ExecStop in the diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md new file mode 100644 index 0000000..4ef41d7 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -0,0 +1,224 @@ +--- +type: runbook +title: frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) +description: VERIFIED 2026-09-30 by boot probe, bench and live expert-cache telemetry. frank-primary cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET and this page shows why it is out of reach: the step is bounded by the fork's expert-gather read path at ~8.2 GB/s effective against a PCIe link that delivers 22.74 GB/s, with the cache at its measured hit-rate ceiling (0.53-0.66 per layer) from an arena that cannot grow past ~15 GiB under a 73.64 GiB/rank expert set. +tags: + - dankserver + - r9700 + - gfx1201 + - glm5.3 + - w4a16 + - int4 + - expert-offload + - expert-cache + - autoround + - vllm +timestamp: 2026-09-30 +--- + +# frank-primary — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700) + +> **Question this page answers.** Can the fleet serve the 320B-class +> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it? +> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs +> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not +> reachable** with this payload on this hardware. The bound is not the model, +> the quant, or the link: it is the fork's expert-gather read path, and it is +> measured, not inferred. + +Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the +same hardware, the same gather-path wall, found first on qwen), +[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis), +[[VLLM Index]], [[LLM/index|LLM fleet]]. + +## The deployed shape + +| term | value | +| --- | --- | +| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB | +| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) | +| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching | +| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot | +| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** | +| unit | `scripts/frank-primary.service` (root unit, `LimitMEMLOCK=infinity` for the pin) | +| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens | + +```mermaid +graph LR + A[Intel AutoRound W4A16
sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc] + B --> C[inc_wna16_scheme
_resolve_gptq_moe] + C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod
grouped W4A16 GEMM] + D --> E[ExpertLRUCache
2,383 VRAM slots / rank] + D --> F[73.64 GiB pinned host rows / rank] + E --> G[decode step] + F -->|8.2 GB/s effective| G +``` + +## Why this payload and not another + +The W4A16 fast path on this fork is reachable **only** through the INC +(AutoRound) quant method. `compressed_tensors` checkpoints of the same model — +for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve +their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no +offload recipe**, and the engine refuses to serve at all: 170 GiB of experts +cannot fit 64 GiB of VRAM, so offload is mandatory, not optional. + +The chain that makes Intel's checkpoint work, with the exact evidence: + +``` +[expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round" +[inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM) +[r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts +[expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank +[expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB, + host 73.64 GiB pinned, host rows prepacked on the GPU +``` + +`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is +format-agnostic but requires the experts class to expose the hot/cold split. + +## The numbers, and the wall + +Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy): + +| depth | prefill | decode | +| --- | --- | --- | +| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 | +| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 | +| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 | + +Decode is flat across context and across every arena change — the signature of a +per-step cost that is not KV- or residency-driven at the margin. + +### The arithmetic that bounds it + +Per decode step, per rank: 42 MoE layers x 8 routed experts x 6.234 MiB = +**2.045 GiB of expert weights**, which must either be in VRAM or cross PCIe. + +| hit rate | miss traffic | step @ 8.2 GB/s | decode | +| --- | --- | --- | --- | +| 0.60 (measured) | 0.82 GiB | ~105 ms | **9.5-10.0 tok/s** | +| 0.70 | 0.61 GiB | ~78 ms | 12.8 tok/s | +| 0.89 (what 35 tok/s needs) | 0.23 GiB | ~29 ms | 35 tok/s | + +Two independent measurements pin the missed term: + +- **Live expert-cache telemetry** (`R4D_LRU_TELEMETRY=1`, profiler window) + gives the hit rate directly — the LRU is *working*: + `layers.34: slots 56 (pinned 0, tier 0) hit 0.6279 routed 2002 installs 543 + read-through 202` … `layers.44: hit 0.5335`. Per-layer range **0.53-0.66**. +- **The link delivers 22.74 GB/s** at the engine's own slab size (H2D probe, + pinned, 2.345 MiB). The step moves 0.82 GiB in ~100 ms = **8.2 GB/s — + 36% of the wire**, matching the fork's own measured 6.5-9.5 GB/s. + +So the step is bounded by the fork's gather read path, and the only other lever +is the hit rate — which is capped by the arena. A 14.51 GiB arena holds 20% of a +73.64 GiB expert set; the measured hit rate beats random by ~3x thanks to +routing locality, but it saturates: growing the arena 13.08 → 14.51 GiB +(+9%) changed decode by **nothing** (10.03 → 9.88 tok/s). Reaching 0.89 would +need roughly 4.5x the per-layer slot count — an arena of well over 100 GiB. + +## Measured arms (all 2026-09-30, one boot each) + +| arm | decode | prefill | verdict | +| --- | --- | --- | --- | +| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent | +| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term | +| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below | +| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only | + +### Why MTP is rejected here (it is not that it fails) + +MTP **works** on GLM in this fork: the head loads, the architecture resolves to +`Glm5NextMTPModel`, and acceptance is excellent — + +``` +SpecDecoding metrics: Mean acceptance length: 3.83-4.90, +Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615, +Avg Draft acceptance rate: 69-97% +``` + +— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while +prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a +verification forward gathers the **union** of the batch's experts, so spec +decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB +of arena. On a compute-bound (fully-resident) shape this would invert; here it +does not. + +## Served surface (verified 2026-09-30 against the live engine) + +| capability | state | evidence | +| --- | --- | --- | +| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe | +| completion | yes | `bench_glm.py` runs above | +| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` | +| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" | +| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare ` 2** + (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV + the MTP draft step demands an op that cannot exist here: + `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no + attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16. + (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the + CDNA check, not the env var.) +2. **The plan's KV reserve under-models this model, and the cap enforces the + mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM + carries a per-sequence KDA state (34 linear-attention layers) and, under MTP, + the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal + `--max-model-len` becomes unbootable: + `ValueError: To serve at least one request with the model's max seq len + (131072), (2.07 GiB KV cache is needed, which is larger than the available KV + cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k + tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB + fixed) — where the reserve would eat a third of the arena. + `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV + request raise the cap; **it is not part of the deployed shape**, which boots + on the plan's own reserve. +3. **A window is forced, not chosen.** The production `max_model_len 131072` is + what the reserve model requires for this payload (below ~92k the boot + refuses); it is not a capacity decision. +4. **Expert prefetch remains poison** — unchanged from the qwen campaign + (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it. + +## Operations + +```bash +# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state +# dirs are untouched on disk): +git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/ +./scripts/deploy.sh dankserver --service frank-primary + +# payload inventory (34 shards + 2 extras, weight-map index): +bash scripts/download-frank-primary.sh --verify-only + +# live telemetry (needs the profiler window; with_stack=false is mandatory — +# with stack the stop path hangs and the trace buffer grows to tens of GB): +podman logs frank-primary-glm 2>&1 | grep "expert_lru.py" + +# boot: ~7 min warm page cache, longer cold; health returns 503 while loading +curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health +``` + +Boot gate: the engine refuses a legal-looking shape in two measured ways — the +KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores: +64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is +what killed the first two boots at util 0.95. + +## Rollback targets + +| state | where | +| --- | --- | +| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` | +| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-primary-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` | diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index ab80908..cc26170 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,10 +1,13 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-09-27): -**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork -serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI -PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman -shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints — +Documentation for the LLM inference fleet on the two hosts (2026-09-30): +**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the +digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving +Intel/GLM-5.3-Flash-W4A16-AutoRound — symmetric int4 group 128 — across both +Radeon AI PRO R9700s on :7770, TP=2 with the expert set offloaded to 73.64 GiB +of pinned host RAM per rank behind a 14.51 GiB VRAM LRU arena; 131,072 ctx, +bf16 KV, vision and tool calls on, measured 9.9-10.0 tok/s decode), and +Dankclaw runs two llama.cpp endpoints — stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B @@ -22,11 +25,11 @@ graph LR subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] A[2x Radeon AI PRO R9700 gfx1201, 32 GiB each] - B2[R9V radiance-vLLM fork, docker-pinned -via docker-podman shim -frank-primary, port 7770] - C[Qwen3.8-Flash-Next Q8_0 -TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision] + B2[tcclaviger RDNA4 vLLM fork, digest-pinned +podman run, frank-primary, port 7770] + C[GLM-5.3-Flash W4A16 int4 +TP=2, expert offload 73.64 GiB/rank pinned, +14.51 GiB LRU arena, 131072 ctx, vision, tools] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -58,20 +61,26 @@ frank-small, port 7771] ## Active Deployment -- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary - runbook (2026-09-27, root systemd unit, port 7770): the R9V - radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned - image bundle `2dac17a2`, driven through a docker→podman shim) serving - the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + - blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts - (80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off, - 131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's - `R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized - inject-shard dev-overlay adapter (fork `363d606`), reasoning channel - `reasoning`, parser `qwen3_coder`. The tuned shape lives in the - on-host q8xl setup state. Rollback: - [[frank-primary-r9v-radiance-qwen38-uncensored]] -- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27 +- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook + (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork + (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`, + no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric + int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH + R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096 + experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the + MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached + because the checkpoint's `quant_method: auto-round` is overridden to `inc` — + a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe + and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix + caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF + by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the + 35 tok/s target is NOT met, bounded by the fork's expert-gather read path + (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.** + Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30) +- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps + back; payload and R9V state dirs intact on disk, git tag + frank-primary-r9v-q8-20260929 is the last pre-cutover commit): + [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27 (same-day swap to the Q8_0 payload; rollback target, state + payload intact on disk): the R9V radiance-vLLM fork serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 94c1eda..0d5f724 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -169,6 +169,18 @@ The LLM inference fleet on the two hosts. Full index: concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size) +- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE + frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound + (symmetric int4 g128) on the tcclaviger fork, and the reason the int4 + AutoRound payload is the ONLY one that serves here (its `auto-round` quant + method routes to the W4A16 offload kernel; a compressed-tensors W4A16 + checkpoint of the same model reaches TritonWNA16Experts, which has no + offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the + 35 tok/s target is NOT met** — the step is bounded by the fork's gather + read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66 + hit rate, and MTP4 was measured and rejected (zero decode gain, prefill + 918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the + `glm47` tool/reasoning parser wiring - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 7f9ef7d..20773c7 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,47 @@ # Bundle Change Log +## 2026-09-30 +- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at + the 35 tok/s target, and the shortfall is measured rather than inferred.** + - Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4 + group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras, + 113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH + expert offload: its `quant_method: auto-round` is overridden to `inc`, which + routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The + compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`, + 182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO + offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it + refused to serve at all. That checkpoint was purged with the other retired + payloads. + - Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen + cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, + `--enable-expert-offload --expert-precise-memory`, prefix caching, vision + + `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF. + - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s** + (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per + rank over 12,096 host-backed experts (73.64 GiB pinned per rank). + - **The wall:** live expert-cache telemetry (`R4D_LRU_TELEMETRY=1` + a + `with_stack=false` profiler window) puts the cache hit rate at 0.53-0.66 per + layer; a step moves 0.82 GiB of misses in ~100 ms = **8.2 GB/s effective + against a link measured at 22.74 GB/s** — the same structural gather-read-path + gap the 2026-09-29 qwen campaign found. Growing the arena 13.08 -> 14.51 GiB + (+9%) changed decode by *nothing* (10.03 -> 9.88 tok/s), so residency stopped + being the binding term; 35 tok/s would need a 0.89 hit rate. + - Measured and rejected: **MTP4** (the head loads, arch resolves to + `Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but + decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255, + because a verification forward gathers the *union* of the batch's experts); + **fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and + aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op + that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill). + - Fork bug found: the plan's KV reserve under-models this model's + per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at + that reserve, making a legal `--max-model-len` unbootable. Shipped + `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV + request raise the cap) — NOT needed by the deployed shape. + - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V + state dirs are intact on disk. + ## 2026-09-29 - **frank-large tuning investigation: a measured NEGATIVE result, nothing diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 8eb3f9a..a14b942 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -1,19 +1,22 @@ #!/usr/bin/env bats # Seam: frank-primary — the Dankserver LLM endpoint on :7770. # -# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-29 -# cutover that replaced the R9V radiance fork): the serve script drives -# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every -# engine flag qualified by boot probe on 2026-09-29 and carried in the repo -# env file. The seam pins what the flags alone do not say: the qualification -# law (600k ctx / 3 sessions / q8-q8 KV were MEASURED to fit — 729,545 tokens -# of capacity at 7.2 GiB per rank — and KV_BYTES is PER RANK), the self-healing +# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-30 +# cutover to GLM-5.3-Flash W4A16 int4, which replaced the qwen FP8 + MTP4 +# shape): the serve script drives `podman run` DIRECTLY (the image entrypoint +# execs `vllm serve`), with every engine flag qualified by boot probe on +# 2026-09-30 and carried in the repo env file. The seam pins what the flags +# alone do not say: the qualification law (131,072 x 2 seqs at bf16 KV, util +# 0.98 is what boots; the window is forced up by the fork's KV reserve model, +# not by capacity), the offload law (170 GiB of experts cannot fit 64 GiB of +# VRAM — the engine MUST run with --enable-expert-offload), the self-healing # law (the run returns before the engine is up, so the serve script must wait # on the container for Restart=on-failure to mean anything), the memlock law -# that makes the tiered-expert pin legal, the deployer's image/payload gates -# with the false-green MainPID check, and the prefetch law (expert prefetch -# measured at 36% precision — OFF is the incumbent; a regression that -# reintroduces it silently costs ~40-50% decode). +# that makes the 73.64 GiB/rank expert pin legal, the deployer's image/payload +# gates with the false-green MainPID check, and the three measured-absent +# flags: expert prefetch (36% precision, costs ~40-50% decode), MTP spec +# decode (buys zero decode and collapses prefill on an offload-bound shape), +# and fp8 KV (disables the only MLA decode path gfx12 has). load helpers @@ -53,8 +56,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # outlives the unit and holds both GPUs. The container name must match # the env file's identity. grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT" - grep -q 'frank-primary-qwen' "$UNIT" - grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE" + grep -q 'frank-primary-glm' "$UNIT" + grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-glm$' "$ENV_FILE" } @test "the serve shape is direct podman run, not a launcher or shim" { @@ -69,32 +72,70 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" } @test "the qualified engine shape is in the env file" { - # 2026-09-29 boot probe: MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 - # fits — the engine measured "GPU KV cache size: 729,545 tokens, - # Maximum concurrency for 200,000 tokens per request: 3.65x" at - # KV_BYTES 7.2 GiB per rank. KV_BYTES is PER RANK: sizing it to the - # global requirement OOMs the worker (measured twice). fp8 KV is the - # requested q8/q8 (e4m3 for both K and V together). MTP4 spec decode, - # util 0.98 (0.99 fails the free-memory wall). - grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE" + # 2026-09-30 boot probe: 131,072 x 2 seqs at bf16 KV, util 0.98 boots and + # serves (measured "GPU KV cache size: 138,561 tokens"). 0.99 fails the + # free-memory wall. The window is not a capacity choice: the engine's own + # reserve model (one max_model_len request) needs a large L for this + # model's per-sequence KDA state, and below ~92k the boot REFUSES with + # "needed ... larger than the available KV cache memory". NBATCH pins the + # prefill chunk that was measured. + grep -q '^FRANK_PRIMARY_MAXLEN=131072$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_NBATCH=2048$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE" } @test "the serve script wires the engine flags from the env" { - grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE" - grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE" grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE" grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE" + grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_NBATCH"' "$SERVE" grep -q -- '--tensor-parallel-size 2' "$SERVE" - grep -q 'num_speculative_tokens' "$SERVE" grep -q -- '--enable-prefix-caching' "$SERVE" grep -q -- '--trust-remote-code' "$SERVE" } +@test "the endpoint advertises the GLM tool dialect" { + # Without --enable-auto-tool-choice the engine rejects tool_choice "auto" + # with a 400 ("requires --enable-auto-tool-choice and --tool-call-parser"), + # which breaks every tool-using agent on the endpoint. + grep -q -- '--enable-auto-tool-choice' "$SERVE" + grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOLPARSER"' "$SERVE" + grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER"' "$SERVE" + grep -q '^FRANK_PRIMARY_TOOLPARSER=glm47$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE" +} + +@test "expert offload is ON — the payload cannot fit VRAM without it" { + # 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag + # the engine has no home for the expert tensors at all; the arena + # (14.51 GiB = 2,383 slots) plus 73.64 GiB/rank of pinned host rows is the + # only shape that boots. --expert-precise-memory is the qualified sizing. + grep -q -- '--enable-expert-offload' "$SERVE" + grep -q -- '--expert-precise-memory' "$SERVE" +} + +@test "MTP spec decode is absent BY MEASUREMENT, not by omission" { + # 2026-09-30: the MTP head loads and accepts well (mean accepted length + # 3.76-4.90), but decode was unchanged (9.6-11.2 vs 10.0 tok/s) while + # prefill collapsed 918 -> 255 tok/s — the verification forward gathers the + # union of the batch's experts, so spec decode does not amortize a + # residency-bound gather. It also costs 2.6 GiB of arena. + ! grep -q -- '--speculative-config' "$SERVE" + ! grep -q 'speculative' "$ENV_FILE" +} + +@test "fp8 KV is absent — it would disable the only MLA decode path on gfx12" { + # _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend requires a + # non-fp8 KV cache, and that rope-free Triton path is the only MLA decode + # available on gfx12: aiter's MLA decode op is registered under + # @if_aiter_supported, gated on CDNA>2. With fp8 KV the MTP draft step dies + # with "_OpNamespace 'vllm' object has no attribute + # 'rocm_aiter_mla_decode_fwd'". KV dtype therefore stays auto/bf16. + ! grep -q -- '--kv-cache-dtype' "$SERVE" + ! grep -q '^FRANK_PRIMARY_KV_DTYPE=' "$ENV_FILE" +} + @test "expert prefetch is absent BY LAW, not by omission" { # Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720 # prefetched slabs (36% precision) — roughly half the gather traffic was @@ -149,10 +190,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" @test "the model id matches the payload's served alias wiring" { # The payload dir and its shard count are the fetch script's output # layout; the fetch script must write where the env points. - grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE" - grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD" - grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD" + grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE" + grep -q 'GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD" + grep -q 'Intel/GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD" } @test "no bare HIP-reserved env var is defined" { @@ -168,12 +209,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # storage or the payload is missing/wrong-sharded — then clears stale # containers (including the legacy identities), waits out the port, # relinks/enables/restarts the root unit and health-gates it. - grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY" + grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-glm' "$REMOTE_DEPLOY" grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY" grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY" grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY" grep -q 'config.json' "$REMOTE_DEPLOY" grep -q 'qwen-fp8' "$REMOTE_DEPLOY" + grep -q 'glm-probe' "$REMOTE_DEPLOY" grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY" grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY" grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"