frank-primary cutover: GLM-5.3-Flash W4A16 int4 (Intel AutoRound)
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
This commit is contained in:
Frank
@@ -1217,8 +1217,8 @@ deploy_llm_small() {
|
||||
# (FRANK_PRIMARY_MODEL_SHARDS), and the unit env must be complete. Per-file
|
||||
# sha256 verification was the fetch script's job at download time, never
|
||||
# re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-qwen
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
LLM_PRIMARY_CONTAINER=frank-primary-glm
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
|
||||
deploy_llm_primary() {
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
@@ -1271,7 +1271,8 @@ deploy_llm_primary() {
|
||||
# races conmon's --rm cleanup, and the orphan holds the port and GPUs.
|
||||
# Stop the unit, then remove the containers BEFORE waiting on the port.
|
||||
# The legacy identities (the retired R9V container and the ad hoc
|
||||
# experiment container qwen-fp8) hold the same GPUs and die here too.
|
||||
# experiment container qwen-fp8, the ad hoc glm-probe) hold the same GPUs
|
||||
# and die here too.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
local ctr
|
||||
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
|
||||
|
||||
@@ -1,17 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Payload inventory/verification for frank-primary:
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
|
||||
# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
|
||||
# became frank-primary on 2026-09-29.
|
||||
# Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4, group 128,
|
||||
# packing "auto_round:auto_gptq": 34 numbered shards + model_extra_conv /
|
||||
# model_extra_tensors, weight-map index, 113,074 tensors, ~170 GiB) — the
|
||||
# int4 payload that became frank-primary on 2026-09-30.
|
||||
#
|
||||
# bash scripts/download-frank-primary.sh --verify-only (the ONLY mode)
|
||||
# bash scripts/download-frank-primary.sh --verify-only (the default)
|
||||
# bash scripts/download-frank-primary.sh --fetch
|
||||
#
|
||||
# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
|
||||
# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
|
||||
# manifest cannot be built without the authed endpoint. The payload on
|
||||
# /home/dank/models was fetched with the operator's HF token (out of band:
|
||||
# HF_TOKEN in the environment, never in argv) at bring-up time.
|
||||
# THIS REPO IS PUBLIC (measured 2026-09-30: the payload fetched anonymously at
|
||||
# bring-up). The LFS oids are still not verified byte-exactly here — the tree
|
||||
# API masks them — so integrity rests on the weight-map index, the shard count
|
||||
# and the loader's own tensor-shape checks at boot.
|
||||
#
|
||||
# WHY THIS QUANT: its quantization_config.quant_method is "auto-round", which
|
||||
# the fork overrides to "inc" — that is the ONLY route to R4dW4a16MoEMethod
|
||||
# (libr4d's grouped W4A16 GEMM) and therefore to an expert-offload recipe. A
|
||||
# compressed-tensors W4A16 checkpoint of the same model (e.g.
|
||||
# dealignai/...-CYBERSECURITY-W4A16) reaches TritonWNA16Experts instead, which
|
||||
# has no offload recipe, and refuses to serve at all on this host.
|
||||
#
|
||||
# Modes:
|
||||
# --fetch resume any missing/truncated shard via `hf download` with
|
||||
@@ -24,16 +31,17 @@
|
||||
# (slow, ~10 min for 173 GiB).
|
||||
set -euo pipefail
|
||||
|
||||
REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
|
||||
DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
|
||||
EXPECTED_SHARDS=131
|
||||
REPO="Intel/GLM-5.3-Flash-W4A16-AutoRound"
|
||||
DEST="/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound"
|
||||
EXPECTED_SHARDS=34
|
||||
|
||||
mode="verify"
|
||||
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||
|
||||
if [[ "$mode" == "fetch" ]]; then
|
||||
: "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
|
||||
export HF_TOKEN
|
||||
# The repo is public, so no token is required; if one is present it is taken
|
||||
# from the environment and never from argv (a -H header is readable in
|
||||
# /proc/<pid>/cmdline — the 2026-09-16 leak).
|
||||
mkdir -p "$DEST"
|
||||
hf download "$REPO" --local-dir "$DEST"
|
||||
fi
|
||||
@@ -81,7 +89,7 @@ if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
|
||||
echo "hashing every shard (slow)..."
|
||||
cd "$DEST"
|
||||
sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
|
||||
sha256sum model-*-of-00131.safetensors > /dev/null # presence check; content hash list printed on demand
|
||||
sha256sum model-*-of-00034.safetensors > /dev/null # presence check; content hash list printed on demand
|
||||
fi
|
||||
|
||||
[[ "$fail" -eq 0 ]] || exit 1
|
||||
|
||||
+87
-37
@@ -1,34 +1,53 @@
|
||||
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
||||
#
|
||||
# 2026-09-29 cutover from the R9V radiance-vLLM fork (its launcher/shim/state
|
||||
# machinery is retired; /home/dank/r9v-state/{q8xl,uncensored} stay on disk as
|
||||
# the rollback target, and git tag frank-primary-r9v-q8-20260929 is the last
|
||||
# pre-cutover commit). The engine is the tcclaviger RDNA4 vLLM fork — the same
|
||||
# digest-pinned image the 2026-09-29 tuning campaign measured — serving
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards) with
|
||||
# the MTP4 drafter on the two R9700s.
|
||||
# 2026-09-30 cutover from orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (MTP4) to
|
||||
# Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet has served
|
||||
# through the fork's W4A16 expert-offload path. ROLLBACK: the pre-cutover qwen
|
||||
# state is commit 2339fae ("frank-primary: fix the SIGPIPE race in both
|
||||
# image-digest gates") — restore scripts/ from it and deploy; the qwen payload
|
||||
# (173 GiB) and the R9V state dirs are untouched on disk.
|
||||
#
|
||||
# Every number here was QUALIFIED by boot probe on 2026-09-29 (not inherited):
|
||||
# * MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8 fits with headroom:
|
||||
# the engine measured "GPU KV cache size: 729,545 tokens; Maximum
|
||||
# concurrency for 200,000 tokens per request: 3.65x" at KV_BYTES 7.2 GiB
|
||||
# per rank. KV_BYTES is PER RANK — sizing it to the global requirement
|
||||
# OOMs the worker (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB
|
||||
# free; 7.2 GiB boots).
|
||||
# * --kv-cache-dtype fp8 is the fleet's q8/q8 KV ask (e4m3 for BOTH K and V
|
||||
# — the stock flag quantizes them together, which is exactly what was
|
||||
# requested here). Accepted by the attention path; qualified with a
|
||||
# completion probe.
|
||||
# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
|
||||
# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
|
||||
# roughly half the gather traffic was waste; prefetch off is the tuned
|
||||
# incumbent and the +40-51% decode came from deleting the flag. Never
|
||||
# re-add --expert-prefetch without a fresh A/B.
|
||||
# QUALIFIED 2026-09-30 by boot probe + bench against the live engine:
|
||||
# * THE 35 tok/s TARGET IS NOT MET: decode 9.9-10.0 tok/s, prefill 918 tok/s
|
||||
# (warm, 2.6k-token prompt) at 131,072 ctx x 2 seqs. See the runbook's
|
||||
# "Why 35 tok/s is out of reach on this hardware" — the bound is the fork's
|
||||
# expert-gather read path (8.2 GB/s effective vs the link's 22.74 GB/s),
|
||||
# the same structural gap the 2026-09-29 qwen campaign measured.
|
||||
# * Arena 14.51 GiB = 2,383 slots x 6.234 MiB per rank; 12,096 host-backed
|
||||
# experts per rank, 73.64 GiB pinned. Live per-layer telemetry
|
||||
# (R4D_LRU_TELEMETRY=1, 2026-09-30): hit rate 0.53-0.66, ~500-730 installs
|
||||
# vs ~2,000 routed per layer over the window.
|
||||
# * --expert-precise-memory is ON: it replaces the 6.25 GiB default
|
||||
# compile-headroom figure with the registry's fitted 5.46 GiB, buying
|
||||
# 1.43 GiB of cache (13.08 -> 14.51 GiB). Decode measured flat across that
|
||||
# change (10.03 -> 9.88 tok/s), which is the evidence that the hit rate
|
||||
# saturates near 0.6 and residency is no longer the binding term.
|
||||
# * KV dtype stays bf16/AUTO — fp8 KV is NOT legal on this model. It disables
|
||||
# _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend (the
|
||||
# predicate requires a non-fp8 cache), and that rope-free Triton path is the
|
||||
# only MLA decode available on gfx12: aiter's MLA decode op is registered
|
||||
# under @if_aiter_supported, i.e. gated on CDNA>2. With fp8 KV the MTP draft
|
||||
# step dies with "'_OpNamespace' 'vllm' object has no attribute
|
||||
# 'rocm_aiter_mla_decode_fwd'".
|
||||
# * MTP4 spec decode is ABSENT DELIBERATELY, unlike the qwen shape. It WORKS
|
||||
# here (the MTP head loads, architecture resolves to Glm5NextMTPModel, and
|
||||
# acceptance is excellent: mean accepted length 3.76-4.90, per-position
|
||||
# 0.92/0.85/0.69/0.58, 69-97%) — but it buys nothing on an offload-bound
|
||||
# deployment: measured decode 9.6-11.2 tok/s (vs 10.0 without) while prefill
|
||||
# COLLAPSED 918 -> 255 tok/s. The verification forward gathers the union of
|
||||
# the batch's experts, so spec decode does not amortize a residency-bound
|
||||
# gather the way it amortizes compute. It also costs 2.6 GiB of arena.
|
||||
# Reinstate only with a fresh A/B on a non-offload-bound shape.
|
||||
# * PREFETCH IS ABSENT BY LAW, unchanged from the qwen campaign: expert-cache
|
||||
# telemetry measured the prefetcher at 36% precision and deleting the flag
|
||||
# bought +40-51% decode. Never re-add --expert-prefetch without a fresh A/B.
|
||||
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
|
||||
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
|
||||
# numerics (repetition loops, early halts, invalid measurements).
|
||||
# Decode at the tuned shape: 40.57/37.44/37.57 tok/s (166/1087/4064 ctx) at
|
||||
# the 32k window; 42.94/34.55/36.09 at this 200k shape (arena 6,032 slots).
|
||||
# * KV is NOT capped explicitly: the fork's plan reserve (1.76 GiB at 131,072)
|
||||
# already covers what max_model_len needs for this shape. It does NOT for an
|
||||
# MTP shape — see the runbook's fork-bug section and
|
||||
# scripts/patches/gpu_worker-kv-cap.patch.
|
||||
#
|
||||
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
||||
# prefixed FRANK_PRIMARY_ so none can leak.
|
||||
@@ -36,26 +55,57 @@
|
||||
# --- runtime ----------------------------------------------------------------
|
||||
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
||||
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
||||
# bytes in rootless storage.
|
||||
# bytes in rootless storage. Same image as the qwen cutover — the W4A16 path
|
||||
# (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM) ships in it.
|
||||
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
||||
|
||||
# --- payload ----------------------------------------------------------------
|
||||
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
FRANK_PRIMARY_MODEL_SHARDS=131
|
||||
# Intel AutoRound W4A16: symmetric int4, group 128, packing "auto_round:auto_gptq".
|
||||
# quantization_config.quant_method is "auto-round", which the fork overrides to
|
||||
# "inc" (inc.py:364) — that is what routes the MoE to R4dW4a16MoEMethod instead
|
||||
# of the compressed-tensors path's Triton experts, which has no offload recipe.
|
||||
# 34 numbered shards + model_extra_{conv,tensors}.safetensors, 113,074 tensors.
|
||||
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound
|
||||
FRANK_PRIMARY_MODEL_SHARDS=34
|
||||
|
||||
# --- identity ---------------------------------------------------------------
|
||||
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
||||
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
||||
FRANK_PRIMARY_CONTAINER=frank-primary-glm
|
||||
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
||||
FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# Containers removed before every start: the retired R9V identity and the ad
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# Containers removed before every start: the pre-cutover qwen identity (it holds
|
||||
# the same port and GPUs), the retired R9V identity, and the ad hoc experiment
|
||||
# identity.
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
|
||||
# --- qualified engine shape ---------------------------------------------------
|
||||
FRANK_PRIMARY_MAXLEN=200000
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
FRANK_PRIMARY_KV_BYTES=7730934784
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_SPEC=4
|
||||
# 131,072 x 2 seqs. The window is NOT chosen for capacity: the engine's own
|
||||
# plan reserve is one max_model_len request, and below ~92k tokens the fork's
|
||||
# reserve model (13,600 B/token + a small mamba term) falls short of what the
|
||||
# engine actually needs for this model's per-sequence KDA state, which makes the
|
||||
# boot refuse ("2.07 GiB needed vs 1.74 GiB available"). 131,072 clears it with
|
||||
# the plan's own cap intact. KV cache as built: 138,561 tokens, 1.06x
|
||||
# concurrency for a 131,072-token request.
|
||||
FRANK_PRIMARY_MAXLEN=131072
|
||||
FRANK_PRIMARY_SEQS=2
|
||||
# 0.99 fails boot ("Free memory on device cuda:1 ..."); 0.98 leaves the
|
||||
# compile/capture headroom the plan models.
|
||||
FRANK_PRIMARY_UTIL=0.98
|
||||
# Prefill chunk. 2,048 is the qualified value; the expert union a chunk routes
|
||||
# to saturates quickly, so the chunk size is an install-amortization knob, not a
|
||||
# residency one. Measured 2026-09-30: NBATCH 8,192 boots (arena 12.51 GiB) and
|
||||
# leads to the SAME warm prefill (922 vs 918 tok/s) — no reason to pay its
|
||||
# 2 GiB of profile buffer.
|
||||
FRANK_PRIMARY_NBATCH=2048
|
||||
# GLM's own tool-call dialect. Without --enable-auto-tool-choice the endpoint
|
||||
# rejects tool_choice "auto" outright ("\"auto\" tool choice requires
|
||||
# --enable-auto-tool-choice and --tool-call-parser to be set"), which breaks
|
||||
# every tool-using agent on the endpoint. Verified 2026-09-30: a get_weather
|
||||
# prompt returns tool_calls with finish_reason "tool_calls".
|
||||
FRANK_PRIMARY_TOOLPARSER=glm47
|
||||
# GLM's reasoning dialect. Without it the model's thinking leaks into `content`
|
||||
# (measured: content began with the answer, then "Let me verify: ... = 391"
|
||||
# followed by a bare "</think") and the reasoning field comes back empty. With
|
||||
# it, the chain of thought lands in `reasoning` — the vLLM field name the fleet
|
||||
# documents (llama.cpp endpoints use reasoning_content).
|
||||
FRANK_PRIMARY_REASONINGPARSER=glm47
|
||||
|
||||
@@ -1,26 +1,24 @@
|
||||
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
|
||||
# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
|
||||
# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
|
||||
# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
|
||||
# launcher/shim/setup-state machinery is retired and the state dirs stay on
|
||||
# disk as rollback).
|
||||
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint
|
||||
# (Intel/GLM-5.3-Flash-W4A16-AutoRound int4, 131,072 x 2 seqs, bf16 KV, TP=2,
|
||||
# expert offload to pinned host RAM, 34-shard payload) on the 2x R9700 pair,
|
||||
# dankserver :7770. 2026-09-30 cutover from orcarouter
|
||||
# Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4; commit 2339fae is the pre-cutover
|
||||
# state and the rollback.
|
||||
#
|
||||
# Same port, same served alias; the harness models.yml id is unchanged.
|
||||
# The serve script drives `podman run` directly (the image entrypoint execs
|
||||
# `vllm serve`) and then waits on the container, so a crashed engine is a
|
||||
# nonzero ExecStart and Restart=on-failure is real.
|
||||
#
|
||||
# WHY A ROOT UNIT (not a rootless quadlet): the tiered-expert shape pins tens
|
||||
# of GiB of host RAM (the expert host slabs, ~20.5 GiB per rank of pinned
|
||||
# anonymous memory). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
|
||||
# WHY A ROOT UNIT (not a rootless quadlet): the expert-offload shape pins
|
||||
# 73.64 GiB per rank of host RAM (the expert host rows). Pinned pages are charged against RLIMIT_MEMLOCK; dank's
|
||||
# login session on this host carries 8 MiB and a user quadlet is capped by the
|
||||
# user manager's own hard limit — raising that means restarting
|
||||
# user@1000.service and bouncing every user container on the host. A root unit
|
||||
# grants LimitMEMLOCK=infinity to the dank process via PID1, and the image
|
||||
# stays in dank's rootless podman storage.
|
||||
[Unit]
|
||||
Description=frank-primary — tcclaviger vLLM Qwen3.8-Flash-Next-Uncensored-FP8 on 2x R9700
|
||||
Description=frank-primary — tcclaviger vLLM GLM-5.3-Flash int4 W4A16 on 2x R9700
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -35,9 +33,9 @@ Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-pr
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
||||
# systemd stop: wait alone cannot stop the container child; stop it
|
||||
# explicitly. vLLM 0.26 drains gracefully on SIGTERM.
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary-qwen
|
||||
# Boot cost: FP8 payload page-in plus torch compile/graph capture —
|
||||
# ~15 min cold (measured 2026-09-29), faster warm.
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary-glm
|
||||
# Boot cost: int4 payload page-in plus torch compile/graph capture —
|
||||
# ~7 min measured 2026-09-30 on a warm page cache, longer cold.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
|
||||
#
|
||||
# STATUS: NOT part of the deployed frank-primary shape. It exists because the
|
||||
# fork's plan under-models this model's KV, and it is the enabler for a future
|
||||
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
|
||||
# plan's own reserve).
|
||||
#
|
||||
# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
|
||||
# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
|
||||
# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
|
||||
# cache at that reserve, so the plan's own arithmetic can make a legal
|
||||
# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
|
||||
# - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB -> boots
|
||||
# - MTP4, max_model_len 131072: reserve 1.74 GiB < need 2.07 GiB -> refuses
|
||||
# - MTP4, max_model_len 262144: reserve 3.43 GiB < need 3.62 GiB -> refuses
|
||||
# ("To serve at least one request with the model's max seq len ...")
|
||||
# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
|
||||
# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
|
||||
# where the KV reserve would eat a third of the expert arena.
|
||||
#
|
||||
# THE FIX: let an explicit operator KV request raise the cap. The compile and
|
||||
# graph-capture headroom still comes off the top, so the blast radius is one
|
||||
# flag the operator chose.
|
||||
#
|
||||
# APPLY (into a host dir that is then bind-mounted read-only over the image):
|
||||
# IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
||||
# D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
|
||||
# podman run --rm --entrypoint cat $IMAGE \
|
||||
# /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
|
||||
# > $D/vllm/v1/worker/gpu_worker.py
|
||||
# ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
|
||||
# # then add to the podman run:
|
||||
# # -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
|
||||
# # -e CLAV_KV_RESERVE_FLOOR_GIB=<needed> (or pass --kv-cache-memory-bytes)
|
||||
#
|
||||
# Base file sha256 (the pristine file out of the pinned image):
|
||||
# 1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
|
||||
# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
|
||||
# these bytes exactly):
|
||||
# 80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
|
||||
#
|
||||
--- a/vllm/v1/worker/gpu_worker.py
|
||||
+++ b/vllm/v1/worker/gpu_worker.py
|
||||
@@ -824,6 +824,21 @@
|
||||
expert_cfg = self.vllm_config.offload_config.expert
|
||||
if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
|
||||
cap = int(expert_cfg.planned_kv_reserve_bytes)
|
||||
+ # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
|
||||
+ # 13600 B/token + a small mamba term, but this model also carries a
|
||||
+ # per-sequence KDA state and (under MTP) the draft's KV, so the
|
||||
+ # reserve can sit BELOW what max_model_len legitimately needs -- and
|
||||
+ # the cap then makes a legal config unbootable (measured: need
|
||||
+ # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
|
||||
+ # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
|
||||
+ # is the operator's word: let it raise the cap. The compile/capture
|
||||
+ # headroom still comes off the top.
|
||||
+ _explicit = self.cache_config.kv_cache_memory_bytes
|
||||
+ _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
|
||||
+ if _explicit:
|
||||
+ cap = max(cap, int(_explicit))
|
||||
+ if _floor:
|
||||
+ cap = max(cap, int(float(_floor) * (1 << 30)))
|
||||
if self.available_kv_cache_memory_bytes > cap:
|
||||
logger.info_once(
|
||||
"expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "
|
||||
@@ -1,27 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
|
||||
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
|
||||
# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
|
||||
# frank-primary-r9v-q8-20260929 is the pre-cutover state).
|
||||
# Intel/GLM-5.3-Flash-W4A16-AutoRound, 2x R9700, dankserver :7770).
|
||||
#
|
||||
# 2026-09-30 cutover from orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4.
|
||||
# Rollback: commit 2339fae carries the whole pre-cutover scripts/ tree; restore
|
||||
# it and deploy. The qwen payload and the R9V state dirs stay on disk.
|
||||
#
|
||||
# The engine is driven by `podman run` DIRECTLY: the image's entrypoint execs
|
||||
# `vllm serve "$@"`, so the run line below IS the engine argv. This replaces
|
||||
# the R9V launcher/shim machinery (that fork needed a docker->podman shim and
|
||||
# a 104-key on-host setup state; this stack is fully expressed by flags, and
|
||||
# every flag here was qualified by boot probe on 2026-09-29 — see
|
||||
# frank-primary.env for the qualification record).
|
||||
# `vllm serve "$@"`, so the run line below IS the engine argv.
|
||||
#
|
||||
# Shape: MAXLEN 200000 x 3 sessions (600k total), KV q8/q8 (--kv-cache-dtype
|
||||
# fp8: e4m3 for both K and V — the stock flag quantizes them together, which
|
||||
# is the requested q8/q8), KV_BYTES 7.2 GiB PER RANK (the engine measured
|
||||
# 729,545 tokens of capacity / 3.65x concurrency at 200k per request), MTP
|
||||
# depth 4, prefix caching, expert prefetch OFF (telemetry measured it at 36%
|
||||
# precision — the tuned incumbent deletes it), TunableOp off.
|
||||
# Why this payload runs the offload path at all: 170 GiB of int4 experts cannot
|
||||
# fit in 64 GiB of VRAM, so the engine keeps 12,096 experts per rank in pinned
|
||||
# host memory and serves them through a 2,383-slot VRAM LRU arena. The MoE
|
||||
# kernel is libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod), reached because the
|
||||
# checkpoint's quant_method "auto-round" is overridden to "inc" by the fork.
|
||||
#
|
||||
# Shape: MAXLEN 131,072 x 2 seqs, bf16 KV, util 0.98, MTP OFF, prefetch OFF.
|
||||
# Measured 2026-09-30: decode 9.9-10.0 tok/s, prefill 918 tok/s warm. The
|
||||
# 35 tok/s target is NOT met — the bound is the fork's expert-gather read path
|
||||
# (8.2 GB/s effective vs the link's 22.74 GB/s); see frank-primary.env and the
|
||||
# runbook's arithmetic section.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host
|
||||
# slabs (~20.5 GiB/rank of pinned anonymous memory). A plain shell session
|
||||
# also works, but the unit is the fleet-parity boot-persistent shape.
|
||||
# LimitMEMLOCK=infinity) grants the pin budget for the 73.64 GiB/rank of pinned
|
||||
# host expert rows. A plain shell session also works, but the unit is the
|
||||
# fleet-parity boot-persistent shape.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -44,8 +47,9 @@ set +a
|
||||
: "${FRANK_PRIMARY_PUBLISH:?FRANK_PRIMARY_PUBLISH must be host:container}"
|
||||
: "${FRANK_PRIMARY_MAXLEN:?FRANK_PRIMARY_MAXLEN must be set}"
|
||||
: "${FRANK_PRIMARY_SEQS:?FRANK_PRIMARY_SEQS must be set}"
|
||||
: "${FRANK_PRIMARY_KV_BYTES:?FRANK_PRIMARY_KV_BYTES must be set (PER RANK)}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set (fp8 = q8/q8 KV)}"
|
||||
: "${FRANK_PRIMARY_NBATCH:?FRANK_PRIMARY_NBATCH must be set}"
|
||||
: "${FRANK_PRIMARY_TOOLPARSER:?FRANK_PRIMARY_TOOLPARSER must be set}"
|
||||
: "${FRANK_PRIMARY_REASONINGPARSER:?FRANK_PRIMARY_REASONINGPARSER must be set}"
|
||||
# Capture podman's output BEFORE grepping: under `set -o pipefail`, the
|
||||
# classic `podman images | grep -q` pattern races — grep exits on its first
|
||||
# match, podman dies SIGPIPE (141) writing the remaining lines, and pipefail
|
||||
@@ -106,14 +110,27 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
--max-num-batched-tokens "$FRANK_PRIMARY_NBATCH" \
|
||||
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
|
||||
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
--enable-expert-offload \
|
||||
--expert-precise-memory \
|
||||
--enable-prefix-caching \
|
||||
--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$FRANK_PRIMARY_SPEC}" \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser "$FRANK_PRIMARY_TOOLPARSER" \
|
||||
--reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER" \
|
||||
--trust-remote-code \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
# NOT passed, deliberately (each measured on this payload — see the env file):
|
||||
# --kv-cache-dtype fp8 disables the sparse MLA backend's rope-free
|
||||
# Triton path, which is the only MLA decode on
|
||||
# gfx12 (aiter's MLA op is gated on CDNA>2).
|
||||
# --speculative-config MTP head loads and accepts well, but buys zero
|
||||
# decode and collapses prefill on an
|
||||
# offload-bound shape.
|
||||
# --expert-prefetch* 36% precision; deleting it bought +40-51%.
|
||||
# --kv-cache-memory-bytes the plan's own reserve covers this shape.
|
||||
|
||||
# From here on, this script's lifetime IS the container's: wait returns the
|
||||
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
||||
# Restart=on-failure is real. (systemd stop goes through ExecStop in the
|
||||
|
||||
Reference in New Issue
Block a user