frank-primary: qwen cutover docs + deployer identity + seam re-pin
- AGENTS.md: fleet table + frank-primary section back to qwen FP8 + MTP4 (37.4-40.6 tok/s, 54% acceptance); GLM runbook moved to the retired index. - runbook: RETIRED verdict — DFlash2 drafter served and drafted but acceptance pinned 1.00-1.38 on every text class including verbatim repetition (mis-wired aux capture, not a weak drafter); 5x slower than baseline at ~1.2 acceptance. - log.md: 2026-10-01 rollback entry. - deploy-remote.sh: LLM_PRIMARY_CONTAINER back to frank-primary-qwen; glm-probe joins the legacy sweep list. - frank_primary_unit.bats: re-pinned to the qwen shape — KV qualification law (7.2 GiB/rank, 3.65x concurrency; 9/16 GiB OOM), fp8 q8/q8 KV PRESENT, MTP4 PRESENT (3.1-3.4x over the 12.11 no-spec baseline), no tool dialect, automatic offload. The old 'MTP buys zero' law carried the prefetch tax and is retired.
This commit is contained in:
Frank committed
1 parent
d91379cff2
commit
26f6f82c98
5 files changed
+150
-188
No files matched your search
@@ -1,22 +1,26 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
|
||||
#
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-30
|
||||
# cutover to GLM-5.3-Flash W4A16 int4, which replaced the qwen FP8 + MTP4
|
||||
# shape): the serve script drives `podman run` DIRECTLY (the image entrypoint
|
||||
# execs `vllm serve`), with every engine flag qualified by boot probe on
|
||||
# 2026-09-30 and carried in the repo env file. The seam pins what the flags
|
||||
# alone do not say: the qualification law (131,072 x 2 seqs at bf16 KV, util
|
||||
# 0.98 is what boots; the window is forced up by the fork's KV reserve model,
|
||||
# not by capacity), the offload law (170 GiB of experts cannot fit 64 GiB of
|
||||
# VRAM — the engine MUST run with --enable-expert-offload), the self-healing
|
||||
# law (the run returns before the engine is up, so the serve script must wait
|
||||
# on the container for Restart=on-failure to mean anything), the memlock law
|
||||
# that makes the 73.64 GiB/rank expert pin legal, the deployer's image/payload
|
||||
# gates with the false-green MainPID check, and the three measured-absent
|
||||
# flags: expert prefetch (36% precision, costs ~40-50% decode), MTP spec
|
||||
# decode (buys zero decode and collapses prefill on an offload-bound shape),
|
||||
# and fp8 KV (disables the only MLA decode path gfx12 has).
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
|
||||
# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which
|
||||
# reversed the failed GLM/DFlash2 experiment): the serve script drives
|
||||
# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
|
||||
# engine flag qualified by boot probe and carried in the repo env file. The
|
||||
# seam pins what the flags alone do not say: the KV qualification law
|
||||
# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB
|
||||
# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV
|
||||
# qualification (e4m3 for BOTH K and V — the stock flag quantizes them
|
||||
# together), the automatic-offload law (the fork plans the tiered-expert
|
||||
# offload itself; no flag exists to set), the self-healing law (the run
|
||||
# returns before the engine is up, so the serve script must wait on the
|
||||
# container for Restart=on-failure to mean anything), the memlock law that
|
||||
# makes the pinned host rows legal, the deployer's image/payload gates with
|
||||
# the false-green MainPID check, and the measured laws: expert prefetch
|
||||
# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved
|
||||
# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance
|
||||
# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading
|
||||
# carried the prefetch tax), and NO tool dialect (the GLM era added
|
||||
# --enable-auto-tool-choice; the 2339fae trio does not serve tools).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -56,8 +60,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# outlives the unit and holds both GPUs. The container name must match
|
||||
# the env file's identity.
|
||||
grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
|
||||
grep -q 'frank-primary-glm' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-glm$' "$ENV_FILE"
|
||||
grep -q 'frank-primary-qwen' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve shape is direct podman run, not a launcher or shim" {
|
||||
@@ -72,77 +76,75 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-09-30 boot probe: 131,072 x 2 seqs at bf16 KV, util 0.98 boots and
|
||||
# serves (measured "GPU KV cache size: 138,561 tokens"). 0.99 fails the
|
||||
# free-memory wall. The window is not a capacity choice: the engine's own
|
||||
# reserve model (one max_model_len request) needs a large L for this
|
||||
# model's per-sequence KDA state, and below ~92k the boot REFUSES with
|
||||
# "needed ... larger than the available KV cache memory". NBATCH pins the
|
||||
# prefill chunk that was measured.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=131072$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
|
||||
# 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES
|
||||
# 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and
|
||||
# reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is
|
||||
# PER RANK — sizing it to the global requirement OOMs the worker
|
||||
# (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4
|
||||
# full-length 200k contexts do NOT fit: four clients share the ~730k
|
||||
# token pool and each runs below 200k when all are deep.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_NBATCH=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
|
||||
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_NBATCH"' "$SERVE"
|
||||
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE"
|
||||
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
||||
grep -q -- '--enable-prefix-caching' "$SERVE"
|
||||
grep -q -- '--trust-remote-code' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the endpoint advertises the GLM tool dialect" {
|
||||
# Without --enable-auto-tool-choice the engine rejects tool_choice "auto"
|
||||
# with a 400 ("requires --enable-auto-tool-choice and --tool-call-parser"),
|
||||
# which breaks every tool-using agent on the endpoint.
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOLPARSER"' "$SERVE"
|
||||
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONINGPARSER"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_TOOLPARSER=glm47$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_REASONINGPARSER=glm47$' "$ENV_FILE"
|
||||
@test "the endpoint serves NO tool dialect — tool agents route elsewhere" {
|
||||
# The 2339fae trio does NOT wire --enable-auto-tool-choice /
|
||||
# --tool-call-parser (the GLM era added glm47 dialect flags; the rollback
|
||||
# dropped them). tool_choice "auto" gets a 400 here — tool-using agents
|
||||
# must route at frank-small or frank-large until a parser is qualified
|
||||
# for this arm (a behavior change beyond the rollback, not a silent fix).
|
||||
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
! grep -q -- '--tool-call-parser' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expandable segments is ON — the cache arena leaves no slack" {
|
||||
# The expert-offload plan fills VRAM to the modelled headroom, so the
|
||||
# profile/graph-capture phase has ~0.2 GiB to work with and the default
|
||||
# allocator's fragmentation fails it: measured 2026-09-30 on the deployed
|
||||
# unit, "Tried to allocate 662.00 MiB ... 240.00 MiB is free" from the
|
||||
# engine core while the identical shape booted with this env set.
|
||||
grep -q 'PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True' "$SERVE"
|
||||
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
|
||||
# ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
|
||||
# expert planner moves the overflow to pinned host rows by itself (the
|
||||
# arena 6,574 slots x 2.344 MiB plus host rows). There is no
|
||||
# --enable-expert-offload flag in this arm; the legality of the pin is
|
||||
# the unit's unbounded LimitMEMLOCK.
|
||||
! grep -q -- '--enable-expert-offload' "$SERVE"
|
||||
! grep -q -- '--expert-precise-memory' "$SERVE"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expert offload is ON — the payload cannot fit VRAM without it" {
|
||||
# 170 GiB of int4 experts against 64 GiB of VRAM. Without the offload flag
|
||||
# the engine has no home for the expert tensors at all; the arena
|
||||
# (14.51 GiB = 2,383 slots) plus 73.64 GiB/rank of pinned host rows is the
|
||||
# only shape that boots. --expert-precise-memory is the qualified sizing.
|
||||
grep -q -- '--enable-expert-offload' "$SERVE"
|
||||
grep -q -- '--expert-precise-memory' "$SERVE"
|
||||
@test "MTP4 spec decode is PRESENT BY MEASUREMENT" {
|
||||
# 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57
|
||||
# tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec
|
||||
# baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured
|
||||
# 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses
|
||||
# (17.4% acceptance). The earlier "MTP buys zero decode" reading carried
|
||||
# the prefetch tax and is retired with it. ngram-mod never booted on this
|
||||
# model (fork bug: PLE inputs prepared only on the MTP path).
|
||||
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
|
||||
grep -q -- '--speculative-config' "$SERVE"
|
||||
}
|
||||
|
||||
@test "MTP spec decode is absent BY MEASUREMENT, not by omission" {
|
||||
# 2026-09-30: the MTP head loads and accepts well (mean accepted length
|
||||
# 3.76-4.90), but decode was unchanged (9.6-11.2 vs 10.0 tok/s) while
|
||||
# prefill collapsed 918 -> 255 tok/s — the verification forward gathers the
|
||||
# union of the batch's experts, so spec decode does not amortize a
|
||||
# residency-bound gather. It also costs 2.6 GiB of arena.
|
||||
! grep -q -- '--speculative-config' "$SERVE"
|
||||
! grep -q 'speculative' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "fp8 KV is absent — it would disable the only MLA decode path on gfx12" {
|
||||
# _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend requires a
|
||||
# non-fp8 KV cache, and that rope-free Triton path is the only MLA decode
|
||||
# available on gfx12: aiter's MLA decode op is registered under
|
||||
# @if_aiter_supported, gated on CDNA>2. With fp8 KV the MTP draft step dies
|
||||
# with "_OpNamespace 'vllm' object has no attribute
|
||||
# 'rocm_aiter_mla_decode_fwd'". KV dtype therefore stays auto/bf16.
|
||||
! grep -q -- '--kv-cache-dtype' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_KV_DTYPE=' "$ENV_FILE"
|
||||
@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" {
|
||||
# --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes
|
||||
# them together; a V-only quant needs a fork patch). The GLM-era MLA
|
||||
# concern (sparse-triton MLA decode path refusing fp8 KV) does not apply
|
||||
# to this arm: qwen's attention path accepted the dtype and a completion
|
||||
# probe qualified it 2026-09-29.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "expert prefetch is absent BY LAW, not by omission" {
|
||||
@@ -199,10 +201,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
@test "the model id matches the payload's served alias wiring" {
|
||||
# The payload dir and its shard count are the fetch script's output
|
||||
# layout; the fetch script must write where the env points.
|
||||
grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=34$' "$ENV_FILE"
|
||||
grep -q 'GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD"
|
||||
grep -q 'Intel/GLM-5.3-Flash-W4A16-AutoRound' "$DOWNLOAD"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
|
||||
grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
|
||||
}
|
||||
|
||||
@test "no bare HIP-reserved env var is defined" {
|
||||
@@ -218,7 +220,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# storage or the payload is missing/wrong-sharded — then clears stale
|
||||
# containers (including the legacy identities), waits out the port,
|
||||
# relinks/enables/restarts the root unit and health-gates it.
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-glm' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
|
||||
grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
|
||||
|
||||
Reference in new issue
Block a user