frank-primary: FreeToken (PR587 ROCm) cutover — trio, seam, deployer
Replaces the tcclaviger vLLM fork with FreeToken at PR #587 head 77a7201 built on dankserver into localhost/frank-primary-freetoken:pr587-77a7201 (scripts/docker/frank-primary-freetoken.Dockerfile: digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev symlink, curated --no-deps install per docs/install_amd.md). Shape (first-boot conservative): ONE R9700 (second card idle; --tensor-parallel-size exists for future 2-card tuning), KV pool --num-tokens 300000 (7.2 GiB at 24 KiB/token; clears the 177,356-token harness worst case and one native 262,144 conversation), MoE LRU --moe-cache-rate 0.10, --max-running-requests 2, --max-output-tokens 36864. Parsers AUTO: thinking -> reasoning_content natively (models.yml flips reasoningContentField on the harness side). Experts + PLE pin ~170 GiB host RAM — root unit keeps LimitMEMLOCK=infinity. Container identity renamed to the unit name (Frank's call): frank-primary; vLLM-era frank-primary-qwen joins the legacy cleanup list. Deployer image gate becomes podman image exists (a local build carries no RepoDigests; the old digest grep refused exactly this shape). Rollback tag: frank-primary-vllm-shapeG-20261003.
This commit is contained in:
Frank
+137
-318
@@ -1,26 +1,25 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
|
||||
#
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
|
||||
# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
|
||||
# script drives `podman run` DIRECTLY (the image entrypoint execs
|
||||
# `vllm serve`), with every engine flag qualified by boot probe and carried
|
||||
# in the repo env file. The seam pins what the flags alone do not say: the
|
||||
# KV qualification law (the night ladder: 262144/245760 OOM at graph
|
||||
# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
|
||||
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
|
||||
# clean cache + 262k windows), the automatic-offload law (the fork plans
|
||||
# the tiered-expert offload itself; no flag exists to set), the self-healing
|
||||
# law (the run returns before the engine is up, so the serve script must
|
||||
# wait on the container for Restart=on-failure to mean anything), the
|
||||
# memlock law that makes the pinned host rows legal, the deployer's
|
||||
# image/payload gates with the false-green MainPID check, and the measured
|
||||
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
|
||||
# no-spec family measured fastest once the prefetch tax was removed; MTP
|
||||
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
|
||||
# MNBT 2048, the CUDA-graph plan term pinned to the measured capture cost, and
|
||||
# the client contract (reasoning parser qwen3,
|
||||
# tool parser qwen3_xml, preserve_thinking on).
|
||||
# This file asserts the FreeToken (PR #587 ROCm) ROOT-UNIT shape (the
|
||||
# 2026-10-05 cutover from the tcclaviger RDNA4 vLLM fork, git tag
|
||||
# frank-primary-vllm-shapeG-20261003): the serve script drives `podman run`
|
||||
# DIRECTLY (the image carries `ft serve`, built on the host from
|
||||
# scripts/docker/frank-primary-freetoken.Dockerfile), one R9700, with every
|
||||
# engine value carried in the repo env file. The seam pins what flags alone
|
||||
# do not say: the KV-pool math (300k tokens at 24 KiB/token clears the
|
||||
# harness worst case of 177,356), the deployed single-GPU shape
|
||||
# (--tensor-parallel-size EXISTS in FreeToken — a 2-card TP shape is future
|
||||
# tuning; HIP_VISIBLE_DEVICES pins THIS shape to one card), the host-RAM
|
||||
# law (experts + PLE pin ~170 GiB — the memlock law that makes the pin
|
||||
# legal), the self-healing law (the run returns before the engine is up, so
|
||||
# the serve script must wait on the container for Restart=always to mean
|
||||
# anything), the image-existence gate (a local build carries no
|
||||
# RepoDigests — the old digest grep refused exactly this shape; pinning
|
||||
# lives in the Dockerfile), the deployer's payload gates with the
|
||||
# false-green MainPID check, and the client contract (parsers AUTO:
|
||||
# reasoning_content emission is the engine's native behavior — models.yml
|
||||
# maps reasoningContentField on the harness side, not here).
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -29,29 +28,30 @@ SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
DOCKERFILE="$REPO_ROOT/scripts/docker/frank-primary-freetoken.Dockerfile"
|
||||
|
||||
@test "frank-primary unit, serve script and env exist in scripts/" {
|
||||
@test "frank-primary unit, serve script, env and Dockerfile exist" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$DOCKERFILE" ]
|
||||
}
|
||||
|
||||
@test "frank-primary is boot-persistent, self-healing and unprivileged" {
|
||||
@test "frank-primary unit is boot-persistent, self-healing and unprivileged" {
|
||||
# Root unit parity with frank-large/frank-small: enable is the [Install]
|
||||
# section, the restart policy survives a crash, and the tiered-expert
|
||||
# pin needs an unbounded memlock (a root unit grants it via PID1; the
|
||||
# dank session's own hard limit is 8 MiB and cannot be raised without
|
||||
# restarting user@1000.service).
|
||||
# section, the restart policy survives a crash, and the pinned host RAM
|
||||
# (expert rows + PLE table) needs an unbounded memlock (a root unit
|
||||
# grants it via PID1; the dank session's own hard limit is 8 MiB).
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
# Restart=always, not on-failure: a runtime worker OOM makes vLLM's API
|
||||
# server exit CLEAN, so ExecStart returns 0 and on-failure never fires
|
||||
# (measured 2026-10-01 — the endpoint stayed down until a human noticed).
|
||||
# Restart=always, not on-failure: the vLLM-era lesson carries over — a
|
||||
# clean front-end exit after a worker fault reads as success to systemd,
|
||||
# so on-failure never fires.
|
||||
grep -q '^Restart=always$' "$UNIT"
|
||||
! grep -q '^Restart=on-failure$' "$UNIT"
|
||||
# Exiting the engine takes as long as vLLM's drain: ExecStop runs
|
||||
# `podman stop -t 120`, so the stop timeout MUST exceed 120 s or systemd
|
||||
# kills the stop and records 'Failed with result timeout' (measured
|
||||
# 2026-10-01). Guard the relationship, not just the literal.
|
||||
# ExecStop runs `podman stop -t 120`, so the stop timeout MUST exceed
|
||||
# 120 s or systemd kills the stop and records 'Failed with result
|
||||
# timeout' (measured 2026-10-01). Guard the relationship, not the
|
||||
# literal.
|
||||
stop_t=$(awk -F= '/^TimeoutStopSec=/{print $2}' "$UNIT")
|
||||
[ -n "$stop_t" ] && [ "$stop_t" -gt 120 ]
|
||||
grep -q '^ExecStop=.*podman stop -t 120' "$UNIT"
|
||||
@@ -69,320 +69,117 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q 'FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the unit stops the container child explicitly" {
|
||||
@test "the unit stops the container child explicitly — container identity is the unit name" {
|
||||
# The serve script blocks on podman wait, which cannot stop the
|
||||
# container child; systemd stop goes through ExecStop or the engine
|
||||
# outlives the unit and holds both GPUs. The container name must match
|
||||
# the env file's identity.
|
||||
# outlives the unit and holds the GPU. Frank's 2026-10-05 call: the
|
||||
# container name IS the service name (frank-primary), and the vLLM-era
|
||||
# name frank-primary-qwen moved into the legacy cleanup list.
|
||||
grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
|
||||
grep -q 'frank-primary-qwen' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
|
||||
grep -q 'podman stop -t 120 frank-primary$' "$UNIT"
|
||||
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary$' "$ENV_FILE"
|
||||
grep -q 'frank-primary-qwen' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve shape is direct podman run, not a launcher or shim" {
|
||||
# The tcclaviger stack is fully expressed by flags; the R9V launcher,
|
||||
# docker->podman shim and setup-state layering are the RETIRED shape.
|
||||
@test "the serve shape is direct podman run, not a launcher, shim, or the vLLM fork" {
|
||||
grep -q 'podman run' "$SERVE"
|
||||
! grep -q 'launch.sh' "$SERVE"
|
||||
! grep -q 'podman-shim' "$SERVE"
|
||||
! grep -q 'R9V_STATE_FILE' "$SERVE"
|
||||
! grep -q -- '--model-draft' "$SERVE"
|
||||
! grep -q -- '--spec-type' "$SERVE"
|
||||
# The tcclaviger vLLM fork is RETIRED — its image name and its
|
||||
# fork-specific flags must not survive in the FreeToken run line
|
||||
# (TP=2 is gone with the fork; FreeToken serves ONE GPU).
|
||||
! grep -q 'tcclaviger' "$SERVE"
|
||||
! grep -q 'tcclaviger' "$ENV_FILE"
|
||||
! grep -q -- '--tensor-parallel-size' "$SERVE"
|
||||
! grep -q -- '--kv-cache-dtype' "$SERVE"
|
||||
! grep -q -- '--max-num-seqs' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-10-02 shape D. History: the 2026-10-01 night ladder put SEQS=3
|
||||
# bf16 at 241664 (KV 732,359 / 3.03x); SEQS=6 fp8 does NOT fit (6 x
|
||||
# 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144
|
||||
# BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token
|
||||
# request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes
|
||||
# free`. UTIL is a REAL arena lever (the fork plans budget = total x util)
|
||||
# but a weak one, and Frank caps it at 0.982 — never 1.00 (see
|
||||
# FRANK_PRIMARY_UTIL for the two ceiling deaths that justify the margin).
|
||||
# SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by
|
||||
# single-variable boot probe against the then-deployed shape: SEQS 2 fails
|
||||
# at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError
|
||||
# (query_start_loc.size()[0] marked dynamic but specialized to a constant
|
||||
# (3)); SEQS 3 passes at both. model_runner sizes that buffer
|
||||
# max_num_reqs + 1, so the width is SEQS + 1 and MNBT only moves the
|
||||
# compile RANGE.
|
||||
# So the shape is 3 slots with a pool that admits only ~2 full-length
|
||||
# conversations: measured live, KV_BYTES 3.15e9/rank reports 398,658
|
||||
# tokens = 1.99x for a 200704-token request (7,892 B/token at THIS shape;
|
||||
# the per-token price is shape-dependent, 7,701 at 262144). Shorter
|
||||
# requests use the spare blocks; a third LONG request queues.
|
||||
# The arena is sized by max_model_len, NOT by KV_BYTES: the planner
|
||||
# reserves registry 9190 B/token x max_model_len (2.48 GiB at 262144 ->
|
||||
# arena 6,248 slots; 1.91 GiB at 200704 -> 6,496). The pool cut buys
|
||||
# runtime headroom, not arena.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
|
||||
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
|
||||
# (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
|
||||
# the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
|
||||
# with production stopped: boots healthy in 420 s, plan reports runtime
|
||||
# overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
|
||||
# (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
|
||||
# pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
|
||||
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
|
||||
# GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
|
||||
# libhsa-runtime64.so), passed inside the container as a bare name — the env
|
||||
# file keeps only the prefixed form.
|
||||
grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
|
||||
# The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
|
||||
# all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
|
||||
# 229 MB of comgr JIT kernels into the ephemeral layer, and every
|
||||
# `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
|
||||
# (173.19 s of it one graph) on every deploy, restart and reboot.
|
||||
grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
# Prefix caching is ON (reverted 2026-10-05 the same day it was
|
||||
# disabled): the fork's QFN align-mode GDN state fault (48 gdn_hip
|
||||
# state faults, slot 42 heads 3/8, at the 13th 3136-token boundary =
|
||||
# 40,768 — the "~40k corruption" report) was mitigated with
|
||||
# --no-enable-prefix-caching, but live long conversations STILL
|
||||
# degraded at ~40k with caching off and zero faults logged — cost
|
||||
# with no fix, so the historical flag returned. The knob stays
|
||||
# commented in the env file for a future fixed fork build; flip it to
|
||||
# 1 only together with a re-verification past 40,768 tokens.
|
||||
grep -qE '^# FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
|
||||
@test "the FreeToken engine shape is in the env file" {
|
||||
# KV pool: TOKENS, not bytes. 24 KiB/token (12 full-attention layers x
|
||||
# 2 KV heads x 256 head_dim x K+V x f16) — the 36 GDN layers use fixed
|
||||
# per-seq state, not per-token KV. 300,000 tokens = 7.2 GiB and clears
|
||||
# the harness worst case (140,492-token prompt + 36,864 max_tokens =
|
||||
# 177,356); one NATIVE 262,144-token conversation + full budget also
|
||||
# fits. The arithmetic guard keeps future resizes honest.
|
||||
grep -q '^FRANK_PRIMARY_FT_NUM_TOKENS=300000$' "$ENV_FILE"
|
||||
nt=$(awk -F= '/^FRANK_PRIMARY_FT_NUM_TOKENS=/{print $2}' "$ENV_FILE")
|
||||
[ "$nt" -ge 177356 ]
|
||||
# First-boot conservative shape: 2 decode slots, 10% expert LRU
|
||||
# (~12 GiB of the ~120 GiB fp8 expert set), output ceiling = the
|
||||
# harness maxTokens, single R9700 at index 0.
|
||||
grep -q '^FRANK_PRIMARY_FT_MAX_RUNNING=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_FT_MOE_RATE=0.10$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS=36864$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_FT_GPU=0$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
|
||||
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
|
||||
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
|
||||
# Speculative decode, the cudagraph capture ladder and the sampling
|
||||
# defaults are all ENV-DRIVEN knobs (unset -> the flag is absent, which is
|
||||
# the shipped shape today). JSON values carry no shell-special characters
|
||||
# that this quoting cannot carry.
|
||||
grep -q -- '${FRANK_PRIMARY_SPEC:+--speculative-config "$FRANK_PRIMARY_SPEC"}' "$SERVE"
|
||||
grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE"
|
||||
grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE"
|
||||
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
||||
# Prefix caching renders exactly ONE flag from the knob, branched
|
||||
# explicitly in the serve script (the ${VAR:--flag} double-expansion is
|
||||
# ambiguous bash — when set it parsed as a substring offset, measured
|
||||
# 2026-10-05): set -> --no-enable-prefix-caching (the fork-named
|
||||
# mitigation, tried and REVERTED the same day — live corruption
|
||||
# persisted at ~40k with zero faults), unset -> the shipped
|
||||
# --enable-prefix-caching. Both directions are spelled out so a fork
|
||||
# default change cannot silently flip the contract.
|
||||
grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching' "$SERVE"
|
||||
grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching' "$SERVE"
|
||||
grep -qF '"$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' "$SERVE"
|
||||
# The observable contract: the exact flag word the engine receives,
|
||||
# rendered from the shipped env through the serve script's branch (the
|
||||
# same render-assert pattern the spec knobs use above).
|
||||
run bash -c '
|
||||
set -a; . "$1" >/dev/null; set +a
|
||||
if [ -n "${FRANK_PRIMARY_NO_PREFIX_CACHING:-}" ]; then
|
||||
FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching
|
||||
else
|
||||
FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
|
||||
fi
|
||||
printf %s "$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' _ "$ENV_FILE"
|
||||
[ "$output" = "--enable-prefix-caching" ]
|
||||
grep -q -- '--trust-remote-code' "$SERVE"
|
||||
# DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
|
||||
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
|
||||
grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
|
||||
# --expert-precise-memory selects the fitted overhead (5.25 GiB) over the
|
||||
# default (7.60 GiB = fitted + compile/graph-capture headroom), which only
|
||||
# moves the plan's `room` term and therefore the expert arena: ~1,025
|
||||
# more slots at 2.344 MiB each.
|
||||
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
|
||||
# Shape E host/CPU wiring: the cache dir is created before it is mounted
|
||||
# (a cold boot must have a target) and the mount lands on /root/.cache,
|
||||
# whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
|
||||
# names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
|
||||
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
|
||||
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
|
||||
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
|
||||
# Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork
|
||||
# reads it at call time inside estimate_cudagraph_memory), and both must be
|
||||
# present — the graph term and UTIL are the ONLY plan terms this repo can
|
||||
# move, and shrinking them is what grows the arena. The base must be shipped
|
||||
# explicitly: unset, the estimator would fall back to its 256 MiB default.
|
||||
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
|
||||
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
|
||||
# The host region is the ONE plan term that autofits per-boot: with no cap
|
||||
# the bound is MemAvailable at that moment (boot 2026-10-05: 237.97 GiB ->
|
||||
# 159.39 GiB usable, while the running host later reads 75.4 GiB available).
|
||||
# An explicit --expert-offload-mem cap makes the plan deterministic whatever
|
||||
# the co-tenant RAM state is at boot; 120 GiB covers the measured 105.48
|
||||
# GiB of swappable experts (52.74/rank x 2) with 14% headroom.
|
||||
grep -q -- '${FRANK_PRIMARY_EXPERT_OFFLOAD_MEM:+--expert-offload-mem "$FRANK_PRIMARY_EXPERT_OFFLOAD_MEM"}' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120$' "$ENV_FILE"
|
||||
grep -q 'ft serve --model /models' "$SERVE"
|
||||
grep -q -- '--served-model-name "$FRANK_PRIMARY_SERVED_NAME"' "$SERVE"
|
||||
grep -q -- '--host 0.0.0.0 --port 7770' "$SERVE"
|
||||
grep -q -- '--gpu "$FRANK_PRIMARY_FT_GPU"' "$SERVE"
|
||||
grep -q -- '--num-tokens "$FRANK_PRIMARY_FT_NUM_TOKENS"' "$SERVE"
|
||||
grep -q -- '--max-running-requests "$FRANK_PRIMARY_FT_MAX_RUNNING"' "$SERVE"
|
||||
grep -q -- '--max-output-tokens "$FRANK_PRIMARY_FT_MAX_OUTPUT_TOKENS"' "$SERVE"
|
||||
grep -q -- '--moe-cache-rate "$FRANK_PRIMARY_FT_MOE_RATE"' "$SERVE"
|
||||
# Single-GPU law: visibility is pinned to one card so in-container
|
||||
# index 0 is the right R9700 (mirrors the env's FT_GPU).
|
||||
grep -q -- '-e HIP_VISIBLE_DEVICES="$FRANK_PRIMARY_FT_GPU"' "$SERVE"
|
||||
# The boot-ladder escape hatch is opt-in: unset renders NO extra argv.
|
||||
grep -q -- '${FRANK_PRIMARY_FT_EXTRA_ARGS:+$FRANK_PRIMARY_FT_EXTRA_ARGS}' "$SERVE"
|
||||
! grep -q '^FRANK_PRIMARY_FT_EXTRA_ARGS=.' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
# 2026-10-01: the rollback-era trio served NO tool dialect and left
|
||||
# thinking inline in `content` with a dangling `</think>` and a null
|
||||
# `reasoning` field (measured: content 'We need answer...Brief.\n</think>
|
||||
# \n\n391') — the malformed thinking block clients choke on. The working
|
||||
# values are the ones ecaad36 shipped for this model and this fork's
|
||||
# registries confirm: reasoning parser `qwen3`, tool parser `qwen3_xml`.
|
||||
# preserve_thinking keeps prior thinking blocks across turns.
|
||||
grep -q '^FRANK_PRIMARY_REASONING_PARSER=qwen3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
|
||||
grep -q -- '--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER"' "$SERVE"
|
||||
grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
grep -q -- '--tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER"' "$SERVE"
|
||||
grep -q -- '--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS"' "$SERVE"
|
||||
grep -q 'preserve_thinking' "$ENV_FILE"
|
||||
grep -q 'enable_thinking' "$ENV_FILE"
|
||||
@test "parsers are AUTO — the engine infers them natively, the harness maps the field" {
|
||||
# FreeToken infers the qwen3.8 reasoning/tool dialects from the model
|
||||
# family and emits thinking in `reasoning_content` natively — NO
|
||||
# --reasoning-parser/--tool-call-parser override flags. The vLLM-era
|
||||
# client-contract wiring (qwen3 parser, qwen3_xml tool parser,
|
||||
# default-chat-template-kwargs) is gone; models.yml's
|
||||
# reasoningContentField flip is the harness-side contract and lives in
|
||||
# the operator's home, not this repo.
|
||||
! grep -q -- '--reasoning-parser' "$SERVE"
|
||||
! grep -q -- '--tool-call-parser' "$SERVE"
|
||||
! grep -q -- '--enable-auto-tool-choice' "$SERVE"
|
||||
! grep -q -- '--default-chat-template-kwargs' "$SERVE"
|
||||
! grep -q 'FRANK_PRIMARY_REASONING_PARSER' "$ENV_FILE"
|
||||
! grep -q 'FRANK_PRIMARY_TOOL_PARSER' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "expert offload is AUTOMATIC — the payload cannot fit VRAM and no flag sets it" {
|
||||
# ~86 GiB/rank of FP8 trunk against 32 GiB of VRAM: the fork's tiered-
|
||||
# expert planner moves the overflow to pinned host rows by itself (the
|
||||
# arena 6,574 slots x 2.344 MiB plus host rows). There is no
|
||||
# --enable-expert-offload flag in this arm; the legality of the pin is
|
||||
# the unit's unbounded LimitMEMLOCK.
|
||||
! grep -q -- '--enable-expert-offload' "$SERVE"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
|
||||
}
|
||||
|
||||
@test "speculative decode is an ENV KNOB — the old retirement verdict was invalid" {
|
||||
# The 2026-10-02 c1248d4 retirement of MTP compared the WRONG PAIR:
|
||||
# fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance) against the bf16 no-spec boot
|
||||
# (36.4-39.4), then bf16 proved unbootable and fp8 came back WITHOUT MTP —
|
||||
# so fp8+spec was never compared against fp8+no-spec on the same shape. The
|
||||
# no-spec incumbent runs ~30 tok/s, under the fleet's 35 tok/s target, while
|
||||
# EVERY vendor flash-next recipe ships MTP3, and our MTP4 run used the
|
||||
# default GREEDY draft sampling (probabilistic is the other fork option and
|
||||
# is what the recipes use under temp>0). So spec is a knob, not a law: the
|
||||
# shipped value is whatever a matched-pair re-measurement wins.
|
||||
grep -q 'THE RETIREMENT VERDICT WAS INVALID' "$ENV_FILE"
|
||||
# The knobs are empty in the shipped env, so the `${VAR:+…}` wiring renders
|
||||
# NO spec/compilation flag — asserted by rendering, not by grepping the
|
||||
# assignment (shape E today has neither flag in its argv).
|
||||
run bash -c 'set -a; . "$1" >/dev/null; set +a
|
||||
printf %s "${FRANK_PRIMARY_SPEC:+--speculative-config $FRANK_PRIMARY_SPEC}${FRANK_PRIMARY_COMPCONFIG:+--compilation-config $FRANK_PRIMARY_COMPCONFIG}"' _ "$ENV_FILE"
|
||||
[ -z "$output" ]
|
||||
# The capture-ladder law: the piecewise graph ladder must contain
|
||||
# num_speculative_tokens + 1, so an MTP3 shape captures [4, 8, 12] and the
|
||||
# 2026-10-02 SEQS=2 failure (query_start_loc specialized to the constant 3)
|
||||
# is why SEQS stays a multiple-compatible value.
|
||||
grep -q 'num_speculative_tokens + 1' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "KV dtype is fp8 — the incumbent, and the only one that boots" {
|
||||
# fp8 costs ~7,701 B/token at 262144 (7,892 at 200704 — shape-dependent),
|
||||
# decomposed as 6,138 attention + 1,563 mamba/side (dtype-independent);
|
||||
# bf16 costs 13,839 (12,276 + 1,563).
|
||||
# bf16 was abandoned 2026-10-01: its 2 x 262144 boot died with 0 bytes
|
||||
# free on a real request, and the SEQS=2 slot count it was designed
|
||||
# around cannot compile this model at all (see the shape test above).
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
|
||||
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
|
||||
# sweep's biggest lever, +18.7% decode vs default geometry, and the win
|
||||
# GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
|
||||
# g64x128 is past the peak; g8x16 mild. They are container ENV (the
|
||||
# fork's knobs), not engine flags — passed through with -e.
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the arena sits just under the ceiling and the KV ask is honoured" {
|
||||
# The planner sizes expert cache as budget - non-expert - KV reserve -
|
||||
# graphs - overhead, and two of those five terms are reachable from a
|
||||
# deploy: budget (= device_total x util) and graphs (VLLM_CUDAGRAPH_MEM_*).
|
||||
# UTIL has a HARD LAW, not a field cap: 0.982, never 1.00. The 1.00 attempt
|
||||
# is UNBOOTABLE on this fork — worker init raises
|
||||
# `ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)`
|
||||
# on both ranks because the worker demands free > budget and budget ==
|
||||
# total — and even if it booted, 0.13-GiB-margin shapes have died under
|
||||
# real load. So the arena tops out at 0.982 budget (31.29 GiB).
|
||||
# Graphs: charge strictly below the fork's analytic 563 MiB default while
|
||||
# still paying per graph (the measured post-capture delta is -0.06 GiB, i.e.
|
||||
# the base pool is the over-charge, but a future spec-decode shape captures
|
||||
# 18 graphs and must still be charged).
|
||||
# MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode-
|
||||
# interchangeable AND arena-identical (the registry prices overhead flat up
|
||||
# to its nbt 4096 anchor), 12288 costs 2.5% plus arena.
|
||||
# The KV ask: the pool is priced at the engine's measured 7,892 B/token at
|
||||
# this shape, and a served conversation needs 200,704 tokens, so the pool
|
||||
# must clear 400,700 tokens to keep two full-length resident.
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
|
||||
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
|
||||
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
|
||||
[ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ]
|
||||
[ $((base + 6 * per_graph)) -lt 563 ]
|
||||
kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE")
|
||||
[ $((kv / 7892)) -ge 400700 ]
|
||||
}
|
||||
|
||||
@test "expert prefetch is absent BY LAW, not by omission" {
|
||||
# Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
|
||||
# prefetched slabs (36% precision) — roughly half the gather traffic was
|
||||
# waste, and the "savings" would have been installs anyway. Deleting the
|
||||
# flag moved decode 26.8 -> 40.6 tok/s. Any future flag must come with a
|
||||
# fresh A/B, so the serve script must NOT wire it unconditionally.
|
||||
! grep -q -- '--expert-prefetch' "$SERVE"
|
||||
! grep -q 'expert_prefetch' "$ENV_FILE"
|
||||
grep -q 'PREFETCH IS ABSENT BY LAW' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "TunableOp stays off" {
|
||||
# CLAV_TUNABLEOP_SWEEP silently skips under expert offload unless
|
||||
# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics
|
||||
# (repetition loops, early halts, invalid measurements).
|
||||
! grep -q 'TUNABLEOP' "$ENV_FILE"
|
||||
! grep -q 'TUNABLEOP' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the serve script gates the digest-pinned image and the payload" {
|
||||
# The image is pinned by OCI manifest digest (the floating :latest moved
|
||||
# once already). The payload gate checks config.json and the exact shard
|
||||
# count from the env — a missing shard means a guaranteed long-timeout
|
||||
# boot instead of a fast failure.
|
||||
@test "the serve script gates the image, the payload and the GPU nodes" {
|
||||
# The image gate is EXISTENCE (podman image exists): the FreeToken
|
||||
# image is built ON THE HOST — a local tag carries no RepoDigests — and
|
||||
# pinning lives in the Dockerfile (pinned PR-head SHA, base image).
|
||||
# The payload gate checks config.json and the exact shard count from
|
||||
# the env — a missing shard means a guaranteed long-timeout boot
|
||||
# instead of a fast failure.
|
||||
grep -q 'FRANK_PRIMARY_IMAGE' "$SERVE"
|
||||
grep -q 'RepoDigests' "$SERVE"
|
||||
grep -q 'podman image exists' "$SERVE"
|
||||
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$SERVE"
|
||||
grep -q 'config.json' "$SERVE"
|
||||
grep -q 'model.safetensors.index.json' "$SERVE"
|
||||
grep -q '/dev/kfd' "$SERVE"
|
||||
grep -q -- '--device /dev/dri' "$SERVE"
|
||||
! grep -q 'tcclaviger/vllm:latest' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the serve script clears stale containers and self-heals via podman wait" {
|
||||
# A crashed unit can leave the container name registered forever after;
|
||||
# the legacy identities (the retired R9V container and the ad hoc
|
||||
# experiment container) hold the same GPUs and die here too. Then block
|
||||
# on the container so a crash is a nonzero ExecStart.
|
||||
# the legacy identities (the vLLM-era frank-primary-qwen container, the
|
||||
# retired R9V container and the ad hoc experiment containers) hold the
|
||||
# same GPU and die here too. Then block on the container so a crash is
|
||||
# a nonzero ExecStart.
|
||||
grep -q 'FRANK_PRIMARY_LEGACY_CONTAINERS' "$ENV_FILE"
|
||||
grep -q 'frank-primary-qwen' "$ENV_FILE"
|
||||
grep -q 'frank-primary-r9v' "$ENV_FILE"
|
||||
grep -q 'qwen-fp8' "$ENV_FILE"
|
||||
grep -q 'exec podman wait "$FRANK_PRIMARY_CONTAINER"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "frank-primary holds the single fleet port under its own alias" {
|
||||
# The scratch identity (qwen3.8-flash-next on :7773) must never leak
|
||||
# into production: LAN publish of :7770, the harness's model id.
|
||||
# LAN publish of :7770, the harness's model id.
|
||||
grep -q '^FRANK_PRIMARY_PUBLISH=7770:7770$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SERVED_NAME=frank-primary$' "$ENV_FILE"
|
||||
}
|
||||
@@ -404,16 +201,38 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the deployer gates the image digest and payload, then health-checks" {
|
||||
# The deployer refuses when the pinned image is absent from rootless
|
||||
# storage or the payload is missing/wrong-sharded — then clears stale
|
||||
# containers (including the legacy identities), waits out the port,
|
||||
@test "the Dockerfile pins the FreeToken source and follows the ROCm install law" {
|
||||
# The engine is the PR #587 head, NOT a moving branch: a future push to
|
||||
# rocm714_torch214 would otherwise silently change the deployed engine.
|
||||
grep -q '77a7201f98f00d93f618c6cf8b81ffe8360e0899' "$DOCKERFILE"
|
||||
grep -q 'pull/587/head' "$DOCKERFILE"
|
||||
# docs/install_amd.md's build law: --no-build-isolation --no-deps (a
|
||||
# plain `pip install -e .` on ROCm can REPLACE the ROCm torch with a
|
||||
# CUDA build via its NVIDIA-authored dependency set), curated portable
|
||||
# deps, flashlib --no-deps.
|
||||
grep -q -- '--no-build-isolation --no-deps -e .' "$DOCKERFILE"
|
||||
grep -q 'flashlib==0.3.0' "$DOCKERFILE"
|
||||
# The CUDA-only backends have no ROCm builds and are intentionally
|
||||
# absent (the PR's 12 pytest failures are exactly those backends;
|
||||
# FreeToken falls back to its Triton kernels).
|
||||
! grep -q 'flashinfer' "$DOCKERFILE"
|
||||
! grep -q 'sgl_kernel' "$DOCKERFILE"
|
||||
! grep -q 'nvidia-cutlass-dsl' "$DOCKERFILE"
|
||||
# The base image is digest-pinned (the registry tag is not trusted).
|
||||
grep -qE '^FROM rocm/pytorch:.*@sha256:[0-9a-f]{64}$' "$DOCKERFILE"
|
||||
}
|
||||
|
||||
@test "the deployer gates the image, payload and containers, then health-checks" {
|
||||
# The deployer refuses when the image is absent from rootless storage
|
||||
# or the payload is missing/wrong-sharded — then clears stale containers
|
||||
# (including the legacy identities), waits out the port,
|
||||
# relinks/enables/restarts the root unit and health-gates it.
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary$' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
|
||||
grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
|
||||
grep -q 'podman image exists' "$REMOTE_DEPLOY"
|
||||
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
|
||||
grep -q 'config.json' "$REMOTE_DEPLOY"
|
||||
grep -q 'frank-primary-qwen' "$REMOTE_DEPLOY"
|
||||
grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
|
||||
grep -q 'glm-probe' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
|
||||
Reference in New Issue
Block a user