dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary: vLLM 0.30 UVA-offload shape (pre-deploy commit)

Browse Source 
The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the
FP8 cutover on :7770 — same port, same alias, no client moves.

Serve shape: locally built ROCm gfx1201 image
(localhost/frank-primary:vllm030) execs via podman as dank from the
root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload:
--offload-backend uva pins the PLE family (47.75 GiB, 138 tensors:
131 n-gram table shards + 7 per-layer conv/proj) + routed experts
(114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined,
~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card
strands 2.6 GiB of experts on the cards). Parsers per the host's
own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM
0.30 emits the channel as `reasoning` — reasoning_content is
renamed server-side) + enable_thinking/preserve_thinking template
kwargs. Spec decode stays env-wired and OFF at bring-up.

Deployer deploy_llm_primary rewritten: image gate, index-driven
shard-count gate (131), stale-container cleanup, port wait, and the
false-green hardening — health is only green when the unit's own
MainPID is live behind the answer (2026-09-26 lesson).

Seam tests rewritten to the vLLM contract (15 gates): offload
arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite
349/349 green.
This commit is contained in:
Frank
2026-09-27 00:16:42 -07:00
parent 82960c77db
commit ecaad36654
5 changed files with 378 additions and 425 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_primary_unit.bats
+112 -116
View File
@@ -1,16 +1,17 @@
#!/usr/bin/env bats
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
#
# This file asserts the llama.cpp ROOT-UNIT shape (the 2026-09-26 swap that
# replaced the Lucebox DS4.1 Quadlet): the unsloth b11030-mix prebuilt on
# the two R9700s (gfx1201) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0
# with the shared Q8_0 MTP head, the BF16 projector, and the
# draft-mtp + ngram-mod chain. The seam pins what the flags alone do not
# say: the CPU-override placement law (experts and the 51B n-gram table
# are COMPUTED in host RAM, never streamed), mmap staying on, the
# shared-head borrow requirement (per_layer_token_embd on CPU), the
# explicit projector (never autoloaded by this build), HIP env hygiene,
# and the deployer's byte-count gates over a six-part payload.
# This file asserts the vLLM 0.30 ROOT-UNIT shape (the 2026-09-27 swap that
# replaced the llama.cpp b11030 endpoint): a locally built ROCm gfx1201
# image (localhost/frank-primary:vllm030) serving Qwen3.8-Flash-Next-FP8
# (Qwen4ExpForConditionalGeneration, 131 shards, 172.78 GiB) on the two
# R9700s with the UVA offload shape. The seam pins what the flags alone do
# not say: the offload budget arithmetic (the bandwidth-huge tensors — the
# 47.75 GiB PLE family and the 114.86 GiB routed experts — are pinned in
# host RAM and read in place, never streamed), the memlock law that makes
# the pin legal, the parser contract that keeps the harness wire formats
# (tool_calls + the reasoning channel) working, HIP env hygiene, and the
# deployer's image/payload gates with the false-green MainPID check.
load helpers
@@ -27,8 +28,10 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "frank-primary is boot-persistent, self-healing and unprivileged" {
# Root unit parity with frank-large/frank-small: enable is the [Install]
# section, the restart policy survives a crash, and the mmap'd payload
# needs an unbounded memlock.
# section, the restart policy survives a crash, and the ~164 GiB UVA pin
# needs an unbounded memlock (a root unit grants it via PID1; the dank
# session's own hard limit is 8 MiB and cannot be raised without
# restarting user@1000.service).
grep -q '^WantedBy=multi-user.target$' "$UNIT"
grep -q '^Restart=on-failure$' "$UNIT"
grep -q '^RestartSec=' "$UNIT"
@@ -41,99 +44,96 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
grep -q 'FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env' "$UNIT"
}
@test "frank-primary drafts with the shared MTP head chained behind ngram-mod" {
# The chain is draft-mtp (context-backed) + ngram-mod (draftless): the
# only legal way to layer two spec implementations in one process. The
# shared Q8_0 head borrows the target's embedding tensors — which is
# why per_layer_token_embd.weight must be CPU-resident (a GPU-resident
# borrow crashes the draft load with "invalid vector subscript").
grep -q -- '--spec-type draft-mtp,ngram-mod' "$SERVE"
grep -q '^SPEC_DRAFT_N_MAX=2$' "$ENV_FILE"
grep -q -- '--spec-draft-n-max "$SPEC_DRAFT_N_MAX"' "$SERVE"
grep -q 'DRAFTER_FILE="$MODEL_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf"' "$SERVE"
grep -q -- '--spec-draft-device "$SPEC_DRAFT_DEVICE"' "$SERVE"
@test "the serve shape is vLLM: no llama.cpp-ism survives the cutover" {
# Clean cutover: the GGUF era's flags would silently no-op or crash a vLLM
# server, so their absence is part of the contract.
! grep -q -- '--model-draft' "$SERVE"
! grep -q -- '--spec-type' "$SERVE"
! grep -q -- '--mmproj' "$SERVE"
! grep -q -- '--override-tensor' "$SERVE"
! grep -q '\.gguf' "$SERVE"
! grep -q '\.gguf' "$ENV_FILE"
}
@test "the bandwidth-huge tensors are computed in host RAM, not streamed" {
# The x4-pair law: streaming experts over PCIe loses (3.7 tok/s) while
# computing them in RAM does not care (20+ tok/s on the same model
# shape). exps=CPU plus the n-gram embedding table on CPU; the PLE
# table must NEVER carry a GPU target (the 23 -> 13.4 t/s regression).
# ONE joined flag (b11030 deprecates repeated --override-tensor), and
# first-match-wins means the block-specific patterns must precede the
# broad exps=CPU sweep — the exact value pins the whole law.
grep -q -- '--override-tensor "$OT_OVERRIDES"' "$SERVE"
! grep -q -- '--no-mmap' "$SERVE"
! grep -q '^MMPROJ_ON_GPU' "$ENV_FILE"
ot="$(sed -nE "s/^OT_OVERRIDES='(.*)'$/\1/p" "$ENV_FILE")"
[ "$ot" = 'blk\.[0-4]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU' ]
@test "the bandwidth-huge tensors are pinned in host RAM, not streamed" {
# The UVA shape (the x4-pair law: streaming experts over PCIe loses,
# computing them in host RAM does not care): --offload-backend uva pins
# the matched weights with zero-copy reads. Measured from the payload
# headers (2026-09-26): `ple` matches 138 tensors (131 n-gram table
# shards + 7 per-layer conv/proj weights) = 47.75 GiB; `experts` matches
# 150,528 tensors = 114.86 GiB; combined 162.60 GiB. cpu-offload-gb is
# PER GPU: under TP=2 each rank needs ~81.3 GiB, hence 82 — 80/card
# would strand 2.6 GiB of experts on the cards.
grep -q '^OFFLOAD_BACKEND=uva$' "$ENV_FILE"
grep -q '^OFFLOAD_GB=82$' "$ENV_FILE"
grep -q '^OFFLOAD_PARAMS=ple,experts$' "$ENV_FILE"
grep -q -- '--offload-backend "$OFFLOAD_BACKEND"' "$SERVE"
grep -q -- '--cpu-offload-gb "$OFFLOAD_GB"' "$SERVE"
grep -q -- '--cpu-offload-params "$OFFLOAD_PARAMS"' "$SERVE"
# auto/prefetch are the copy paths: this deployment is only qualified
# for uva, so the serve script refuses anything else.
grep -q 'only qualified for uva' "$SERVE"
}
@test "the KV pool is V-quantized, K held at f16" {
# V q8_0 halves nothing (it is 1/4 of the per-token law: 18,816 of the
# 24,576 B/token f16/f16 cost) but it is the deploy's stated shape; the
# flags must be env-wired tunables so the quant can be reverted
# without touching the serve script.
grep -q '^CACHE_TYPE_K=f16$' "$ENV_FILE"
grep -q '^CACHE_TYPE_V=q8_0$' "$ENV_FILE"
grep -q -- '--cache-type-k "$CACHE_TYPE_K"' "$SERVE"
grep -q -- '--cache-type-v "$CACHE_TYPE_V"' "$SERVE"
@test "the engine shape: TP=2 over both R9700s, full native window, batch 8" {
grep -q '^TP=2$' "$ENV_FILE"
grep -q '^GPU_UTIL=0.97$' "$ENV_FILE"
grep -q '^CTX_SIZE=262144$' "$ENV_FILE"
grep -q '^MAX_SEQS=8$' "$ENV_FILE"
grep -q -- '--tensor-parallel-size "$TP"' "$SERVE"
grep -q -- '--gpu-memory-utilization "$GPU_UTIL"' "$SERVE"
grep -q -- '--max-model-len "$CTX_SIZE"' "$SERVE"
grep -q -- '--max-num-seqs "$MAX_SEQS"' "$SERVE"
}
@test "the projector is explicit, never autoloaded" {
# This build never autoloads mmproj from the model directory: without
# the flag the server comes up healthy and text-only, answering image
# requests with 500 (the frank-medium 2026-09-17 trap).
grep -q -- '--mmproj "$MMPROJ_FILE"' "$SERVE"
grep -q 'MMPROJ_FILE="$MODEL_ROOT/mmproj-BF16.gguf"' "$SERVE"
@test "the payload is the FP8 tree, vision served from the model's own config" {
# Qwen4ExpForConditionalGeneration: vLLM auto-loads the ViT and the
# mtp.* weights from the payload config — no explicit projector flag
# exists in this runtime (the llama.cpp --mmproj trap does not apply).
grep -q '^MODEL_DIR=/models/Qwen3.8-Flash-Next-FP8$' "$ENV_FILE"
grep -q '^IMAGE=localhost/frank-primary:vllm030$' "$ENV_FILE"
}
@test "frank-primary is vision-serving but holds the single fleet port" {
@test "frank-primary holds the single fleet port under its own alias" {
grep -q '^PORT=7770$' "$ENV_FILE"
grep -q '^BIND_HOST=0.0.0.0$' "$ENV_FILE"
grep -q -- '--alias frank-primary' "$SERVE"
grep -q '^SERVED_NAME=frank-primary$' "$ENV_FILE"
grep -q -- '--served-model-name "$SERVED_NAME"' "$SERVE"
grep -q -- '--host "$BIND_HOST"' "$SERVE"
grep -q -- '--port "$PORT"' "$SERVE"
}
@test "four streams, each the model's full native window" {
# --parallel divides --ctx-size into slots; the contract is four
# conversations each holding the FULL native 262144 window, so the
# per-slot window (ctx / parallel) equals the native size.
local ctx parallel
ctx="$(sed -nE 's/^CONTEXT_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
parallel="$(sed -nE 's/^PARALLEL_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
[ -n "$ctx" ] && [ -n "$parallel" ]
[ "$parallel" -eq 4 ]
[ "$((ctx / parallel))" -eq 262144 ]
grep -q -- '--ctx-size "$CONTEXT_SIZE"' "$SERVE"
grep -q -- '--parallel "$PARALLEL_SLOTS"' "$SERVE"
@test "tool_calls and the reasoning channel are wired for the harness" {
# The wire contract: vLLM 0.30 emits the thinking channel as `reasoning`
# (the deprecated `reasoning_content` is renamed server-side), the qwen3
# parser extracts the template's think block into it, and qwen3_xml +
# auto-tool-choice structure tool calls. Precedent: the host's own
# Ornith-era scripts (~/vllm/failedAttempts/OrnithFP8.sh); the FP8
# template consumes enable_thinking/preserve_thinking (verified in
# tokenizer_config.json).
grep -q '^TOOL_PARSER=qwen3_xml$' "$ENV_FILE"
grep -q '^REASONING_PARSER=qwen3$' "$ENV_FILE"
grep -q -- '--enable-auto-tool-choice --tool-call-parser "$TOOL_PARSER"' "$SERVE"
grep -q -- '--reasoning-parser "$REASONING_PARSER"' "$SERVE"
# The template kwargs are JSON: the env value must be single-quoted or
# bash splits it and CHAT_TEMPLATE_KWARGS arrives empty (the -ot lesson).
grep -q "^CHAT_TEMPLATE_KWARGS='{\"enable_thinking\": true, \"preserve_thinking\": true}'$" "$ENV_FILE"
grep -q -- '--default-chat-template-kwargs "$CHAT_TEMPLATE_KWARGS"' "$SERVE"
}
@test "reasoning is extracted to reasoning_content and tools are legal" {
# The harness contract: llama.cpp endpoints emit reasoning_content
# (the deepseek format is the one that fills it), and tools require
# --jinja (the ik_llama 500 lesson). The Qwen3.8 template gates
# reasoning_effort to {xhigh, medium, low} — folded in models.yml.
grep -q -- '--jinja' "$SERVE"
grep -q -- '--reasoning auto' "$SERVE"
grep -q -- '--reasoning-format deepseek' "$SERVE"
grep -q -- '--reasoning-budget -1' "$SERVE"
}
@test "the greedy server default is explicit" {
# Deterministic e2e probes; the harness overrides sampling per request.
grep -q -- '--temp 0' "$SERVE"
grep -q -- '--top-p 1.0' "$SERVE"
grep -q -- '--top-k 0' "$SERVE"
grep -q -- '--min-p 0.0' "$SERVE"
@test "speculative decode is env-wired and OFF at bring-up" {
# MTP drafting comes from the model's own mtp.* weights via
# --speculative-config; bring-up measures the base shape clean, so
# SPEC_CONFIG is empty and the flag is only added when non-empty.
grep -q '^SPEC_CONFIG=$' "$ENV_FILE"
grep -q -- '--speculative-config "$SPEC_CONFIG"' "$SERVE"
}
@test "both R9700s are visible and the Raphael iGPU is not" {
# The host has a third (Raphael) GPU the server must never place
# tensors on; the bundled ROCm runtime is the only one linked.
grep -q 'export HIP_VISIBLE_DEVICES=0,1$' "$SERVE"
grep -q 'export LD_LIBRARY_PATH="$RUNTIME_DIR' "$SERVE"
grep -q '^ROCR_VISIBLE_DEVICES=0,1$' "$ENV_FILE"
grep -q '^HIP_VISIBLE_DEVICES=0,1$' "$ENV_FILE"
grep -q -- '-e ROCR_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"' "$SERVE"
grep -q -- '-e HIP_VISIBLE_DEVICES="$HIP_VISIBLE_DEVICES"' "$SERVE"
}
@test "no bare HIP-reserved env var is defined" {
@@ -145,40 +145,36 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
}
@test "the serve script refuses to start on a half-finished download" {
# Six split parts plus the head and the projector: a missing tail part
# fails at LOAD time with a confusing partial-mmap error, so every
# artifact is checked before the server execs.
grep -q 'missing asset' "$SERVE"
grep -q 'download-frank-primary.sh' "$SERVE"
@test "the serve script refuses to start on a half-finished payload or missing image" {
# A payload missing its wrapper config or shard index turns the
# container start into a guaranteed 20-minute timeout; the checks fail
# fast with a pointer at the fetch script instead.
grep -q 'config.json or index missing' "$SERVE"
grep -q 'download-frank-primary-fp8.sh' "$SERVE"
grep -q 'podman image inspect' "$SERVE"
}
@test "the prompt-cache cap is set below the RAM the host can spare" {
grep -q '^CACHE_RAM=16384$' "$ENV_FILE"
grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
}
@test "the deployer retires the lucebox quadlet, gates every part and waits out the port" {
# The cutover order: the user-scope quadlet is stopped and its
# generated .container moved to dated home trash BEFORE the root unit
# binds 7770 (nothing starts until the port is released — the 2026-09-26
# and 2026-09-16 cutover failures were both here).
grep -q 'retire_lucebox_primary_quadlet' "$REMOTE_DEPLOY"
grep -q 'systemctl --user stop frank-primary.service' "$REMOTE_DEPLOY"
@test "the deployer gates the image and the full shard set before starting" {
# The deployer refuses when the image or any index-named shard is
# absent (never serve a partial download), clears any stale container
# that would break `podman run --name`, and waits out the port before
# the new engine binds it.
grep -q 'LLM_PRIMARY_IMAGE=localhost/frank-primary:vllm030' "$REMOTE_DEPLOY"
grep -q 'LLM_PRIMARY_ROOT=/home/dank/models/Qwen3.8-Flash-Next-FP8' "$REMOTE_DEPLOY"
grep -q 'weight_map' "$REMOTE_DEPLOY"
grep -q 'download-frank-primary-fp8.sh' "$REMOTE_DEPLOY"
grep -q "podman rm -f \"\$LLM_PRIMARY_CONTAINER\"" "$REMOTE_DEPLOY"
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"
grep -q 'systemctl restart frank-primary.service' "$REMOTE_DEPLOY"
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
}
@test "the deployer gates the six-part payload at declared byte counts" {
# stat -c %s, never wc -c — wc read the whole 377 GiB lucebox file for
# minutes on 2026-09-26 before it could reach the cutover. A partial
# download must never be served.
grep -q 'LLM_PRIMARY_PART_SIZES' "$REMOTE_DEPLOY"
grep -q 'stat -c %s' "$REMOTE_DEPLOY"
grep -q '2786568256' "$REMOTE_DEPLOY"
grep -q '907542944' "$REMOTE_DEPLOY"
grep -q 'wait_for_health frank-primary http://127.0.0.1:7770/health 1800' "$REMOTE_DEPLOY"
}
@test "the deployer will not report green on a stray port-holder" {
# The 2026-09-26 false green: a leftover probe held :7770 and satisfied
# the health curl while the real unit was stopped. The deploy is only
# green when the unit's own MainPID is live behind the health answer.
grep -q 'systemctl show -p MainPID --value frank-primary.service' "$REMOTE_DEPLOY"
grep -q 'stray process holds the port' "$REMOTE_DEPLOY"
}