dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary cutover: tcclaviger RDNA4 stack, 600k ctx / 3 sessions / q8-q8 KV

Browse Source 
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the
digest-pinned image the 2026-09-29 tuning campaign measured), serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB,
already on disk) with MTP4 spec decode.

Qualified by boot probe on 2026-09-29:
- MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV):
  'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000
  tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK.
- KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to
  allocate the full size; ~7.5 GiB/rank is the ceiling with the arena).
- prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic);
  TunableOp off (corrupts numerics).
- decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots.

Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh
(direct podman run; entrypoint execs vllm serve; waits on the container),
frank-primary.service (root unit, same ExecStop law, new container name
frank-primary-qwen). Deployer gates rewritten (image digest + shard count
+ weight-map index). download-frank-primary.sh is now inventory/verify
(repo is GATED — 401 anonymous, masked LFS hashes; payload already on
disk). Seam rewritten for the new contract. Pre-cutover tag:
frank-primary-r9v-q8-20260929.
This commit is contained in:
Frank committed 2026-09-30 00:46:30 -07:00
1 parent 9cf4b1de82
commit 3bb0e82ccf
6 files changed
+409 -389
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_primary_unit.bats
+113 -68
View File
@@ -1,17 +1,19 @@
#!/usr/bin/env bats
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
#
# This file asserts the R9V radiance-vLLM ROOT-UNIT shape (the 2026-09-27
# swap that replaced the llama.cpp b11030 endpoint): the R9V fork's own
# launcher (scripts/launch.sh in the pinned fork checkout) driven through a
# docker->podman shim, with the qualified on-host setup state layered under
# the repo's production overrides. The seam pins what the flags alone do not
# say: the state-layering contract (104 tuned R9V_* keys live on-host; the
# repo env carries only production overrides), the self-healing law (the
# launcher detaches, so the serve script must wait on the container for
# Restart=on-failure to mean anything), the memlock law that makes the
# tiered-expert pin legal, and the deployer's state/payload/image gates with
# the false-green MainPID check.
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the 2026-09-29
# cutover that replaced the R9V radiance fork): the serve script drives
# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
# engine flag qualified by boot probe on 2026-09-29 and carried in the repo
# env file. The seam pins what the flags alone do not say: the qualification
# law (600k ctx / 3 sessions / q8-q8 KV were MEASURED to fit — 729,545 tokens
# of capacity at 7.2 GiB per rank — and KV_BYTES is PER RANK), the self-healing
# law (the run returns before the engine is up, so the serve script must wait
# on the container for Restart=on-failure to mean anything), the memlock law
# that makes the tiered-expert pin legal, the deployer's image/payload gates
# with the false-green MainPID check, and the prefetch law (expert prefetch
# measured at 36% precision — OFF is the incumbent; a regression that
# reintroduces it silently costs ~40-50% decode).
load helpers
@@ -19,6 +21,7 @@ UNIT="$REPO_ROOT/scripts/frank-primary.service"
SERVE="$REPO_ROOT/scripts/serve-frank-primary.sh"
ENV_FILE="$REPO_ROOT/scripts/frank-primary.env"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
@test "frank-primary unit, serve script and env exist in scripts/" {
[ -f "$UNIT" ]
@@ -45,90 +48,132 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
}
@test "the unit stops the container child explicitly" {
# The serve script blocks on docker wait, which cannot stop the
# The serve script blocks on podman wait, which cannot stop the
# container child; systemd stop goes through ExecStop or the engine
# outlives the unit and holds both GPUs.
# outlives the unit and holds both GPUs. The container name must match
# the env file's identity.
grep -q '^ExecStop=/usr/bin/podman stop' "$UNIT"
grep -q 'frank-primary-r9v' "$UNIT"
grep -q 'frank-primary-qwen' "$UNIT"
grep -q '^FRANK_PRIMARY_CONTAINER=frank-primary-qwen$' "$ENV_FILE"
}
@test "the serve shape is the R9V launcher, not a frozen podman run" {
# The launcher owns the resolved container args (binds, overlays, device
# passthrough); a frozen copy of its argv would drift on every fork
# update. The llama.cpp era's flags would also silently no-op here.
grep -q 'launch.sh' "$SERVE"
@test "the serve shape is direct podman run, not a launcher or shim" {
# The tcclaviger stack is fully expressed by flags; the R9V launcher,
# docker->podman shim and setup-state layering are the RETIRED shape.
grep -q 'podman run' "$SERVE"
! grep -q 'launch.sh' "$SERVE"
! grep -q 'podman-shim' "$SERVE"
! grep -q 'R9V_STATE_FILE' "$SERVE"
! grep -q -- '--model-draft' "$SERVE"
! grep -q -- '--spec-type' "$SERVE"
! grep -q -- '--mmproj' "$SERVE"
! grep -q -- '--override-tensor' "$SERVE"
! grep -q 'podman run' "$SERVE"
}
@test "the tuned engine shape comes from the on-host setup state, not the repo" {
# The profile setup qualified the tuned shape (tiered-expert placement,
# per-rank offload budgets, MTP4, parser wiring). Copying it into
# the repo would drift; the serve script layers it and skips keys the
# env file defines, so production overrides always win.
grep -q 'R9V_STATE_FILE=/home/dank/r9v-state/q8xl/setup.json' "$ENV_FILE"
grep -q 'state must carry R9V_MODEL_DIR' "$SERVE"
grep -q 'is not ready' "$SERVE"
grep -q 'key not in env_keys' "$SERVE"
@test "the qualified engine shape is in the env file" {
# 2026-09-29 boot probe: MAXLEN 200000 x SEQS 3 (600k total) at KV q8/q8
# fits — the engine measured "GPU KV cache size: 729,545 tokens,
# Maximum concurrency for 200,000 tokens per request: 3.65x" at
# KV_BYTES 7.2 GiB per rank. KV_BYTES is PER RANK: sizing it to the
# global requirement OOMs the worker (measured twice). fp8 KV is the
# requested q8/q8 (e4m3 for both K and V together). MTP4 spec decode,
# util 0.98 (0.99 fails the free-memory wall).
grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
}
@test "the serve script gates the pinned image, PLE bytes and GGUF shards" {
# The image is answered by its OCI manifest digest (podman keeps the
# manifest digest in RepoDigests, not as the image ID — the shim's ID
# law). The PLE payload is checked at its pinned byte count
# (stat -c %s, never wc -c). A missing shard means a guaranteed
# long-timeout boot instead of a fast failure.
grep -q 'R9V_PLE_EXPECTED_BYTES' "$SERVE"
grep -q 'stat -c %s' "$SERVE"
grep -q 'R9V_TARGET_SHARD2_REL' "$SERVE"
grep -q 'R9V_MMPROJ_REL' "$SERVE"
@test "the serve script wires the engine flags from the env" {
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
grep -q -- '--tensor-parallel-size 2' "$SERVE"
grep -q 'num_speculative_tokens' "$SERVE"
grep -q -- '--enable-prefix-caching' "$SERVE"
grep -q -- '--trust-remote-code' "$SERVE"
}
@test "expert prefetch is absent BY LAW, not by omission" {
# Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
# prefetched slabs (36% precision) — roughly half the gather traffic was
# waste, and the "savings" would have been installs anyway. Deleting the
# flag moved decode 26.8 -> 40.6 tok/s. Any future flag must come with a
# fresh A/B, so the serve script must NOT wire it unconditionally.
! grep -q -- '--expert-prefetch' "$SERVE"
! grep -q 'expert_prefetch' "$ENV_FILE"
grep -q 'PREFETCH IS ABSENT BY LAW' "$ENV_FILE"
}
@test "TunableOp stays off" {
# CLAV_TUNABLEOP_SWEEP silently skips under expert offload unless
# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics
# (repetition loops, early halts, invalid measurements).
! grep -q 'TUNABLEOP' "$ENV_FILE"
! grep -q 'TUNABLEOP' "$SERVE"
}
@test "the serve script gates the digest-pinned image and the payload" {
# The image is pinned by OCI manifest digest (the floating :latest moved
# once already). The payload gate checks config.json and the exact shard
# count from the env — a missing shard means a guaranteed long-timeout
# boot instead of a fast failure.
grep -q 'FRANK_PRIMARY_IMAGE' "$SERVE"
grep -q 'RepoDigests' "$SERVE"
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$SERVE"
grep -q 'config.json' "$SERVE"
grep -q '/dev/kfd' "$SERVE"
grep -q -- '--device /dev/dri' "$SERVE"
! grep -q 'tcclaviger/vllm:latest' "$ENV_FILE"
}
@test "the serve script clears the stale container and self-heals via docker wait" {
# launch.sh detaches and refuses an existing container name; a crashed
# unit can leave the name registered forever after. Clear our own name
# first (idempotent), then block on the container so a crash is a
# nonzero ExecStart.
grep -q 'docker rm -f "$R9V_CONTAINER_NAME"' "$SERVE"
grep -q 'exec docker wait "$R9V_CONTAINER_NAME"' "$SERVE"
@test "the serve script clears stale containers and self-heals via podman wait" {
# A crashed unit can leave the container name registered forever after;
# the legacy identities (the retired R9V container and the ad hoc
# experiment container) hold the same GPUs and die here too. Then block
# on the container so a crash is a nonzero ExecStart.
grep -q 'FRANK_PRIMARY_LEGACY_CONTAINERS' "$ENV_FILE"
grep -q 'frank-primary-r9v' "$ENV_FILE"
grep -q 'qwen-fp8' "$ENV_FILE"
grep -q 'exec podman wait "$FRANK_PRIMARY_CONTAINER"' "$SERVE"
}
@test "frank-primary holds the single fleet port under its own alias" {
# The scratch identity (r9v-qwen38-flash-next on 127.0.0.1:7772) must
# never leak into production: LAN bind, :7770, the harness's model id.
grep -q '^R9V_HOST_BIND=0.0.0.0$' "$ENV_FILE"
grep -q '^R9V_HOST_PORT=7770$' "$ENV_FILE"
grep -q '^R9V_SERVED_MODEL_NAME=frank-primary$' "$ENV_FILE"
grep -q '^R9V_CONTAINER_NAME=frank-primary-r9v$' "$ENV_FILE"
grep -q '^R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0$' "$ENV_FILE"
# The scratch identity (qwen3.8-flash-next on :7773) must never leak
# into production: LAN publish of :7770, the harness's model id.
grep -q '^FRANK_PRIMARY_PUBLISH=7770:7770$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SERVED_NAME=frank-primary$' "$ENV_FILE"
}
@test "the model id matches the payload's served alias wiring" {
# The payload dir and its shard count are the fetch script's output
# layout; the fetch script must write where the env points.
grep -q '^FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
grep -q 'Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
grep -q 'orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8' "$DOWNLOAD"
}
@test "no bare HIP-reserved env var is defined" {
# HIP reads bare TEMP for its JIT cache path (2026-08-13: TEMP=1.0
# segfaulted every load), and HOST/GPU_MAX_HEAP_SIZE are the same
# class. The bind address is R9V_HOST_BIND, not HOST.
# class. Every key here is FRANK_PRIMARY_-prefixed.
! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
! grep -qE '^(export )?(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$SERVE"
}
@test "the deployer gates the setup state, image digest, PLE bytes and shards" {
# The deployer refuses when the on-host setup state is missing or not
# ready, when the pinned image is absent from rootless storage, when
# the PLE payload is off its pinned byte count, or when any GGUF shard
# or the mmproj is missing — then clears stale containers, waits out
# the port, relinks/enables/restarts the root unit and health-gates it.
grep -q 'LLM_PRIMARY_STATE=/home/dank/r9v-state/q8xl/setup.json' "$REMOTE_DEPLOY"
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-r9v' "$REMOTE_DEPLOY"
grep -q 'is not ready' "$REMOTE_DEPLOY"
@test "the deployer gates the image digest and payload, then health-checks" {
# The deployer refuses when the pinned image is absent from rootless
# storage or the payload is missing/wrong-sharded — then clears stale
# containers (including the legacy identities), waits out the port,
# relinks/enables/restarts the root unit and health-gates it.
grep -q 'LLM_PRIMARY_CONTAINER=frank-primary-qwen' "$REMOTE_DEPLOY"
grep -q 'FRANK_PRIMARY_IMAGE' "$REMOTE_DEPLOY"
grep -q "podman images --format '{{.RepoDigests}}'" "$REMOTE_DEPLOY"
grep -q 'stat -c %s' "$REMOTE_DEPLOY"
grep -q 'R9V_TARGET_SHARD2_REL' "$REMOTE_DEPLOY"
grep -q 'podman rm -f "$LLM_PRIMARY_CONTAINER"' "$REMOTE_DEPLOY"
grep -q 'FRANK_PRIMARY_MODEL_SHARDS' "$REMOTE_DEPLOY"
grep -q 'config.json' "$REMOTE_DEPLOY"
grep -q 'qwen-fp8' "$REMOTE_DEPLOY"
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
grep -q 'relink_unit frank-primary' "$REMOTE_DEPLOY"
grep -q 'systemctl enable frank-primary.service' "$REMOTE_DEPLOY"