frank-medium: llama.cpp layer split across both R9700s, DFlash2 drafting
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
This commit is contained in:
Frank
@@ -21,10 +21,10 @@ load helpers
|
||||
[[ "$output" != *"tests/homepage_pages.bats"* ]]
|
||||
}
|
||||
|
||||
@test "changed-test wrapper maps the frank-medium quadlet to its focused tests" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths dankserver/frank-medium/quadlet/frank-medium.container
|
||||
@test "changed-test wrapper maps the frank-medium trio to its focused tests" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/frank-medium.env
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/frank_medium_quadlet.bats"* ]]
|
||||
[[ "$output" == *"tests/frank_medium_unit.bats"* ]]
|
||||
[[ "$output" == *"tests/deployment_scoping.bats"* ]]
|
||||
[[ "$output" != *"tests/llm_units.bats"* ]]
|
||||
[[ "$output" != *"tests/homepage_pages.bats"* ]]
|
||||
}
|
||||
|
||||
@@ -46,13 +46,20 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "deployment planner maps the frank-medium service dir to frank-medium only" {
|
||||
@test "deployment planner maps the frank-medium scripts trio to frank-medium only" {
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
dankserver/frank-medium/quadlet/frank-medium.container \
|
||||
dankserver/frank-medium/radiance-entrypoint.sh \
|
||||
dankserver/frank-medium/frank-medium-healthcheck.py
|
||||
scripts/frank-medium.env scripts/frank-medium.service scripts/serve-frank-medium.sh
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-medium" ]
|
||||
|
||||
# The path is not the assertion: with the vllm-radiance service dir
|
||||
# retired into the trash, a planner rule still naming dankserver/frank-medium
|
||||
# would plan nothing and look identical to a correct one. Assert the
|
||||
# invariant (the dir is gone) as well as the mapping above.
|
||||
[ ! -d "$REPO_ROOT/dankserver/frank-medium" ]
|
||||
run python3 "$PLANNER" --target dankserver --paths dankserver/frank-medium/quadlet/frank-medium-a.container
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "the small endpoint is Dankclaw's again, and Dankserver's copy stays retired" {
|
||||
|
||||
Executable
+163
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-medium mainline-llama.cpp root unit on dankserver.
|
||||
# frank-medium became a native llama.cpp service on 2026-09-16, replacing the
|
||||
# vllm-radiance replica pair + routing proxy that served this endpoint for a
|
||||
# day. What this seam pins is what the flags alone do not say: boot persistence
|
||||
# for a NATIVE runtime (no generated quadlet is involved), that BOTH cards work
|
||||
# on ONE model through a layer split with no collective in the path, that the
|
||||
# DFlash2 drafter is wired with a depth its block size actually allows, that the
|
||||
# runtime and payload are pinned rather than floated, and that one slot's KV
|
||||
# still fits a 32 GiB card beside the weights it holds.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-medium.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-medium.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-medium.env"
|
||||
BUILD="$REPO_ROOT/scripts/build-llama-rdna4.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-medium-gguf.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
# Measured constants (rocm-smi / HF blob sizes), used by the fit arithmetic.
|
||||
CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it
|
||||
MODEL_BYTES=31457990112 # UD-Q8_K_XL target
|
||||
DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
|
||||
|
||||
@test "frank-medium's unit, serve script, env and its build/fetch scripts exist" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$BUILD" ]
|
||||
[ -f "$FETCH" ]
|
||||
}
|
||||
|
||||
@test "frank-medium is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-medium runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-medium.sh$' "$UNIT"
|
||||
grep -q 'FRANK_MEDIUM_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-medium.env' "$UNIT"
|
||||
}
|
||||
|
||||
@test "both R9700s serve ONE model, layer-split inside one process" {
|
||||
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
|
||||
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
|
||||
grep -q '^TENSOR_SPLIT=1,1$' "$ENV_FILE"
|
||||
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
|
||||
grep -q -- '--device "$DEVICES"' "$SERVE"
|
||||
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
|
||||
# Tensor split is the one shape this host cannot run: it needs a cross-card
|
||||
# collective, and the second card cannot dispatch one (see the env file).
|
||||
! grep -q '^SPLIT_MODE=tensor$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the iGPU is hidden so no layer can land on system RAM" {
|
||||
grep -q '^HIP_VISIBLE_DEVICES=0,1$' "$ENV_FILE"
|
||||
grep -q '^export HIP_VISIBLE_DEVICES$' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the DFlash2 drafter is a draft model at a depth its block allows" {
|
||||
grep -q '^SPEC_TYPE=draft-dflash$' "$ENV_FILE"
|
||||
grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DFlash2-Q8_0\.gguf$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
grep -q -- '--spec-draft-model "$DRAFT_MODEL"' "$SERVE"
|
||||
# DFlash2 drafts into a block of 8, so at most 7 draft tokens per
|
||||
# verification step -- llama.cpp asserts n_max <= block_size - 1.
|
||||
local n_max
|
||||
n_max="$(sed -nE 's/^SPEC_DRAFT_N_MAX=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 7 ]
|
||||
}
|
||||
|
||||
@test "the runtime is pinned to a commit and to gfx1201 alone" {
|
||||
# A floating ref would silently move the service between engine versions,
|
||||
# and an unpinned arch would build for a sibling GPU that loads and then
|
||||
# segfaults on these cards.
|
||||
grep -Eq '^LLAMA_REF="\$\{LLAMA_REF:-[0-9a-f]{40}\}"$' "$BUILD"
|
||||
grep -Eq '^HIP_ARCH="\$\{HIP_ARCH:-gfx1201\}"$' "$BUILD"
|
||||
grep -q 'GGML_HIP=ON' "$BUILD"
|
||||
grep -q 'frank-medium.env' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the payload is fetched resumably and verified by size" {
|
||||
grep -q 'UD-Q8_K_XL\.gguf' "$FETCH"
|
||||
grep -q 'Qwen3\.8-27B-DFlash2-Q8_0\.gguf' "$FETCH"
|
||||
grep -q 'curl -fL -C -' "$FETCH"
|
||||
grep -q 'verifying sizes' "$FETCH"
|
||||
}
|
||||
|
||||
@test "one slot's KV still fits a card beside the weights that card holds" {
|
||||
local ctx parallel cache_k cache_v
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
# qwen35 spends KV on its 16 full-attention layers only: 2 (K,V) x 4 heads
|
||||
# x 256 dim x 2 B = 64 KiB per token at f16. Worst case for the fit is ONE
|
||||
# card carrying the drafter and every attention layer's KV, with ~2 GiB of
|
||||
# compute buffers: half the weights, plus that, must stay inside the card.
|
||||
local per_card
|
||||
per_card=$(( MODEL_BYTES / 2 + DRAFT_BYTES + ctx * 65536 + 2 * 1024 * 1024 * 1024 ))
|
||||
[ "$per_card" -le "$CARD_BYTES" ]
|
||||
}
|
||||
|
||||
@test "frank-medium is text only: the vision tower stays on frank-small" {
|
||||
grep -q '^NO_MMPROJ=1$' "$ENV_FILE"
|
||||
grep -q -- '--no-mmproj' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the fleet endpoint and the predecessor's sampler defaults survive" {
|
||||
grep -q '^ALIAS=frank-medium$' "$ENV_FILE"
|
||||
grep -q '^LISTEN_PORT=7770$' "$ENV_FILE"
|
||||
grep -q -- '--alias "$ALIAS"' "$SERVE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint for LLM fleet metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
# Requests that send no sampler fields get these instead of an error.
|
||||
grep -q '^TEMPERATURE=0.7$' "$ENV_FILE"
|
||||
grep -q '^TOP_P=0.95$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing payload" {
|
||||
grep -q 'deploy_llm_medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-medium.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-medium.service' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'build-llama-rdna4.sh' "$REMOTE_DEPLOY"
|
||||
grep -q 'download-frank-medium-gguf.sh' "$REMOTE_DEPLOY"
|
||||
grep -q '\[\[ "$missing" -eq 0 \]\] || return 1' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the vLLM quadlets and frees the port they held" {
|
||||
grep -q 'retire_vllm_medium_quadlets' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user stop frank-medium-a.service frank-medium-b.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
# No image left to tag: the runtime is a native binary now.
|
||||
# ...and no image work or quadlet install of the predecessor survives.
|
||||
! grep -q 'install_quadlet frank-medium-a' "$REMOTE_DEPLOY"
|
||||
! grep -q 'podman tag docker.io/stilldeadcode' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the retired vLLM deployment is gone from the repo, not merely unmapped" {
|
||||
[ ! -d "$REPO_ROOT/dankserver/frank-medium" ]
|
||||
[ ! -f "$REPO_ROOT/tests/frank_medium_quadlet.bats" ]
|
||||
# ...and the archive is present, so the two assertions above cannot pass
|
||||
# because a path went missing rather than because it stayed missing.
|
||||
[ -f "$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-a.container" ]
|
||||
[ -f "$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance/radiance-entrypoint.sh" ]
|
||||
}
|
||||
|
||||
@test "the serve script refuses to start on an incomplete host" {
|
||||
grep -q 'frank-medium: missing\|missing executable runtime\|missing or unreadable' "$SERVE"
|
||||
grep -q 'missing or unreadable target model' "$SERVE"
|
||||
grep -q 'missing or unreadable draft model' "$SERVE"
|
||||
}
|
||||
Reference in New Issue
Block a user