frank-medium: llama.cpp layer split across both R9700s, DFlash2 drafting

The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.

llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.

Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.

Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
This commit is contained in:
Frank
2026-09-16 14:59:23 -07:00
parent 49e8dfff66
commit 05c0d3a00a
21 changed files with 608 additions and 72 deletions
+11 -6
View File
@@ -353,7 +353,6 @@ def services_for_path(path: str, target: str) -> set[str]:
"copyparty": {"copyparty"},
"plex": {"plex"},
"tautulli": {"tautulli"},
"frank-medium": {"frank-medium"},
}
_add(services, *quadlet_services.get(parts[1], set()))
if len(parts) >= 2 and parts[1] == "observability":
@@ -367,11 +366,11 @@ def services_for_path(path: str, target: str) -> set[str]:
_add(services, "node-exporter")
return services
if len(parts) >= 2 and parts[1] == "frank-medium":
# The vllm-radiance runtime since 2026-09-15: the quadlet file
# above, plus the entrypoint and healthcheck it mounts straight
# out of the checkout. Every file in the service dir is runtime
# state.
_add(services, "frank-medium")
# frank-medium is a ROOT llama.cpp unit since 2026-09-16: its
# runtime is a native gfx1201 binary, not a container, so there is
# nothing under dankserver/frank-medium/ to plan any more. The
# retired vllm-radiance quadlets live in .AGENTTRASH/, which is
# archive material and plans nothing.
return services
return services
@@ -398,6 +397,12 @@ def services_for_path(path: str, target: str) -> set[str]:
# trio that lived here for two days is in scripts/.AGENTTRASH/,
# which is archive material and plans nothing.
_add(services, "frank-small")
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
# files here the same way. The files that BUILD and FETCH its
# runtime and payload are host-side one-offs: they change no
# deployed state, so they select nothing.
_add(services, "frank-medium")
# Deployment/test/benchmark scripts and other repo tooling do not
# restart a runtime service.
return services