frank-medium: llama.cpp layer split across both R9700s, DFlash2 drafting
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
This commit is contained in:
Frank
Executable
+85
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""bench-vllm-corpus.py — build a code-shaped benchmark corpus from the repo.
|
||||
|
||||
Why this exists: `vllm bench serve --dataset-name random` produces synthetic
|
||||
token soup, and speculative decoding accepts it nothing like real code — measured
|
||||
on frank-medium, random prompts accepted 2.9-3.6 tokens/update against the ~5.8
|
||||
the upstream reference reports on a code mix, which makes the random corpus
|
||||
under-report throughput by a wide margin and would pick the wrong SPEC depth.
|
||||
|
||||
The prompts are the shape agents actually send: a repository artifact plus an
|
||||
instruction that requires reading it. Output length is fixed at bench time
|
||||
(`--custom-output-len`), so the corpus carries prompts only.
|
||||
|
||||
Usage:
|
||||
bench-vllm-corpus.py [repo-root] [out.jsonl]
|
||||
|
||||
Then feed it to the endpoint (see scripts/bench-vllm-serve.sh, which passes the
|
||||
tokenizer the client needs and salts prompts so no run inherits a warm prefix
|
||||
cache). Methodology and the numbers these corpora produced are in
|
||||
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
REPO = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "/home/dank/serverAppConfigs")
|
||||
OUT = pathlib.Path(sys.argv[2] if len(sys.argv) > 2 else "/home/dank/bench/corpus-code.jsonl")
|
||||
|
||||
TARGETS = [
|
||||
"scripts/deploy-remote.sh",
|
||||
"scripts/deploy-plan.py",
|
||||
"scripts/deploy.sh",
|
||||
"dankserver/frank-medium/radiance-entrypoint.sh",
|
||||
"dankserver/frank-medium/quadlet/frank-medium-a.container",
|
||||
"dankserver/frank-medium/proxy/affinity-proxy.py",
|
||||
"dankserver/frank-medium/proxy/routing.py",
|
||||
"tests/frank_medium_quadlet.bats",
|
||||
"tests/deployment_scoping.bats",
|
||||
"dankclaw/hermes/quadlet/hermes.container",
|
||||
"dankclaw/anubis/quadlet/anubis.container",
|
||||
"shared/prometheus/prometheus.yml",
|
||||
"shared/prometheus/rules/llm-fleet.yml",
|
||||
"shared/loki/loki-config.yaml",
|
||||
"dankclaw/compose.yaml",
|
||||
"dankserver/copyparty/quadlet/copyparty.container",
|
||||
]
|
||||
|
||||
PROMPTS = [
|
||||
"Read this repository artifact and summarise what it does, then list every "
|
||||
"assumption it makes about the host it runs on.\n\n```\n{body}\n```",
|
||||
"Here is a file from our homelab repository. Identify any bug, race or "
|
||||
"missing error path, and show the minimal patch that fixes it.\n\n```\n{body}\n```",
|
||||
"Explain this configuration to a new operator, then rewrite it with "
|
||||
"comments that state the constraint each line protects.\n\n```\n{body}\n```",
|
||||
"This file changed in a recent deploy. Write the changelog entry and a test "
|
||||
"that would fail if the change regressed.\n\n```\n{body}\n```",
|
||||
]
|
||||
|
||||
MAX_LINES = 220
|
||||
|
||||
|
||||
def main() -> int:
|
||||
samples = []
|
||||
for i, rel in enumerate(TARGETS):
|
||||
path = REPO / rel
|
||||
if not path.is_file():
|
||||
continue
|
||||
body = "\n".join(path.read_text(errors="replace").splitlines()[:MAX_LINES])
|
||||
samples.append({"prompt": PROMPTS[i % len(PROMPTS)].format(body=body)})
|
||||
if not samples:
|
||||
print(f"no corpus sources found under {REPO}", file=sys.stderr)
|
||||
return 1
|
||||
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
with OUT.open("w") as fh:
|
||||
for sample in samples:
|
||||
fh.write(json.dumps(sample) + "\n")
|
||||
print(f"wrote {len(samples)} prompts to {OUT}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/bin/bash
|
||||
# bench-vllm-serve.sh — concurrency sweep against a vLLM endpoint.
|
||||
#
|
||||
# Usage: bench-vllm-serve.sh <tag> <port> [concurrency ...]
|
||||
# e.g. bench-vllm-serve.sh baseline 7770 1 4 8
|
||||
#
|
||||
# Runs `vllm bench serve` from inside the serving image (so the client matches
|
||||
# the server build) through an OpenAI-compatible endpoint, on the code corpus
|
||||
# scripts/bench-vllm-corpus.py writes.
|
||||
#
|
||||
# Two traps this script exists to encode, both of which produced silent garbage
|
||||
# during the 2026-09-16 frank-medium work:
|
||||
#
|
||||
# 1. The tokenizer must be passed explicitly and /models mounted. With only
|
||||
# `--model frank-medium` the client tries to resolve that alias on
|
||||
# HuggingFace and dies before sending a request — a run that looks like a
|
||||
# benchmark and reports nothing.
|
||||
# 2. `--ignore-eos` is deliberate: it measures decode throughput at a fixed
|
||||
# output length. Drop it when you care about real stop behaviour, but then
|
||||
# do not compare the numbers with anything measured under it.
|
||||
#
|
||||
# Also note the corpora are code-shaped on purpose (see the generator's header):
|
||||
# synthetic random prompts misreport speculative acceptance badly.
|
||||
#
|
||||
# Methodology, and the numbers these runs produced on 2x R9700, are in
|
||||
# serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
||||
set -u
|
||||
|
||||
TAG=${1:?usage: bench-vllm-serve.sh <tag> <port> [concurrency ...]}
|
||||
PORT=${2:?usage: bench-vllm-serve.sh <tag> <port> [concurrency ...]}
|
||||
shift 2
|
||||
CONCS=("$@")
|
||||
[ "${#CONCS[@]}" -eq 0 ] && CONCS=(1 4 8)
|
||||
|
||||
IMG=${IMG:-localhost/radiance-vllm:0.9.3}
|
||||
MODELS_DIR=${MODELS_DIR:-/home/dank/models}
|
||||
TOK=${TOK:-/models/Qwen3.8-27B-Uncensored-MXFP4-awq}
|
||||
BENCH_DIR=${BENCH_DIR:-/home/dank/bench}
|
||||
CORPUS=${CORPUS:-$BENCH_DIR/corpus-code.jsonl}
|
||||
OUT=${OUT:-$BENCH_DIR/$TAG}
|
||||
mkdir -p "$OUT"
|
||||
|
||||
if [ ! -f "$CORPUS" ]; then
|
||||
echo "corpus $CORPUS is missing; run scripts/bench-vllm-corpus.py first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for c in "${CONCS[@]}"; do
|
||||
echo "== $TAG conc=$c (port $PORT)"
|
||||
podman run --rm --network=host -v "$MODELS_DIR":/models:ro -v "$BENCH_DIR":/bench:ro \
|
||||
-e HF_HUB_OFFLINE=1 --entrypoint bash "$IMG" -lc "
|
||||
vllm bench serve --backend openai-chat \
|
||||
--base-url http://127.0.0.1:$PORT --endpoint /v1/chat/completions \
|
||||
--model frank-medium --tokenizer $TOK \
|
||||
--dataset-name custom --dataset-path /bench/$(basename "$CORPUS") \
|
||||
--custom-output-len 512 --num-prompts 16 --max-concurrency $c \
|
||||
--disable-shuffle --seed 0 --ignore-eos \
|
||||
--percentile-metrics ttft,tpot --metric-percentiles 50,99 \
|
||||
--disable-tqdm 2>&1 | grep -E 'Successful requests|Benchmark duration|Output token throughput|Total Token throughput|Mean TTFT|P50 TTFT|P50 TPOT|Acceptance rate|Acceptance length'
|
||||
" 2>&1 | sed 's/^/ /'
|
||||
echo
|
||||
done
|
||||
echo "results for $TAG written under $OUT"
|
||||
Reference in New Issue
Block a user