dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-medium): cut the endpoint over to radiance TP=2

Browse Source 
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:

  arm I  llama.cpp layer split 32:33 + DFlash2, Q8_0  geo decode  45.7 t/s, TTFT 0.72 s, burst-4  71.7 t/s
  arm L  llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4  54.3 t/s
  arm R  radiance vLLM TP=2 + DFlash2-FP8, MXFP4      geo decode  95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s

Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.

  - un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
    it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
    The replica pair and its affinity proxy stay archived; one process owns
    both cards now.
  - retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
    bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
    rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
    stay: they are the tooling that rollback needs.
  - deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
    re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
    the drafter, the pinned patch clone and the patched libr4d; retires the
    llama root unit into the home trash instead of relinking it; then
    install_quadlet + wait_container_healthy 1800s (first start compiles).
    retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
    a/b/proxy trio only, since frank-medium is installed again.
  - planner + test-changed map dankserver/frank-medium/** to frank-medium
    and its seam; the scripts-trio mapping is gone with the files.
  - tests/frank_medium_unit.bats rewritten for the container contract: boot
    persistence of a GENERATED unit (no enable on this host's v1 generator),
    TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
    the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
    was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
    contract, and the archive invariants.
  - homepage card + docs updated, and they were stale twice over: they still
    described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.

Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
This commit is contained in:
Frank
2026-09-20 18:15:27 -07:00
parent bcdd898750
commit c2116658f8
15 changed files with 345 additions and 433 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/.AGENTTRASH/20260920-llama-layersplit/bench-spec-ab.py.llamacpp-layersplit
Executable
+309
View File
@@ -0,0 +1,309 @@
#!/usr/bin/env python3
"""bench-spec-ab.py — settle "which proposer should this endpoint use?" with data.
Runs one payload through several speculative-decoding configurations of the same
binary, one at a time, with the unit stopped between runs: on a single-GPU host
there is no room for two copies of a 29 GiB model, and the benchmarking policy
applies. Every configuration gets identical prompts at temperature 0, and the
greedy text is compared across them — speculation may buy speed, never change
the answer.
bench-spec-ab.py <env-file> <serve-script> [options]
bench-spec-ab.py /home/dank/serverAppConfigs/scripts/frank-medium.env \
/home/dank/serverAppConfigs/scripts/serve-frank-medium.sh \
--configs "dflash=draft-dflash@7;ngram=ngram-mod@64;chain=draft-dflash,ngram-mod@7"
Config syntax: `name=SPEC_TYPE[@n_max][|ENV=VAL|ENV=VAL]`, entries joined by `;`.
The systemd unit is restored in a finally block, so it comes back on whatever the
env file says even if a configuration crashes the harness.
THREE SHAPES, because a single one answers the wrong question. A proposer can
only draft text it has seen, so the shape decides the verdict:
prose a 1k prompt asking for one sentence. Nothing to copy and nothing
repetitive, so only a model-backed proposer can help: this is the
shape that compares drafters.
echo a ~1.5k passage of unique lines, "repeat it back verbatim". The
output IS the prompt, which is ngram-mod's home turf.
structured "25 JSON records, one per line" — repetitive OUTPUT with no prompt
overlap, which is what an ngram matcher is really for in an
agentic fleet (config edits, tables, record lists).
echo and structured disable thinking (`chat_template_kwargs`), otherwise the
model thinks the answer into `reasoning_content`, `content` comes back empty and
the cross-configuration identity check passes vacuously — which is exactly how
this harness first "verified" an echo it never looked at.
Results on Dankserver (frank-medium payload, 2026-09-16, temperature 0). Decode:
shape dflash=7 ngram-mod(48/64) chain (tok/s)
prose 1k/512 43.8 17.7 (0 drafts) 43.7
echo 1.5k/400 76.8 102.5 200.6
structured/400 91.1 17.8 (0 drafts) 91.2
identity 70.4 17.8 (0 drafts) 70.4
`draft-mtp,draft-dflash` fails at load (a chain has ONE draft context and MTP
builds against whatever model that context holds); ngram-mod has no such
constraint — it is draftless and joins the chain unconditionally.
"""
import argparse
import json
import os
import re
import subprocess
import sys
import time
import urllib.request
FILLER = ("The quick brown fox jumps over the lazy dog near the river bank. "
"A cache is a component that stores data so future requests are faster. ")
TOKENS_PER_REP = 28
UNIT = "frank-medium.service"
MODEL = "frank-medium"
SPEC_GREP = ("adding speculative implementation|- n_max=|- block_size=|creating MTP|"
"draft acceptance")
DEFAULT_CONFIGS = [
("none", {"SPEC_TYPE": "none"}),
("mtp-n3", {"SPEC_TYPE": "draft-mtp", "SPEC_DRAFT_N_MAX": "3"}),
("mtp-n7", {"SPEC_TYPE": "draft-mtp", "SPEC_DRAFT_N_MAX": "7"}),
("dflash-n7", {"SPEC_TYPE": "draft-dflash", "SPEC_DRAFT_N_MAX": "7"}),
]
WORDS = ["latency", "throughput", "cache", "adapter", "scheduler", "allocator",
"queue", "kernel", "buffer", "shard", "replica", "window"]
def echo_passage(lines=90):
"""Unique lines: no long span repeats inside the prompt, so an n-gram can
only fire when the OUTPUT copies the INPUT."""
return "\n".join(
"Record %03d: the %s of subsystem %d reports state %d with checksum %05d."
% (i, WORDS[i % len(WORDS)], i % 7, i % 3, (i * 7919) % 100000)
for i in range(lines))
ECHO = (echo_passage() + "\n\nRepeat the passage above verbatim, every line, exactly as "
"written, with no commentary before or after it.")
STRUCTURED = ("Output exactly 25 JSON records, one per line, no prose and no code fence. "
'Each line must look like {"id": N, "state": "active", "region": "us-west-2"} '
"with N running from 1 to 25.")
PROSE = FILLER * (1000 // TOKENS_PER_REP) + "\n\nReply with a single short sentence."
IDENTITY = "List the first five prime numbers, comma separated, then stop."
NO_THINK = {"chat_template_kwargs": {"enable_thinking": False}}
def log(msg, logfile):
line = "[" + time.strftime("%H:%M:%S") + "] " + str(msg)
print(line, flush=True)
with open(logfile, "a") as fh:
fh.write(line + "\n")
def sh(cmd):
return subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout.strip()
def parse_configs(spec):
"""`name=SPEC_TYPE[@n_max][|KEY=VALUE]...`, entries joined by ';'.
KEY=VALUE parts override any env file key, which is how an impl's own window
(SPEC_NGRAM_N_MIN/MAX, for instance) gets swept without hand-editing env
files."""
out = []
for chunk in spec.split(";"):
chunk = chunk.strip()
if not chunk:
continue
name, _, rest = chunk.partition("=")
over = {}
if "|" in rest:
rest, _, extras = rest.partition("|")
for item in extras.split("|"):
if "=" in item:
k, _, v = item.partition("=")
over[k.strip()] = v.strip()
n_max = None
if "@" in rest:
rest, _, n_max = rest.partition("@")
over["SPEC_TYPE"] = rest.strip()
if n_max:
over["SPEC_DRAFT_N_MAX"] = n_max.strip()
out.append((name.strip(), over))
return out
def write_env(src, name, overrides):
"""The env file with this configuration's overrides applied."""
path = "/tmp/spec-ab-" + name.replace("/", "_") + ".env"
with open(src) as fh:
lines = fh.read().splitlines()
out = []
for line in lines:
for k, v in overrides.items():
if line.startswith(k + "="):
line = k + "=" + v
break
out.append(line)
with open(path, "w") as fh:
fh.write("\n".join(out) + "\n")
return path
def post(base, body, timeout=1800):
req = urllib.request.Request(
base + "/v1/chat/completions", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.load(r)
def vram():
out = sh("rocm-smi --showmeminfo vram 2>/dev/null | grep 'Used Memory'")
return [round(int(x) / 1e9, 2) for x in re.findall(r":\s+(\d+)\s*$", out, re.M)][:2]
def wait_health(base, timeout=420):
t0 = time.time()
while time.time() - t0 < timeout:
if not sh("pgrep -x llama-server >/dev/null && echo up"):
return False
try:
with urllib.request.urlopen(base + "/health", timeout=5) as r:
if r.status == 200 and json.load(r).get("status") == "ok":
return True
except Exception:
time.sleep(3)
return False
def stop_unit():
sh("sudo systemctl stop " + UNIT)
sh("pkill -x llama-server")
for _ in range(40):
if not sh("pgrep -x llama-server || true"):
return
time.sleep(1)
def restore_unit(base, logfile):
"""Bring the endpoint back on the deployed configuration. Runs in a
finally, because a harness that can leave production down is a hazard."""
log("restoring " + UNIT, logfile)
sh("sudo systemctl start " + UNIT)
for _ in range(120):
try:
with urllib.request.urlopen(base + "/health", timeout=5) as r:
if json.load(r).get("status") == "ok":
log(UNIT + " healthy again", logfile)
return True
except Exception:
time.sleep(5)
log("WARNING: " + UNIT + " did not come back healthy -- check it by hand", logfile)
return False
def run_shape(base, prompt, max_tokens, label, extra=None):
body = {"model": MODEL, "messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens, "temperature": 0.0}
if extra:
body.update(extra)
d = post(base, body)
tm = d["timings"]
msg = d["choices"][0]["message"]
content = (msg.get("content") or "").strip()
row = {"shape": label,
"prompt_n": tm["prompt_n"],
"prefill_tps": round(1000 * tm["prompt_n"] / tm["prompt_ms"], 1),
"predicted_n": tm["predicted_n"],
"decode_tps": round(tm["predicted_per_second"], 2),
"draft_n": tm.get("draft_n", 0),
"accepted": tm.get("draft_n_accepted", 0),
"answer": content,
"reasoning_len": len(msg.get("reasoning_content") or "")}
if label.startswith("echo"):
# How much of the answer really is the passage: a configuration that
# "wins" by answering something else is not a win.
want = {l.strip() for l in echo_passage().splitlines()}
got = [l.strip() for l in content.splitlines() if l.strip()]
row["copy_ratio"] = round(sum(1 for l in got if l in want) / len(got), 3) if got else 0.0
return row
def measure(base):
run_shape(base, FILLER * (4000 // TOKENS_PER_REP) +
"\n\nReply with a single short sentence.", 32, "warmup")
return [
run_shape(base, PROSE, 512, "prose 1k/512"),
run_shape(base, ECHO, 400, "echo 1.5k/400", NO_THINK),
run_shape(base, STRUCTURED, 400, "structured/400", NO_THINK),
run_shape(base, IDENTITY, 192, "identity"),
]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("env_file")
ap.add_argument("serve_script")
ap.add_argument("--base-url", default="http://127.0.0.1:7770")
ap.add_argument("--log", default="/tmp/bench-spec-ab.log")
ap.add_argument("--configs", default=None,
help="name=SPEC_TYPE[@n_max][|ENV=VAL] joined by ';' "
"(default: the four configs measured on 2026-09-16)")
ap.add_argument("--json", default="/tmp/bench-spec-ab.json")
args = ap.parse_args()
configs = parse_configs(args.configs) if args.configs else DEFAULT_CONFIGS
# Absolute paths: the launcher must not depend on the caller's cwd, and a
# relative path plus a `cd` to its own directory double-counts the prefix.
env_file = os.path.abspath(args.env_file)
serve_script = os.path.abspath(args.serve_script)
base, logfile = args.base_url, args.log
results = {}
log("spec A/B start: " + ", ".join(n for n, _ in configs) +
" (unit " + UNIT + " stopped for the duration)", logfile)
try:
stop_unit()
for name, over in configs:
env_path = write_env(env_file, name, over)
logf = "/tmp/spec-ab-" + name.replace("/", "_") + ".log"
sh("FRANK_MEDIUM_ENV_FILE=" + env_path + " setsid bash " + serve_script +
" > " + logf + " 2>&1 < /dev/null &")
if not wait_health(base):
log(name + ": FAILED to become healthy -- " + sh("tail -3 " + logf), logfile)
results[name] = {"error": "not healthy", "tail": sh("tail -6 " + logf)}
stop_unit()
continue
rows = measure(base)
results[name] = {"vram_gb": vram(), "rows": rows,
"spec_log": sh("grep -E '" + SPEC_GREP + "' " + logf + " | tail -4")}
for r in rows:
log(name + " | " + r["shape"] + " prefill " + str(r["prefill_tps"]) +
" tok/s | decode " + str(r["decode_tps"]) + " tok/s | draft " +
str(r["draft_n"]) + "/" + str(r["accepted"]) +
(" (" + str(round(100.0 * r["accepted"] / r["draft_n"])) + " %)"
if r["draft_n"] else "") +
(" | copy " + str(r["copy_ratio"]) if "copy_ratio" in r else "") +
(" | EMPTY ANSWER" if not r["answer"] else ""), logfile)
log(name + " | vram " + str(results[name]["vram_gb"]), logfile)
stop_unit()
finally:
restore_unit(base, logfile)
with open(args.json, "w") as fh:
json.dump(results, fh, indent=2)
for shape in ("prose 1k/512", "echo 1.5k/400", "structured/400", "identity"):
texts = [r["answer"] for v in results.values() if "rows" in v
for r in v["rows"] if r["shape"] == shape]
if any(not t for t in texts):
log("WARNING: '" + shape + "' produced an empty answer in some configuration -- "
"its identity check proves nothing", logfile)
elif len(set(texts)) > 1:
log("WARNING: configurations disagree on greedy text for '" + shape +
"' -- speculation is changing answers", logfile)
log("done; results in " + args.json, logfile)
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.env.llamacpp-layersplit
+318
View File
@@ -0,0 +1,318 @@
# Tunables for frank-medium (orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 +
# z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M drafter), mainline llama.cpp on
# Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this is
# the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
# routing proxy are retired.
#
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
# IOMMU groups -- every cross-card byte goes over PCIe. The Raphael iGPU
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
# nothing can drift onto shared system RAM.
#
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
# on a device without PCIe atomics -- the second card sits behind the B650
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
# touches it dies with hipErrorIllegalState. Measured on this box 2026-09-15
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
#
# single R9700 1160.9 / 19.42
# layer 2x 1991.2 / 18.76 (859.1 / 17.64 at depth 32768)
# tensor 2x 271.8 / 26.26 (no RCCL: butterfly fallback, -7x prefill)
#
# Layer split distributes layers in order; only the activation at the boundary
# crosses the wire, once per forward pass. That is the "pipeline parallelism"
# this service wanted, and it needs no collective at all.
#
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
#
# Model: orcarouter/Qwen3.8-27B-Uncensored-GGUF @ Q8_0, 27.0 GiB, arch qwen35
# (verified from the GGUF header 2026-09-19: 262144 trained window, 24 heads /
# 4 KV heads, nextn_predict_layers=1). The file ALSO carries the model's MTP
# head (blk.64.nextn.*), so --spec-type draft-mtp is available without a second
# copy in VRAM -- one SPEC_TYPE change, the drafter below stays on disk.
#
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q4_K_M, 1.06 GiB (operator call
# 2026-09-19: the Q4 rung -- the drafter's job is throughput, not output
# quality, and the target verifies every drafted token). 5 layers, hidden
# 5120, a block-diffusion proposal (block_size 8) under a 2048-token sliding
# window (dflash.attention.sliding_window=2048, verified from the header).
# The window is what makes it cheap HERE: its KV is bounded at 2048 positions
# (~40 MiB) instead of spanning the context (see SIZING).
#
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
# 256 head_dim. A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
# state at any depth.
#
# SIZING, the deployed shape (32:33 / ctx 327680 / 6144x512 / two slots /
# DFlash2 Q4_K_M), measured 2026-09-19 after the model swap (rocm-smi):
#
# card 0 (x16) 26.9 GiB used, 5.7 GiB free
# card 1 (x1) 27.8 GiB used, 4.8 GiB free
#
# The predecessor's measured table (rocm-smi, 2026-09-16, UD-Q8_K_XL target)
# stays as the calibration history:
#
# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536)
# 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start
# 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB
# 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB
# 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB
# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB
# 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB
# 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired)
#
# ...and the same deployed shape re-measured at ubatch 512, after llama-bench
# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the
# batch block below):
#
# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only
# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB
#
# ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid
# stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a
# DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB
# (0.45 of weights plus ~1.0 GiB of its own f16 KV at 262144). Both step sizes
# were measured, which is why the balance is found by measurement rather than by
# interpolating between splits.
#
# WHY THE SPLIT IS 32:33 (operator call, 2026-09-19). The operator asked for
# the tensors BALANCED across the two cards, which for 65 layers is 32:33 --
# the most even split that exists. The predecessor's 35:30 skew existed to lean
# work onto the x16 card (ROCm0), because ROCm1's Gen4 x1 chipset path made it
# the 98 % utilization bottleneck; the measured trade of the balanced point is
# that the x1 card runs hotter while the x16 card idles a little more, and the
# operator accepted that for symmetry. Do not re-skew without asking.
#
# THE TWO CARDS' PATHS ARE NOT SYMMETRIC (measured, 2026-09-17): ROCm0
# (03:00.0) hangs off CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s);
# ROCm1 (0b:00.0) rides the B650 chipset through bridges that each negotiate
# Gen4 x1 (~1.97 GB/s). The layer-split boundary itself is not the cost
# (~5.2 MB of activation per 512-token ubatch rides the wire); the x1 card is
# COMPUTE-bound, not PCIe-bound.
#
# A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth
# request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine
# surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort,
# i.e. a fault in the cross-card boundary copy, this layer split's hot path. It
# has not reproduced since, but the signature is worth recognising -- it is NOT
# an OOM, and no amount of headroom prevents it.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
# as flags only.
# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
MODEL_DIR=/home/dank/models/Qwen3.8-27B-Uncensored-GGUF
MODEL_FILE=Qwen3.8-27B-Uncensored-Q8_0.gguf
# The f16 vision projector, same repo as the target. llama-server loads it via
# the explicit --mmproj flag (no autoload in this build); the serve script
# refuses to start without it when vision is enabled, so a missing file is a
# deploy error, not a text-only server that pretends to be multimodal.
MM_PROJ_FILE=mmproj-Qwen3.8-27B-Uncensored-f16.gguf
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
# at /v1/models and what every client asks for.
ALIAS=frank-medium
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Device placement. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
# across, and neither of the two cards has an XGMI link to the other.
HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# 32:33 -- balanced (operator call 2026-09-19): 65 layers split as evenly as
# they can be, ROCm1 (the last-layers card, which also carries the drafter and
# the output layers) holding the extra one. See the WHY THE SPLIT IS 32:33
# block above for the trade against the predecessor's 35:30 x16-lean.
TENSOR_SPLIT=32,33
NGL=999
# Auto-fit has no business resizing a context we set explicitly.
FIT=off
FLASH_ATTN=on
# 327680 positions over TWO slots, so each conversation gets 163840 --
# --parallel DIVIDES --ctx-size, so more slots would shrink every
# conversation rather than grow the cache. 163840 is inside the checkpoint's
# trained 262144 window, so the whole pool is trained territory (no rope
# extrapolation, unlike the predecessor's 393216).
CTX_SIZE=327680
PARALLEL=2
# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench,
# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps,
# one invocation per cell, the deployed split and f16 KV. --ubatch-size is the
# physical batch the graph is built at -- and the only one of the two knobs that
# costs VRAM:
#
# batch ubatch prefill tok/s GPU 0 / GPU 1 free
# 6144 1536 1164.96 +- 2.30 10.28 / 11.71 GiB
# 6144 768 1135.03 +- 1.09 11.29 / 13.28 GiB
# 6144 512 1192.93 +- 0.91 11.63 / 13.81 GiB <- deployed
# 6144 384 1068.04 +- 0.79 11.80 / 14.07 GiB
# 6144 256 1045.98 +- 1.69 11.97 / 14.33 GiB
# 4096 1536 1161.90 +- 0.96 10.28 / 11.71 GiB
# 2048 1536 1170.23 +- 0.86 10.28 / 11.71 GiB
# 2048 512 1193.88 +- 1.01 11.63 / 13.81 GiB
#
# THE PLATEAU STARTS AT 512. 512 -> 1536 buys ~2 % of prefill -- inside the
# run-to-run spread, and the 512 cells were the faster pair in BOTH batch
# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the
# deployment takes the cheap end. BELOW 512 the SMs are no longer saturated:
# 384 and 256 fall ~11 % and ~12 %. The logical batch is free either way --
# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use
# the SAME VRAM, because the compute buffers are sized by ubatch alone.
#
# The prefill figure is a property of the SHAPE, not of this knob: the same
# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and
# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two
# tables is what attending over 64k of resident KV costs. Decode is 15.88 tok/s
# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch
# generation at all.
#
# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on
# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is
# no longer a reason to want it: batch buys nothing measurable here.
BATCH_SIZE=6144
UBATCH_SIZE=512
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
NUM_THREADS=8
THREAD_BATCH=8
# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
# Context checkpoints: 0 -- no per-slot KV snapshots at all. The build's
# default is 32 (this binary's --help), so the 0 below is a deliberate
# deviation, not an inherited default: a checkpoint is a VRAM->host-RAM copy
# of part of a slot's KV, taken so a partially-matching prompt can be restored
# instead of re-prefilled. This endpoint trades that away -- its reuse comes
# from the slot prompt and from the RAM-side prompt cache below. Stated
# explicitly so the tunable stays visible rather than merely absent.
CTX_CHECKPOINTS=0
# PROMPT CACHE, HOST RAM -- this is the one cache tunable that is NOT VRAM.
# --cache-ram caps the total bytes of saved prompt states
# (server_prompt_cache, tools/server/server-task.cpp): update() evicts
# oldest-first past the cap, and alloc() REFUSES outright any single state
# bigger than the whole cap. A state is the serialized KV of a whole sequence
# plus its DeltaNet state plus the draft context's own KV, so at this shape it
# costs ~68 KiB per token --
# 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB
# plus the draft context's window-bounded KV (a ~40 MiB constant for DFlash2)
# -- which makes the cap a TOKEN budget in disguise: 16384 MiB holds ~240k
# tokens, i.e. a FULL 163840-token slot (~10.6 GiB) with ~5 GiB of the budget
# spare for a second entry.
#
# The cost is host RAM, allocated on demand and never pinned (plain heap
# vectors), so no VRAM number above moves. dankserver has 62 GiB beside the
# media-stack co-tenants, so the cache is bounded rather than unlimited (-1):
# the seam test refuses a budget over half the host's RAM.
CACHE_RAM=16384
# Sampling defaults for requests that send none -- the values the vLLM
# predecessor served on this endpoint (`--override-generation-config`), so a
# client sees no change across the runtime swap. The model card's own
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
# models.yml, so this is the floor, not the contract.
TEMPERATURE=0.7
TOP_P=0.95
TOP_K=20
MIN_P=0.0
PRESENCE_PENALTY=0.0
# Reasoning on, projected into reasoning_content (deepseek format) with no
# budget: the model decides how long to think.
REASONING=on
REASONING_FORMAT=deepseek
REASONING_BUDGET=-1
# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
CHAT_TEMPLATE_FILE=
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
# Speculative decoding: the DFlash2 sidecar (operator call 2026-09-19). The
# target file also carries its own MTP head (nextn_predict_layers=1), so
# draft-mtp is one SPEC_TYPE change away -- the drafter file stays on disk.
# n_max 7 is the measured DFlash2 winner (see the table below); the drafter
# rides ROCm1, the card that holds the last layers.
SPEC_TYPE=draft-dflash
SPEC_DRAFT_N_MAX=7
SPEC_DRAFT_NGL=999
SPEC_DRAFT_DEVICE=ROCm1
SPEC_DRAFT_TYPE_K=f16
SPEC_DRAFT_TYPE_V=f16
# DFlash2 vs the model's own MTP head -- MEASURED 2026-09-16 on the predecessor
# payload (temperature 0, 1k prompt / 512 out; the autoregressive baseline and
# both MTP depths were run in one session, same binary, same prompts):
#
# none 17.7 tok/s card1 26.07 GB
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
#
# All four produced IDENTICAL greedy text, so verification is lossless
# whichever proposer runs. DFlash2 is the faster proposer (2.46x over
# autoregressive, 34 % over the best MTP). The MTP head adds ~3.3 GB to the
# card that holds it, against 1.06 GB for the Q4_K_M drafter's file -- at the
# 327680 pool the drafter's window-bounded KV (~40 MiB) leaves headroom on
# both cards, so the speed wins.
#
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
# context type MTP requested but model doesn't contain MTP layers
# failed to create MTP context
# A chain has ONE draft context, and MTP builds against whatever model that
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
# only loaded when MTP IS the draft model ("creating MTP draft context against
# the target model ..."), so a chain of the two is a structural dead end, not a
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
# ngram-mod: the draftless n-gram matcher (CPU hash table in host RAM, no VRAM,
# no model file, no draft context). It joins the chain unconditionally --
# unlike MTP, which needs a draft context and therefore cannot share one with
# DFlash2. NOT CURRENTLY ACTIVE: SPEC_TYPE is draft-dflash alone, and the serve
# script only passes --spec-ngram-mod-* when the type contains "ngram". The
# window below is retained because the DFlash2 + ngram chain was measured free
# on prose and 2.6x on prompt-echo -- one SPEC_TYPE change brings it back.
#
# WINDOW MEASURED 2026-09-16 (chain at each window, decode tok/s):
#
# n_min/n_max prose 1k/512 prompt-echo 1.5k/400
# 48/64 43.9 200.6
# 16/32 36.0 161.0
# 8/16 36.2 113.8
#
# The long window is the right one, and not only because it wins on echo: a
# SHORT window makes the matcher propose on short fragments of ordinary prose,
# where the proposals are rejected and the verification still costs -- which is
# the "ngram hurts" result the fleet measured on other payloads twice before.
# With 48/64 the chain is byte-identical to DFlash2 alone on prose (same 119
# drafts / 49 accepted) and on the identity prompt (70/53).
#
# Do not shorten these without re-running scripts/bench-spec-ab.py: the
# no-penalty property is what makes the chain safe as a default.
SPEC_NGRAM_N_MATCH=24
SPEC_NGRAM_N_MIN=48
SPEC_NGRAM_N_MAX=64
# Vision enabled: llama-server loads mmproj-Qwen3.8-27B-Uncensored-f16.gguf
# (931 MiB, lands on ROCm0 -- the fit charges it there) via the explicit
# --mmproj flag. This keeps frank-medium multimodal across the model swap.
NO_MMPROJ=0
scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.service.llamacpp-layersplit
+20
View File
@@ -0,0 +1,20 @@
[Unit]
Description=llama.cpp Qwen3.8-27B Uncensored UD-Q8_K_XL + DFlash2 drafter (dankserver, frank-medium, 2x Radeon AI PRO R9700)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=dank
Group=dank
# The engine pins staging buffers on the first card; keep the memlock budget
# unbounded so an RLIMIT that is smaller than the split buffers cannot fail the
# load. Parity with the other LLM units on this fleet.
LimitMEMLOCK=infinity
Environment="FRANK_MEDIUM_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-medium.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-medium.sh
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
scripts/.AGENTTRASH/20260920-llama-layersplit/serve-frank-medium.sh.llamacpp-layersplit
Executable
+203
View File
@@ -0,0 +1,203 @@
#!/usr/bin/env bash
# Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as
# frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the
# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp.
#
# Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that
# bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by
# scripts/build-llama-rdna4.sh. Two features pin that ref: DFlash2
# (ggml-org/llama.cpp#27342) and qwen35 MTP. The charlie12345/ROCmFPX fork
# this deployment replaced has neither (its tip is c49ebdb, 2026-08-22).
#
# Parallelism: --split-mode layer across both cards. Tensor split would need a
# cross-card allreduce and this host cannot dispatch one -- RCCL's kernels
# require a hostcall buffer that ROCr refuses on a card without PCIe atomics,
# and the second R9700 sits behind the chipset (AtomicOpsCap: Routing-). Layer
# split crosses the wire once per forward pass instead of once per layer.
#
# The vLLM replica pair and its routing proxy were retired with this cutover:
# one process, one model, both cards.
#
# Tunables come from frank-medium.env unless FRANK_MEDIUM_ENV_FILE overrides it.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_MEDIUM_ENV_FILE:-$SCRIPT_DIR/frank-medium.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) ------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}"
: "${MODEL_DIR:=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF}"
: "${MODEL_FILE:=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
: "${DRAFT_MODEL_DIR:=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF}"
: "${DRAFT_MODEL_FILE:=Qwen3.8-27B-DFlash2-Q8_0.gguf}"
: "${ALIAS:=frank-medium}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
# Devices. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2) so nothing
# can drift onto shared system RAM; DEVICES is what llama.cpp splits across.
: "${HIP_VISIBLE_DEVICES:=0,1}"
: "${DEVICES:=ROCm0,ROCm1}"
: "${SPLIT_MODE:=layer}"
: "${TENSOR_SPLIT:=1,1}"
: "${NGL:=999}"
: "${FIT:=off}"
: "${FLASH_ATTN:=on}"
: "${CTX_SIZE:=131072}"
: "${PARALLEL:=1}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${NUM_THREADS:=8}"
: "${THREAD_BATCH:=8}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CTX_CHECKPOINTS:=8}"
: "${CACHE_RAM:=8192}"
# Sampling defaults for silent clients: the values the vLLM predecessor served.
: "${TEMPERATURE:=0.7}"
: "${TOP_P:=0.95}"
: "${TOP_K:=20}"
: "${MIN_P:=0.0}"
: "${PRESENCE_PENALTY:=0.0}"
: "${REASONING:=on}"
: "${REASONING_FORMAT:=deepseek}"
: "${REASONING_BUDGET:=-1}"
: "${CHAT_TEMPLATE_FILE:=}"
: "${CHAT_TEMPLATE_KWARGS:={\"enable_thinking\":true,\"preserve_thinking\":true}}"
# Speculative decoding: draft-dflash or draft-dspark (both are MODEL drafters --
# a file, not a flag), or draft-mtp (the target GGUF's own MTP head, no draft
# file). The drafter's own tuning knobs (n-min, p-min, p-split, poll) are left
# at the build's defaults: they were only ever passed for dspark, and dspark is
# not the deployed drafter.
: "${SPEC_TYPE:=draft-dflash}"
: "${SPEC_DRAFT_N_MAX:=7}"
: "${SPEC_DRAFT_NGL:=999}"
: "${SPEC_DRAFT_DEVICE:=}"
: "${SPEC_DRAFT_TYPE_K:=f16}"
: "${SPEC_DRAFT_TYPE_V:=f16}"
: "${SPEC_NGRAM_N_MATCH:=24}"
: "${SPEC_NGRAM_N_MIN:=48}"
: "${SPEC_NGRAM_N_MAX:=64}"
NO_MMPROJ="${NO_MMPROJ:-0}"
: "${MM_PROJ_FILE:=mmproj-BF16.gguf}"
export HIP_VISIBLE_DEVICES
MODEL="$MODEL_DIR/$MODEL_FILE"
DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE"
if [[ ! -x "$LLAMA_BIN" ]]; then
echo "missing executable runtime: $LLAMA_BIN" >&2
echo "build it with: bash $SCRIPT_DIR/build-llama-rdna4.sh" >&2
exit 1
fi
if [[ ! -r "$MODEL" ]]; then
echo "missing or unreadable target model: $MODEL" >&2
echo "fetch it with: bash $SCRIPT_DIR/download-frank-medium-gguf.sh" >&2
exit 1
fi
if [[ "$NO_MMPROJ" != "1" ]]; then
MM_PROJ="$MODEL_DIR/$MM_PROJ_FILE"
if [[ ! -r "$MM_PROJ" ]]; then
echo "vision is enabled but the projector is missing: $MM_PROJ" >&2
echo "fetch it with: bash $SCRIPT_DIR/download-frank-medium-gguf.sh" >&2
exit 1
fi
fi
if [[ -n "$CHAT_TEMPLATE_FILE" && ! -r "$CHAT_TEMPLATE_FILE" ]]; then
echo "missing or unreadable chat template: $CHAT_TEMPLATE_FILE" >&2
exit 1
fi
# --- speculative decoding -------------------------------------------------
SPEC_ARGS=()
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
if [[ "$SPEC_TYPE" == *dflash* || "$SPEC_TYPE" == *dspark* ]]; then
if [[ ! -r "$DRAFT_MODEL" ]]; then
echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
exit 1
fi
SPEC_ARGS+=(
--spec-draft-model "$DRAFT_MODEL"
--spec-draft-ngl "$SPEC_DRAFT_NGL"
--spec-draft-type-k "$SPEC_DRAFT_TYPE_K"
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
)
fi
# ngram-mod is draftless: it needs no draft context and its window is its
# own (--spec-ngram-mod-*), so it can ride in the chain beside a model
# drafter. It also costs no VRAM -- a CPU hash table.
if [[ "$SPEC_TYPE" == *ngram* ]]; then
SPEC_ARGS+=(
--spec-ngram-mod-n-match "$SPEC_NGRAM_N_MATCH"
--spec-ngram-mod-n-min "$SPEC_NGRAM_N_MIN"
--spec-ngram-mod-n-max "$SPEC_NGRAM_N_MAX"
)
fi
if [[ -n "$SPEC_DRAFT_DEVICE" ]]; then
SPEC_ARGS+=(--spec-draft-device "$SPEC_DRAFT_DEVICE")
fi
fi
# --- template + placement -------------------------------------------------
TEMPLATE_ARGS=()
if [[ -n "$CHAT_TEMPLATE_FILE" ]]; then
TEMPLATE_ARGS+=(--chat-template-file "$CHAT_TEMPLATE_FILE")
fi
TEMPLATE_ARGS+=(--chat-template-kwargs "$CHAT_TEMPLATE_KWARGS")
PLACEMENT_ARGS=(--device "$DEVICES" --split-mode "$SPLIT_MODE" --tensor-split "$TENSOR_SPLIT")
MMPROJ_ARGS=()
if [[ "$NO_MMPROJ" == "1" ]]; then
MMPROJ_ARGS=(--no-mmproj)
else
# No autoload in this build: a server without the flag starts text-only
# and answers image requests with a 500, so name the projector.
MMPROJ_ARGS=(--mmproj "$MM_PROJ")
fi
exec "$LLAMA_BIN" \
--model "$MODEL" \
--alias "$ALIAS" \
--host "$BIND_ADDR" \
--port "$LISTEN_PORT" \
--jinja \
--gpu-layers "$NGL" \
--fit "$FIT" \
--parallel "$PARALLEL" \
--ctx-size "$CTX_SIZE" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--flash-attn "$FLASH_ATTN" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--threads "$NUM_THREADS" \
--threads-batch "$THREAD_BATCH" \
--ctx-checkpoints "$CTX_CHECKPOINTS" \
--cache-ram "$CACHE_RAM" \
--cache-prompt \
--metrics \
--temp "$TEMPERATURE" \
--top-p "$TOP_P" \
--top-k "$TOP_K" \
--min-p "$MIN_P" \
--presence-penalty "$PRESENCE_PENALTY" \
--reasoning "$REASONING" \
--reasoning-format "$REASONING_FORMAT" \
--reasoning-budget "$REASONING_BUDGET" \
"${TEMPLATE_ARGS[@]}" \
"${PLACEMENT_ARGS[@]}" \
"${MMPROJ_ARGS[@]}" \
"${SPEC_ARGS[@]}"