feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain

Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256> manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape.
2026-09-20 21:53:00 -07:00
parent 2458d754ce
commit c1a4ccbbcd
8 changed files with 534 additions and 45 deletions
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).

 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | Qwen3.8-27B-Uncensored MXFP4-awq with the z-lab DFlash2-FP8 drafter (depth 7, live mean accept length ~4.1), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 943,581-token KV pool = 3.59x the window), fp8 KV, prefix caching, vision | 7770 |
+| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with MTP self-speculation (depth 8, live mean accept length 5.66/8), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 461,540-token KV pool = 1.76x the window), fp8 KV, prefix caching, vision | 7770 |
 | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |

@@ -334,18 +334,31 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
 Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
 Rollback: [[frank-small-qwen38-gsq-rco]].

-### frank-medium — radiance TP=2, DFlash2-FP8 drafter (Dankserver, ACTIVE)
+### frank-medium — radiance TP=2, MTP self-speculation (Dankserver, ACTIVE)

 vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
 `dankserver/frank-medium/quadlet/frank-medium.container`) serving
-`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4) with the
-**z-lab DFlash2 FP8 drafter** (`num_speculative_tokens: 7`, live mean accept
-~4.1) in **one tensor-parallel process across both R9700s**
+`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic
+activation quant, 30.9 GB / 7 shards) with **MTP self-speculation**
+(`num_speculative_tokens: 8`, the model's own single-layer MTP head, live mean
+accept 5.66/8) in **one tensor-parallel process across both R9700s**
 (`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8
 KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized
-to 943,581 tokens (3.59× the full window). Tools via `qwen3_coder`, reasoning
-via `qwen3` (the field is `reasoning`, not llama.cpp's `reasoning_content`),
-vision through the Gemma4 merge path with `MM_MERGE_ENGINE=AXROW_SEPARATE`.
+to 461,540 tokens (1.76× the full window; smaller than the MXFP4 rollback
+target's 943,581 because the FP8 weights are bigger). Tools via `qwen3_coder`,
+reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's
+`reasoning_content`), vision through the Gemma4 merge path.
+
+**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads
+`quantization_config.quant_method` from the checkpoint's `config.json` and
+serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`,
+`ROCM_AITER_UNIFIED_ATTN`, MTP self-speculation) or the MXFP4 patch-and-serve
+path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet
+env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads
+the serving shape from the quadlet and gates on the right assets. The MXFP4
+checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` + the z-lab
+DFlash2-FP8 drafter) remains on disk as the rollback target. Full swap
+process + toolchain: [[frank-medium-model-swap]].

 **TP=2 runs because the board changed.** The predecessor board (TUF GAMING
 B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-`
@@ -365,18 +378,26 @@ vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured
 against production after deploy: 92.4 t/s, 0.22 s, 248.9.

 Files: `dankserver/frank-medium/quadlet/frank-medium.container`,
-`dankserver/frank-medium/radiance-entrypoint.sh`,
-`dankserver/frank-medium/frank-medium-healthcheck.py`. The llama.cpp layer-split
-trio (`scripts/frank-medium.service`, `.env`, `serve-frank-medium.sh`) is
-archived in `scripts/.AGENTTRASH/20260920-llama-layersplit/` and is the
-rollback target; `deploy-remote.sh`'s `retire_llama_medium_unit` moved the
-installed unit symlink there rather than `systemctl disable`, which would delete
-it. The deploy handler gates on the image tag, both model configs, the patch
-clone + jinja template, and `libr4d`, then waits for :7770 and gates on
-`/health`.
+`dankserver/frank-medium/radiance-entrypoint.sh` (quantization-aware),
+`dankserver/frank-medium/frank-medium-healthcheck.py`. Swap toolchain:
+`scripts/hf-manifest.py` (generate a `<filename> <size> <sha256>` manifest
+from the live repo tree), `scripts/hf-fetch.sh` (resumable, byte + sha256
+verified, idempotent fetch), `scripts/swap-frank-medium.sh` (generate
+manifest → download → detect quantization → edit the quadlet → deploy). The
+llama.cpp layer-split trio (`scripts/frank-medium.service`, `.env`,
+`serve-frank-medium.sh`) is archived in `scripts/.AGENTTRASH/
+20260920-llama-layersplit/`; `deploy-remote.sh`'s `retire_llama_medium_unit`
+moved the installed unit symlink there rather than `systemctl disable`, which
+would delete it. The deploy handler reads `SNAP`/`SPEC_METHOD` from the
+quadlet, gates on the image tag + the checkpoint's `config.json` (drafter only
+for `dflash`, `libr4d` only for `mxfp4`/`quark`), then waits for :7770 and
+gates on `/health`.

-Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
-Rollback: [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).
+Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` (serving
+details); swap process + toolchain: `serverAppConfigs-docs/LLM/
+frank-medium-model-swap.md`.
+Rollback: the MXFP4 checkpoint on disk (`SPEC_METHOD=dflash`, `SPEC=7`);
+before that, [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).

 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -721,7 +742,8 @@ Active:
 |---------|---------|
 | frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
 | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-medium (radiance TP=2 + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
+| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
+| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""hf-manifest.py — generate an hf-fetch.sh manifest from a HuggingFace repo.
+
+Usage: python3 scripts/hf-manifest.py <owner/repo> [outfile]
+
+Lists the repo tree via the HF API and, for each file, records the size and
+the LFS sha256 (the X-Linked-ETag header on the resolve endpoint). Non-LFS
+files (config.json, tokenizer, ...) have no LFS sha256, so they get a blank
+hash field — hf-fetch.sh then checks size only for them.
+
+Output: one line per file, three space-separated fields:
+    <filename> <size_bytes> <sha256-or-blank>
+
+Token: $HF_TOKEN, else ~/.cache/huggingface/token, else the llamaswap conf
+(grep hf_...). Gated repos need a token that has been granted access.
+"""
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.request
+
+
+def get_token():
+    t = os.environ.get("HF_TOKEN", "")
+    if not t:
+        p = os.path.expanduser("~/.cache/huggingface/token")
+        if os.path.exists(p):
+            with open(p) as f:
+                t = f.read().strip()
+    if not t:
+        p = os.path.expanduser("~/.config/llamaswap.conf.yaml")
+        if os.path.exists(p):
+            with open(p) as f:
+                m = re.search(r"hf_[a-zA-Z0-9]+", f.read())
+                if m:
+                    t = m.group(0)
+    return t
+
+
+def head(url, token):
+    h = {"User-Agent": "hf-manifest/1.0"}
+    if token:
+        h["Authorization"] = f"Bearer {token}"
+    req = urllib.request.Request(url, method="HEAD", headers=h)
+    return urllib.request.urlopen(req, timeout=30)
+
+
+def main():
+    if len(sys.argv) < 2:
+        print(f"usage: {sys.argv[0]} <owner/repo> [outfile]", file=sys.stderr)
+        sys.exit(2)
+    repo = sys.argv[1]
+    outfile = sys.argv[2] if len(sys.argv) > 2 else None
+    token = get_token()
+    # 1. List the repo tree (siblings[].rfilename). Sizes and LFS sha256s
+    # come from the resolve endpoint's HEAD (Content-Length + X-Linked-ETag);
+    # the API's siblings carry only rfilename.
+    api = f"https://huggingface.co/api/models/{repo}"
+    req = urllib.request.Request(
+        api, headers={"User-Agent": "hf-manifest/1.0",
+                      **({"Authorization": f"Bearer {token}"} if token else {})})
+    with urllib.request.urlopen(req, timeout=30) as r:
+        meta = json.load(r)
+    siblings = meta.get("siblings", [])
+    print(f"# {repo}: {len(siblings)} entries", file=sys.stderr)
+
+    lines = []
+    for s in siblings:
+        name = s.get("rfilename", "")
+        if not name or name.endswith("/"):
+            continue
+        # 2. Size (Content-Length) + LFS sha256 (X-Linked-ETag) from the
+        # resolve endpoint's HEAD. Non-LFS files have no 64-hex etag, so the
+        # hash field stays blank (hf-fetch.sh checks size only for them).
+        size = 0
+        sha = ""
+        resolve = f"https://huggingface.co/{repo}/resolve/main/{name}"
+        try:
+            with head(resolve, token) as r:
+                size = int(r.headers.get("Content-Length") or 0)
+                etag = (r.headers.get("X-Linked-ETag")
+                        or r.headers.get("ETag") or "").strip('"')
+                if re.fullmatch(r"[0-9a-f]{64}", etag):
+                    sha = etag
+        except urllib.error.HTTPError as e:
+            if e.code != 404:
+                raise
+        lines.append(f"{name} {size} {sha}")
+        print(f"  {name}  {size:>12}  {sha or '(size only)'}", file=sys.stderr)
+
+    out = "\n".join(lines) + "\n"
+    if outfile:
+        with open(outfile, "w") as f:
+            f.write(out)
+        print(f"wrote {outfile}", file=sys.stderr)
+    else:
+        sys.stdout.write(out)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+# swap-frank-medium.sh — orchestrate a frank-medium model swap.
+#
+# A swap is four steps, and this script drives all of them:
+#   1. generate a manifest for the new HF repo (hf-manifest.py),
+#   2. download it to the host, resumable + byte/ sha256-verified (hf-fetch.sh),
+#   3. detect the checkpoint's weight quantization and point the quadlet at it
+#      (SNAP + the matching SPEC_METHOD/SPEC + KV_MEM),
+#   4. hand off to deploy.sh (pull, restart, health gate).
+#
+# The entrypoint is QUANTIZATION-AWARE: it reads quant_method from the
+# checkpoint's config.json and serves the plain FP8 path (no MXFP4 patches,
+# --quantization=fp8, ROCM_AITER unified attention, MTP self-speculation) or
+# the MXFP4 patch-and-serve path (DFlash2 external drafter). So the ONLY repo
+# change a swap makes is the quadlet's four env vars — this script edits them.
+#
+# Usage:
+#   scripts/swap-frank-medium.sh --repo <owner/repo> --target <host-dir> \
+#       [--generate-manifest] [--manifest <path>] [--spec N] [--kv MB] \
+#       [--apply] [--deploy]
+#
+#   --repo <owner/repo>    the Hugging Face repo to serve
+#   --target <host-dir>    the host dir the model lands in
+#                          (/home/dank/models/<name>); the quadlet SNAP is
+#                          derived as /models/<name>
+#   --generate-manifest    write scripts/frank-medium-<name>.manifest from the
+#                          live repo tree (needs the host's HF token)
+#   --manifest <path>      use an existing manifest instead of generating one
+#   --spec N               speculative tokens (default 8 for fp8, 7 for mxfp4)
+#   --kv MB                KV pool pin in MiB (default 0 = auto-size)
+#   --apply                edit the quadlet (SNAP + SPEC_METHOD + SPEC + KV_MEM)
+#   --deploy               run deploy.sh after --apply
+#
+# Without --apply the script stops after the download + quantization
+# detection and prints the quadlet lines to set. The download and the
+# quantization detection run on the host (the model dir and the token are
+# there); the quadlet edit runs in the repo.
+
+set -euo pipefail
+
+HOST="dank@192.168.0.235"
+QUADLET="dankserver/frank-medium/quadlet/frank-medium.container"
+MODELS_ROOT="/home/dank/models"
+
+repo="" target="" manifest="" spec="" kv=0 gen=0 apply=0 deploy=0
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --repo) repo="$2"; shift 2 ;;
+    --target) target="$2"; shift 2 ;;
+    --manifest) manifest="$2"; shift 2 ;;
+    --spec) spec="$2"; shift 2 ;;
+    --kv) kv="$2"; shift 2 ;;
+    --generate-manifest) gen=1; shift ;;
+    --apply) apply=1; shift ;;
+    --deploy) deploy=1; shift ;;
+    *) echo "unknown arg: $1" >&2; exit 2 ;;
+  esac
+done
+
+[[ -n "$repo" && -n "$target" ]] || {
+  echo "usage: $0 --repo <owner/repo> --target <host-dir> [--generate-manifest|--manifest <path>] [--spec N] [--kv MB] [--apply] [--deploy]" >&2
+  exit 2;
+}
+
+name="$(basename "$target")"
+snap_c="/models/$name"
+
+# --- 1. manifest -------------------------------------------------------------
+if [[ "$gen" = 1 ]]; then
+  manifest="scripts/frank-medium-${name}.manifest"
+  echo "==> Generating manifest -> $manifest"
+  scp -q scripts/hf-manifest.py "$HOST:/tmp/hf-manifest.py"
+  ssh "$HOST" "python3 /tmp/hf-manifest.py '$repo' /tmp/swap-manifest"
+  scp -q "$HOST:/tmp/swap-manifest" "$manifest"
+elif [[ -z "$manifest" ]]; then
+  echo "ERROR: pass --generate-manifest or --manifest <path>" >&2
+  exit 2
+fi
+[[ -f "$manifest" ]] || { echo "ERROR: manifest not found: $manifest" >&2; exit 2; }
+
+# --- 2. download (resumable, verified) ---------------------------------------
+echo "==> Staging manifest + hf-fetch.sh on $HOST"
+scp -q "$manifest" "$HOST:/tmp/swap-manifest"
+scp -q scripts/hf-fetch.sh "$HOST:/tmp/swap-hf-fetch.sh"
+echo "==> Downloading $repo -> $target"
+ssh "$HOST" "bash /tmp/swap-hf-fetch.sh '$target' /tmp/swap-manifest '$repo'"
+
+# --- 3. detect the quantization on the host ----------------------------------
+quant="$(ssh "$HOST" "python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get(\"quantization_config\",{}).get(\"quant_method\",\"\"))' '$target/config.json' 2>/dev/null || true")"
+echo "==> Detected quantization: ${quant:-<none>}"
+
+case "$quant" in
+  fp8)              spec_method=mtp;    default_spec=8  ;;
+  mxfp4|quark|"")   spec_method=dflash; default_spec=7  ;;
+  *)                spec_method=dflash; default_spec=7
+                    echo "WARNING: unknown quant '$quant'; defaulting to the MXFP4 path" >&2 ;;
+esac
+[[ -n "$spec" ]] || spec="$default_spec"
+
+echo "==> Quadlet env vars for this checkpoint:"
+echo "    Environment=SNAP=$snap_c"
+echo "    Environment=SPEC_METHOD=$spec_method"
+echo "    Environment=SPEC=$spec"
+echo "    Environment=KV_MEM=$kv"
+
+# --- 4. apply the quadlet edit (in the repo) ---------------------------------
+if [[ "$apply" = 1 ]]; then
+  echo "==> Editing $QUADLET"
+  sed -i -E "s|^Environment=SNAP=.*|Environment=SNAP=$snap_c|" "$QUADLET"
+  sed -i -E "s|^Environment=SPEC_METHOD=.*|Environment=SPEC_METHOD=$spec_method|" "$QUADLET"
+  sed -i -E "s|^Environment=SPEC=.*|Environment=SPEC=$spec|" "$QUADLET"
+  sed -i -E "s|^Environment=KV_MEM=.*|Environment=KV_MEM=$kv|" "$QUADLET"
+  echo "==> Quadlet updated."
+fi
+
+# --- 5. deploy ----------------------------------------------------------------
+if [[ "$deploy" = 1 ]]; then
+  [[ "$apply" = 1 ]] || { echo "ERROR: --deploy requires --apply" >&2; exit 2; }
+  echo "==> Deploying (pull, restart, health gate)"
+  ./scripts/deploy.sh dankserver --service frank-medium --skip-e2e
+  echo "==> Next: run the capability battery"
+  echo "    python3 scripts/bench-conversation.py http://192.168.0.235:7770 --model frank-medium --salt ${name} --max-tokens 512 --burst 4"
+fi
@@ -0,0 +1,173 @@
+---
+type: runbook
+title: frank-medium Model Swap
+description: How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path.
+tags:
+  - frank-medium
+  - dankserver
+  - model-swap
+  - vllm-radiance
+  - huggingface
+  - runbook
+timestamp: 2026-09-20
+---
+
+# frank-medium Model Swap
+
+Swapping the checkpoint behind **frank-medium** (Dankserver `:7770`,
+vllm-radiance TP=2) is a four-step process, and the toolchain below drives all
+of it. The design goal (2026-09-20, after the MXFP4 → FP8 swap) is that a swap
+is **one command**: generate a manifest, download it verified, point the
+quadlet at it, deploy. The entrypoint is quantization-aware, so the only repo
+change a swap makes is the quadlet's four env vars.
+
+## Why this exists
+
+The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not
+the GPU — it was the **download** (30.9 GB across 7 shards, hand-verified) and
+the **serving-path decision** (which attention backend, which drafter, which
+quantization flag). Both are now mechanical:
+
+- The download is `hf-fetch.sh` — resumable, byte-count + sha256-verified,
+  idempotent (a re-run only fetches what is missing or corrupt).
+- The serving path is the **quantization-aware entrypoint**
+  (`dankserver/frank-medium/radiance-entrypoint.sh`) — it reads
+  `quantization_config.quant_method` from the checkpoint's `config.json` and
+  picks the path. The swap operator never chooses it by hand.
+
+## The toolchain
+
+| Tool | Role |
+|---|---|
+| `scripts/hf-manifest.py` | Generate a fetch manifest from a live HF repo tree: `<filename> <size> <sha256-or-blank>` per line. LFS sha256s come from the resolve endpoint's `X-Linked-ETag`; non-LFS files get a blank hash (size-only check). |
+| `scripts/hf-fetch.sh` | Fetch a manifest to a dir, resumable (`curl -C -`), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 `curl -K` file. |
+| `scripts/swap-frank-medium.sh` | The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy. |
+
+The HF token is resolved on the **host** (dankserver) from `$HF_TOKEN`,
+`~/.cache/huggingface/token`, or the llamaswap conf. Gated repos need a token
+that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant
+for `orcarouter/Qwen3.8-27B-Uncensored-FP8`).
+
+## The swap (one command)
+
+```
+scripts/swap-frank-medium.sh \
+    --repo <owner/repo> \
+    --target /home/dank/models/<name> \
+    --generate-manifest \
+    --apply \
+    --deploy
+```
+
+What it does, in order:
+
+1. **Manifest** — `hf-manifest.py` lists the repo tree and writes
+   `scripts/frank-medium-<name>.manifest` (committed, so the payload is
+   reproducible). Pass `--manifest <path>` to reuse an existing one.
+2. **Download** — stages the manifest + `hf-fetch.sh` on the host and fetches
+   to `--target`. Resumable: an interrupted swap re-runs and skips what is
+   already verified.
+3. **Detect** — reads `quantization_config.quant_method` from the downloaded
+   `config.json` and maps it to the serving shape (below).
+4. **Apply** — edits the quadlet's four env vars (`SNAP`, `SPEC_METHOD`,
+   `SPEC`, `KV_MEM`).
+5. **Deploy** — `deploy.sh dankserver --service frank-medium --skip-e2e`
+   (pull, restart, health gate, up to 1800 s — the first start compiles
+   kernels).
+
+Without `--apply` the script stops after the download + detection and prints
+the quadlet lines to set, so you can eyeball them before committing.
+
+## Quantization → serving path
+
+The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches:
+
+| `quant_method` | Path | Attention | Speculation | Drafter |
+|---|---|---|---|---|
+| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | MTP self-speculation (`--quantization fp8`) | the model's own MTP head (`mtp.layers.0`) |
+| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` |
+
+The quadlet `SPEC_METHOD`/`SPEC` the swap script sets must match the path:
+`fp8 → mtp/8`, `mxfp4|quark → dflash/7`. The FP8 model carries its own
+single-layer MTP head (22 tensors), so MTP self-speculation needs no external
+drafter and no `fp8_mtp.py` rewrite. The MXFP4 path applies the
+`radiance-vllm-mxfp4` patches at every container start and uses the DFlash2
+drafter.
+
+**R4D allreduce is independent of the weight quantization.** The FP8 path
+uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be
+tried later for a speed bump (`R4D_ATTN=1`), but the MXFP4 path is the one
+tested with it.
+
+## The quadlet contract
+
+A swap edits exactly four env vars in
+`dankserver/frank-medium/quadlet/frank-medium.container`:
+
+- `SNAP=/models/<name>` — the container path (the host dir
+  `/home/dank/models/<name>` is mounted at `/models`).
+- `SPEC_METHOD=mtp|dflash` — the speculative-decoding method.
+- `SPEC=8|7` — the speculative depth.
+- `KV_MEM=0` — the KV pool pin in MiB; `0` = auto-size (the documented
+  plain-FP8 default). After the first healthy start, recalibrate with
+  `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards.
+
+The deploy handler (`deploy-remote.sh deploy_llm_medium`) reads `SNAP` and
+`SPEC_METHOD` from the quadlet (the source of truth), maps `/models/` →
+`/home/dank/models/`, and gates on the right assets: the drafter only for
+`dflash`, `r4d.so` only for `mxfp4`/`quark`. So a swap that points the quadlet
+at a new checkpoint is enough — the handler follows.
+
+## Verify (capability battery)
+
+After the deploy is green, run the battery (read-only — it does not stop the
+service):
+
+```
+python3 scripts/bench-conversation.py http://192.168.0.235:7770 \
+    --model frank-medium --salt <name> --max-tokens 512 --burst 4
+```
+
+Then the capability checks (all must pass):
+
+- `/health` → 200, `/v1/models` → `frank-medium`.
+- A chat completion returns `content` **and** reasoning (vLLM emits the
+  `reasoning` field; the harness auto-detects it).
+- MTP speculation: the boot log's `SpecDecoding metrics` line shows a mean
+  acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2%
+  draft acceptance).
+- Tool calls: a `tools` request returns `tool_calls`.
+- Vision: an image request is answered (the FP8 checkpoint serves vision
+  despite `language_model_only: true` in `config.json`).
+
+## Rollback
+
+The previous checkpoint stays on disk (a swap never deletes it). To roll
+back:
+
+1. Point the quadlet at the old checkpoint (`SNAP`, `SPEC_METHOD`, `SPEC` —
+   the old values are in git history).
+2. `deploy.sh dankserver --service frank-medium --skip-e2e`.
+3. Re-run the capability battery.
+
+The 2026-09-20 rollback target is the MXFP4 checkpoint at
+`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single
+safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`.
+
+## 2026-09-20 swap record (MXFP4 → FP8)
+
+- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
+  19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise
+  128×128, dynamic activation quant, 30.9 GB / 7 shards).
+- **Serving path**: DFlash2 external drafter → MTP self-speculation
+  (`SPEC_METHOD=mtp`, `SPEC=8`); `R4D` → `ROCM_AITER_UNIFIED_ATTN`.
+- **MTP**: mean acceptance length 5.66/8, 66.2% draft acceptance,
+  per-position rates 0.89 → 0.41.
+- **KV pool**: auto-sized to 461,540 tokens (1.76× full-window) — smaller than
+  MXFP4's 943,581 because FP8 weights are bigger (30.9 vs 19.4 GiB).
+- **Battery** (vs MXFP4 baseline): geomean decode 85.9 vs 92.4 t/s (−7%),
+  mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
+  (−27%), burst-4 aggregate 179.7 vs 248.9 t/s (−28%). The trade-off is the
+  expected one: higher precision (8-bit vs 4-bit weights), decode within 7%,
+  TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
+  vs R4D).
@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: frank-medium vllm-radiance MXFP4 Runbook
-description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201), with a DFlash2-FP8 drafter. Vision is served by the loaded checkpoint (verified on synthetic images 2026-09-20). The sections below describe the 2026-09-16 two-replica era.
+description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
 tags:
  - frank-medium
  - dankserver
@@ -28,6 +28,16 @@ It replaced the mainline-llama.cpp layer-split deployment
 model had earlier run here as two single-GPU replicas, which is the shape most
 of this page describes.

+**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves
+`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7
+shards) with MTP self-speculation — the quantization-aware entrypoint reads
+`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4
+patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4
+checkpoint above remains on disk as the rollback target. The swap process,
+the toolchain, and the measured trade-off are in [[frank-medium-model-swap]];
+the "Deployed flags" table below still shows the MXFP4 shape (the rollback
+target).
+
 **Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
 The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
 ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every
@@ -43,8 +53,8 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:

 | Setting | Value |
 |---|---|
-| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` |
-| Speculative | `--speculative-config {"method":"dflash","model":"/models/draft","num_speculative_tokens":7}` (z-lab DFlash2-FP8) |
+| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
+| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) |
 | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
 | Context | `--max-model-len 262144` |
 | Concurrency | `--max-num-seqs 8`, chunk 8192 |
@@ -56,17 +66,17 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
 | Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
 | Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) |

-**The KV pool is auto-sized, not pinned.** The arm-R benchmark run used
-`KV_CACHE_MEMORY_MB=18563072000`, which bought exactly one 262,144-token
-sequence (262,720 tokens at 70,656 B/token — a profile inflated by the
-8192-chunk activation peak). Production ships no such pin: the pool came up at
-**943,581 tokens = 3.59× the full window**, and the boot line read
-`Available KV cache memory: 18.17 GiB → GPU KV cache size: 943,581 tokens`.
+**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
+(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8
+checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it
+leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540
+tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`,
+`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4
+rollback target auto-sized to 943,581 tokens (3.59×) on the same cards.
 Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16
-full-attention layers plus the drafter's. Container RSS measured 28.4 GiB
-against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
-max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
-two-replica vision shape, not this one.
+full-attention layers plus the MTP drafter's. After the first healthy start,
+recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
+cards.

 ## Prefix caching — measured granularity (2026-09-20)

@@ -58,13 +58,20 @@ frank-small, port 7771]
 ## Active Deployment

 - [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
-  2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 +
-  ggz14/radiance-vllm-mxfp4 patches) serving
-  just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8
-  drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm
-  battery that chose it, the auto-sized 943,581-token KV pool, verified
-  capabilities (tools, reasoning, vision), and the record of the 2026-09-16
-  two-replica era
+  2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
+  orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
+  shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP
+  self-speculation (the model's own single-layer MTP head). The quantization-
+  aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint
+  (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8
+  drafter) remains on disk as the rollback target. The three-arm battery that
+  chose TP=2, the auto-sized KV pool, and verified capabilities (tools,
+  reasoning, vision)
+- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
+  the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
+  swap-frank-medium.sh), the quantization-aware entrypoint contract, the
+  quadlet edit, the deploy, the capability battery, and the rollback path.
+  The 2026-09-20 MXFP4 → FP8 swap is the worked example
 - [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
  llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
  DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
@@ -116,11 +116,16 @@ The LLM inference fleet on the two hosts. Full index:
 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
 - [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
-  2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the
-  R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq
-  with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700
-  behind the :7770 endpoint — deployed flags, the three-arm battery that chose
-  it, verified capabilities, and the record of the 2026-09-16 two-replica era
+  2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D
+  one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3
+  block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700
+  behind the :7770 endpoint — the quantization-aware entrypoint, the
+  three-arm battery that chose TP=2, verified capabilities, and the record of
+  the 2026-09-16 two-replica era
+- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
+  the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
+  swap-frank-medium.sh), the quantization-aware entrypoint contract, the
+  quadlet edit, the deploy, the capability battery, and the rollback path
 - [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
  mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
  Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
@@ -11,6 +11,51 @@ timestamp: 2026-09-14

 # Bundle Change Log

+## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
+
+- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
+  19.37 GiB single shard, DFlash2 external drafter) →
+  `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128,
+  dynamic activation quant, 30.9 GB / 7 shards, its own single-layer MTP head).
+  The quantization-aware entrypoint (`radiance-entrypoint.sh`) reads
+  `quant_method` from `config.json` and serves the plain FP8 path — no MXFP4
+  patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`, MTP
+  self-speculation (`SPEC_METHOD=mtp`, `SPEC=8`). The quadlet edit was four
+  env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`); the deploy handler
+  (`deploy-remote.sh`) now reads the serving shape from the quadlet and gates
+  on the right assets (drafter only for `dflash`, `r4d.so` only for
+  `mxfp4`/`quark`).
+- **MTP measured, not inferred.** Boot log `SpecDecoding metrics`: mean
+  acceptance length **5.66/8**, 66.2 % draft acceptance, per-position rates
+  0.89 → 0.41. The FP8 model carries its own single-layer MTP head (22
+  tensors), so self-speculation needs no external drafter and no `fp8_mtp.py`
+  rewrite.
+- **KV pool auto-sized to 461,540 tokens (1.76× the window)** — smaller than
+  MXFP4's 943,581 (3.59×) because the FP8 weights are bigger (30.9 vs 19.4
+  GiB). `KV_MEM=0` (the documented plain-FP8 default); recalibrate with
+  `calibrate-kv.sh` after the first healthy start.
+- **Battery (vs the MXFP4 baseline).** geomean decode 85.9 vs 92.4 t/s (−7 %),
+  mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
+  (−27 %), burst-4 aggregate 179.7 vs 248.9 t/s (−28 %). The trade-off is the
+  expected one: higher precision (8-bit vs 4-bit weights), decode within 7 %,
+  TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
+  vs R4D). Vision, tools, and reasoning all verified live.
+- **Streamlining — the long pole was the download, not the GPU.** Three tools
+  make a swap one command: `scripts/hf-manifest.py` (generate a
+  `<filename> <size> <sha256>` manifest from the live repo tree — LFS sha256s
+  from the resolve endpoint's `X-Linked-ETag`, sizes from `Content-Length`),
+  `scripts/hf-fetch.sh` (resumable, byte + sha256 verified, idempotent, token
+  out-of-band via a mode-600 `curl -K` file), and
+  `scripts/swap-frank-medium.sh` (generate manifest → download → detect
+  quantization → edit the quadlet → deploy). A swap is now
+  `swap-frank-medium.sh --repo <owner/repo> --target /home/dank/models/<name>
+  --generate-manifest --apply --deploy`. Full process: [[frank-medium-model-swap]].
+- **Docs.** [[frank-medium-radiance-mxfp4]] description + deployed-flags table
+  + KV-pool note flipped to the FP8 shape (the MXFP4 checkpoint is the
+  rollback target); [[frank-medium-model-swap]] is the new swap runbook; the
+  LLM index's frank-medium entry flipped. The MXFP4 checkpoint stays on disk
+  at `/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq`.
+
 ## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime

 - **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`