From c1a4ccbbcd527685d9d81f28c7b240a907f7e394 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 20 Sep 2026 21:53:00 -0700 Subject: [PATCH] feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape. --- AGENTS.md | 62 +++++-- scripts/hf-manifest.py | 103 +++++++++++ scripts/swap-frank-medium.sh | 124 +++++++++++++ .../LLM/frank-medium-model-swap.md | 173 ++++++++++++++++++ .../LLM/frank-medium-radiance-mxfp4.md | 36 ++-- serverAppConfigs-docs/LLM/index.md | 21 ++- serverAppConfigs-docs/index.md | 15 +- serverAppConfigs-docs/log.md | 45 +++++ 8 files changed, 534 insertions(+), 45 deletions(-) create mode 100755 scripts/hf-manifest.py create mode 100755 scripts/swap-frank-medium.sh create mode 100644 serverAppConfigs-docs/LLM/frank-medium-model-swap.md diff --git a/AGENTS.md b/AGENTS.md index eeb2ab5..6170106 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | Qwen3.8-27B-Uncensored MXFP4-awq with the z-lab DFlash2-FP8 drafter (depth 7, live mean accept length ~4.1), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 943,581-token KV pool = 3.59x the window), fp8 KV, prefix caching, vision | 7770 | +| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with MTP self-speculation (depth 8, live mean accept length 5.66/8), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 461,540-token KV pool = 1.76x the window), fp8 KV, prefix caching, vision | 7770 | | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | @@ -334,18 +334,31 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-medium — radiance TP=2, DFlash2-FP8 drafter (Dankserver, ACTIVE) +### frank-medium — radiance TP=2, MTP self-speculation (Dankserver, ACTIVE) vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet** `dankserver/frank-medium/quadlet/frank-medium.container`) serving -`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4) with the -**z-lab DFlash2 FP8 drafter** (`num_speculative_tokens: 7`, live mean accept -~4.1) in **one tensor-parallel process across both R9700s** +`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic +activation quant, 30.9 GB / 7 shards) with **MTP self-speculation** +(`num_speculative_tokens: 8`, the model's own single-layer MTP head, live mean +accept 5.66/8) in **one tensor-parallel process across both R9700s** (`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8 KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized -to 943,581 tokens (3.59× the full window). Tools via `qwen3_coder`, reasoning -via `qwen3` (the field is `reasoning`, not llama.cpp's `reasoning_content`), -vision through the Gemma4 merge path with `MM_MERGE_ENGINE=AXROW_SEPARATE`. +to 461,540 tokens (1.76× the full window; smaller than the MXFP4 rollback +target's 943,581 because the FP8 weights are bigger). Tools via `qwen3_coder`, +reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's +`reasoning_content`), vision through the Gemma4 merge path. + +**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads +`quantization_config.quant_method` from the checkpoint's `config.json` and +serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`, +`ROCM_AITER_UNIFIED_ATTN`, MTP self-speculation) or the MXFP4 patch-and-serve +path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet +env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads +the serving shape from the quadlet and gates on the right assets. The MXFP4 +checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` + the z-lab +DFlash2-FP8 drafter) remains on disk as the rollback target. Full swap +process + toolchain: [[frank-medium-model-swap]]. **TP=2 runs because the board changed.** The predecessor board (TUF GAMING B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-` @@ -365,18 +378,26 @@ vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured against production after deploy: 92.4 t/s, 0.22 s, 248.9. Files: `dankserver/frank-medium/quadlet/frank-medium.container`, -`dankserver/frank-medium/radiance-entrypoint.sh`, -`dankserver/frank-medium/frank-medium-healthcheck.py`. The llama.cpp layer-split -trio (`scripts/frank-medium.service`, `.env`, `serve-frank-medium.sh`) is -archived in `scripts/.AGENTTRASH/20260920-llama-layersplit/` and is the -rollback target; `deploy-remote.sh`'s `retire_llama_medium_unit` moved the -installed unit symlink there rather than `systemctl disable`, which would delete -it. The deploy handler gates on the image tag, both model configs, the patch -clone + jinja template, and `libr4d`, then waits for :7770 and gates on -`/health`. +`dankserver/frank-medium/radiance-entrypoint.sh` (quantization-aware), +`dankserver/frank-medium/frank-medium-healthcheck.py`. Swap toolchain: +`scripts/hf-manifest.py` (generate a ` ` manifest +from the live repo tree), `scripts/hf-fetch.sh` (resumable, byte + sha256 +verified, idempotent fetch), `scripts/swap-frank-medium.sh` (generate +manifest → download → detect quantization → edit the quadlet → deploy). The +llama.cpp layer-split trio (`scripts/frank-medium.service`, `.env`, +`serve-frank-medium.sh`) is archived in `scripts/.AGENTTRASH/ +20260920-llama-layersplit/`; `deploy-remote.sh`'s `retire_llama_medium_unit` +moved the installed unit symlink there rather than `systemctl disable`, which +would delete it. The deploy handler reads `SNAP`/`SPEC_METHOD` from the +quadlet, gates on the image tag + the checkpoint's `config.json` (drafter only +for `dflash`, `libr4d` only for `mxfp4`/`quark`), then waits for :7770 and +gates on `/health`. -Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`. -Rollback: [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20). +Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` (serving +details); swap process + toolchain: `serverAppConfigs-docs/LLM/ +frank-medium-model-swap.md`. +Rollback: the MXFP4 checkpoint on disk (`SPEC_METHOD=dflash`, `SPEC=7`); +before that, [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20). ## Key Lessons Learned Durable operational lessons from the fleet's history. Full detail in the @@ -721,7 +742,8 @@ Active: |---------|---------| | frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | -| frank-medium (radiance TP=2 + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | +| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | +| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | diff --git a/scripts/hf-manifest.py b/scripts/hf-manifest.py new file mode 100755 index 0000000..ea0eea0 --- /dev/null +++ b/scripts/hf-manifest.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +"""hf-manifest.py — generate an hf-fetch.sh manifest from a HuggingFace repo. + +Usage: python3 scripts/hf-manifest.py [outfile] + +Lists the repo tree via the HF API and, for each file, records the size and +the LFS sha256 (the X-Linked-ETag header on the resolve endpoint). Non-LFS +files (config.json, tokenizer, ...) have no LFS sha256, so they get a blank +hash field — hf-fetch.sh then checks size only for them. + +Output: one line per file, three space-separated fields: + + +Token: $HF_TOKEN, else ~/.cache/huggingface/token, else the llamaswap conf +(grep hf_...). Gated repos need a token that has been granted access. +""" +import json +import os +import re +import sys +import urllib.error +import urllib.request + + +def get_token(): + t = os.environ.get("HF_TOKEN", "") + if not t: + p = os.path.expanduser("~/.cache/huggingface/token") + if os.path.exists(p): + with open(p) as f: + t = f.read().strip() + if not t: + p = os.path.expanduser("~/.config/llamaswap.conf.yaml") + if os.path.exists(p): + with open(p) as f: + m = re.search(r"hf_[a-zA-Z0-9]+", f.read()) + if m: + t = m.group(0) + return t + + +def head(url, token): + h = {"User-Agent": "hf-manifest/1.0"} + if token: + h["Authorization"] = f"Bearer {token}" + req = urllib.request.Request(url, method="HEAD", headers=h) + return urllib.request.urlopen(req, timeout=30) + + +def main(): + if len(sys.argv) < 2: + print(f"usage: {sys.argv[0]} [outfile]", file=sys.stderr) + sys.exit(2) + repo = sys.argv[1] + outfile = sys.argv[2] if len(sys.argv) > 2 else None + token = get_token() + # 1. List the repo tree (siblings[].rfilename). Sizes and LFS sha256s + # come from the resolve endpoint's HEAD (Content-Length + X-Linked-ETag); + # the API's siblings carry only rfilename. + api = f"https://huggingface.co/api/models/{repo}" + req = urllib.request.Request( + api, headers={"User-Agent": "hf-manifest/1.0", + **({"Authorization": f"Bearer {token}"} if token else {})}) + with urllib.request.urlopen(req, timeout=30) as r: + meta = json.load(r) + siblings = meta.get("siblings", []) + print(f"# {repo}: {len(siblings)} entries", file=sys.stderr) + + lines = [] + for s in siblings: + name = s.get("rfilename", "") + if not name or name.endswith("/"): + continue + # 2. Size (Content-Length) + LFS sha256 (X-Linked-ETag) from the + # resolve endpoint's HEAD. Non-LFS files have no 64-hex etag, so the + # hash field stays blank (hf-fetch.sh checks size only for them). + size = 0 + sha = "" + resolve = f"https://huggingface.co/{repo}/resolve/main/{name}" + try: + with head(resolve, token) as r: + size = int(r.headers.get("Content-Length") or 0) + etag = (r.headers.get("X-Linked-ETag") + or r.headers.get("ETag") or "").strip('"') + if re.fullmatch(r"[0-9a-f]{64}", etag): + sha = etag + except urllib.error.HTTPError as e: + if e.code != 404: + raise + lines.append(f"{name} {size} {sha}") + print(f" {name} {size:>12} {sha or '(size only)'}", file=sys.stderr) + + out = "\n".join(lines) + "\n" + if outfile: + with open(outfile, "w") as f: + f.write(out) + print(f"wrote {outfile}", file=sys.stderr) + else: + sys.stdout.write(out) + + +if __name__ == "__main__": + main() diff --git a/scripts/swap-frank-medium.sh b/scripts/swap-frank-medium.sh new file mode 100755 index 0000000..e6a887a --- /dev/null +++ b/scripts/swap-frank-medium.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# swap-frank-medium.sh — orchestrate a frank-medium model swap. +# +# A swap is four steps, and this script drives all of them: +# 1. generate a manifest for the new HF repo (hf-manifest.py), +# 2. download it to the host, resumable + byte/ sha256-verified (hf-fetch.sh), +# 3. detect the checkpoint's weight quantization and point the quadlet at it +# (SNAP + the matching SPEC_METHOD/SPEC + KV_MEM), +# 4. hand off to deploy.sh (pull, restart, health gate). +# +# The entrypoint is QUANTIZATION-AWARE: it reads quant_method from the +# checkpoint's config.json and serves the plain FP8 path (no MXFP4 patches, +# --quantization=fp8, ROCM_AITER unified attention, MTP self-speculation) or +# the MXFP4 patch-and-serve path (DFlash2 external drafter). So the ONLY repo +# change a swap makes is the quadlet's four env vars — this script edits them. +# +# Usage: +# scripts/swap-frank-medium.sh --repo --target \ +# [--generate-manifest] [--manifest ] [--spec N] [--kv MB] \ +# [--apply] [--deploy] +# +# --repo the Hugging Face repo to serve +# --target the host dir the model lands in +# (/home/dank/models/); the quadlet SNAP is +# derived as /models/ +# --generate-manifest write scripts/frank-medium-.manifest from the +# live repo tree (needs the host's HF token) +# --manifest use an existing manifest instead of generating one +# --spec N speculative tokens (default 8 for fp8, 7 for mxfp4) +# --kv MB KV pool pin in MiB (default 0 = auto-size) +# --apply edit the quadlet (SNAP + SPEC_METHOD + SPEC + KV_MEM) +# --deploy run deploy.sh after --apply +# +# Without --apply the script stops after the download + quantization +# detection and prints the quadlet lines to set. The download and the +# quantization detection run on the host (the model dir and the token are +# there); the quadlet edit runs in the repo. + +set -euo pipefail + +HOST="dank@192.168.0.235" +QUADLET="dankserver/frank-medium/quadlet/frank-medium.container" +MODELS_ROOT="/home/dank/models" + +repo="" target="" manifest="" spec="" kv=0 gen=0 apply=0 deploy=0 + +while [ $# -gt 0 ]; do + case "$1" in + --repo) repo="$2"; shift 2 ;; + --target) target="$2"; shift 2 ;; + --manifest) manifest="$2"; shift 2 ;; + --spec) spec="$2"; shift 2 ;; + --kv) kv="$2"; shift 2 ;; + --generate-manifest) gen=1; shift ;; + --apply) apply=1; shift ;; + --deploy) deploy=1; shift ;; + *) echo "unknown arg: $1" >&2; exit 2 ;; + esac +done + +[[ -n "$repo" && -n "$target" ]] || { + echo "usage: $0 --repo --target [--generate-manifest|--manifest ] [--spec N] [--kv MB] [--apply] [--deploy]" >&2 + exit 2; +} + +name="$(basename "$target")" +snap_c="/models/$name" + +# --- 1. manifest ------------------------------------------------------------- +if [[ "$gen" = 1 ]]; then + manifest="scripts/frank-medium-${name}.manifest" + echo "==> Generating manifest -> $manifest" + scp -q scripts/hf-manifest.py "$HOST:/tmp/hf-manifest.py" + ssh "$HOST" "python3 /tmp/hf-manifest.py '$repo' /tmp/swap-manifest" + scp -q "$HOST:/tmp/swap-manifest" "$manifest" +elif [[ -z "$manifest" ]]; then + echo "ERROR: pass --generate-manifest or --manifest " >&2 + exit 2 +fi +[[ -f "$manifest" ]] || { echo "ERROR: manifest not found: $manifest" >&2; exit 2; } + +# --- 2. download (resumable, verified) --------------------------------------- +echo "==> Staging manifest + hf-fetch.sh on $HOST" +scp -q "$manifest" "$HOST:/tmp/swap-manifest" +scp -q scripts/hf-fetch.sh "$HOST:/tmp/swap-hf-fetch.sh" +echo "==> Downloading $repo -> $target" +ssh "$HOST" "bash /tmp/swap-hf-fetch.sh '$target' /tmp/swap-manifest '$repo'" + +# --- 3. detect the quantization on the host ---------------------------------- +quant="$(ssh "$HOST" "python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get(\"quantization_config\",{}).get(\"quant_method\",\"\"))' '$target/config.json' 2>/dev/null || true")" +echo "==> Detected quantization: ${quant:-}" + +case "$quant" in + fp8) spec_method=mtp; default_spec=8 ;; + mxfp4|quark|"") spec_method=dflash; default_spec=7 ;; + *) spec_method=dflash; default_spec=7 + echo "WARNING: unknown quant '$quant'; defaulting to the MXFP4 path" >&2 ;; +esac +[[ -n "$spec" ]] || spec="$default_spec" + +echo "==> Quadlet env vars for this checkpoint:" +echo " Environment=SNAP=$snap_c" +echo " Environment=SPEC_METHOD=$spec_method" +echo " Environment=SPEC=$spec" +echo " Environment=KV_MEM=$kv" + +# --- 4. apply the quadlet edit (in the repo) --------------------------------- +if [[ "$apply" = 1 ]]; then + echo "==> Editing $QUADLET" + sed -i -E "s|^Environment=SNAP=.*|Environment=SNAP=$snap_c|" "$QUADLET" + sed -i -E "s|^Environment=SPEC_METHOD=.*|Environment=SPEC_METHOD=$spec_method|" "$QUADLET" + sed -i -E "s|^Environment=SPEC=.*|Environment=SPEC=$spec|" "$QUADLET" + sed -i -E "s|^Environment=KV_MEM=.*|Environment=KV_MEM=$kv|" "$QUADLET" + echo "==> Quadlet updated." +fi + +# --- 5. deploy ---------------------------------------------------------------- +if [[ "$deploy" = 1 ]]; then + [[ "$apply" = 1 ]] || { echo "ERROR: --deploy requires --apply" >&2; exit 2; } + echo "==> Deploying (pull, restart, health gate)" + ./scripts/deploy.sh dankserver --service frank-medium --skip-e2e + echo "==> Next: run the capability battery" + echo " python3 scripts/bench-conversation.py http://192.168.0.235:7770 --model frank-medium --salt ${name} --max-tokens 512 --burst 4" +fi diff --git a/serverAppConfigs-docs/LLM/frank-medium-model-swap.md b/serverAppConfigs-docs/LLM/frank-medium-model-swap.md new file mode 100644 index 0000000..2122912 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-medium-model-swap.md @@ -0,0 +1,173 @@ +--- +type: runbook +title: frank-medium Model Swap +description: How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path. +tags: + - frank-medium + - dankserver + - model-swap + - vllm-radiance + - huggingface + - runbook +timestamp: 2026-09-20 +--- + +# frank-medium Model Swap + +Swapping the checkpoint behind **frank-medium** (Dankserver `:7770`, +vllm-radiance TP=2) is a four-step process, and the toolchain below drives all +of it. The design goal (2026-09-20, after the MXFP4 → FP8 swap) is that a swap +is **one command**: generate a manifest, download it verified, point the +quadlet at it, deploy. The entrypoint is quantization-aware, so the only repo +change a swap makes is the quadlet's four env vars. + +## Why this exists + +The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not +the GPU — it was the **download** (30.9 GB across 7 shards, hand-verified) and +the **serving-path decision** (which attention backend, which drafter, which +quantization flag). Both are now mechanical: + +- The download is `hf-fetch.sh` — resumable, byte-count + sha256-verified, + idempotent (a re-run only fetches what is missing or corrupt). +- The serving path is the **quantization-aware entrypoint** + (`dankserver/frank-medium/radiance-entrypoint.sh`) — it reads + `quantization_config.quant_method` from the checkpoint's `config.json` and + picks the path. The swap operator never chooses it by hand. + +## The toolchain + +| Tool | Role | +|---|---| +| `scripts/hf-manifest.py` | Generate a fetch manifest from a live HF repo tree: ` ` per line. LFS sha256s come from the resolve endpoint's `X-Linked-ETag`; non-LFS files get a blank hash (size-only check). | +| `scripts/hf-fetch.sh` | Fetch a manifest to a dir, resumable (`curl -C -`), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 `curl -K` file. | +| `scripts/swap-frank-medium.sh` | The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy. | + +The HF token is resolved on the **host** (dankserver) from `$HF_TOKEN`, +`~/.cache/huggingface/token`, or the llamaswap conf. Gated repos need a token +that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant +for `orcarouter/Qwen3.8-27B-Uncensored-FP8`). + +## The swap (one command) + +``` +scripts/swap-frank-medium.sh \ + --repo \ + --target /home/dank/models/ \ + --generate-manifest \ + --apply \ + --deploy +``` + +What it does, in order: + +1. **Manifest** — `hf-manifest.py` lists the repo tree and writes + `scripts/frank-medium-.manifest` (committed, so the payload is + reproducible). Pass `--manifest ` to reuse an existing one. +2. **Download** — stages the manifest + `hf-fetch.sh` on the host and fetches + to `--target`. Resumable: an interrupted swap re-runs and skips what is + already verified. +3. **Detect** — reads `quantization_config.quant_method` from the downloaded + `config.json` and maps it to the serving shape (below). +4. **Apply** — edits the quadlet's four env vars (`SNAP`, `SPEC_METHOD`, + `SPEC`, `KV_MEM`). +5. **Deploy** — `deploy.sh dankserver --service frank-medium --skip-e2e` + (pull, restart, health gate, up to 1800 s — the first start compiles + kernels). + +Without `--apply` the script stops after the download + detection and prints +the quadlet lines to set, so you can eyeball them before committing. + +## Quantization → serving path + +The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches: + +| `quant_method` | Path | Attention | Speculation | Drafter | +|---|---|---|---|---| +| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | MTP self-speculation (`--quantization fp8`) | the model's own MTP head (`mtp.layers.0`) | +| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` | + +The quadlet `SPEC_METHOD`/`SPEC` the swap script sets must match the path: +`fp8 → mtp/8`, `mxfp4|quark → dflash/7`. The FP8 model carries its own +single-layer MTP head (22 tensors), so MTP self-speculation needs no external +drafter and no `fp8_mtp.py` rewrite. The MXFP4 path applies the +`radiance-vllm-mxfp4` patches at every container start and uses the DFlash2 +drafter. + +**R4D allreduce is independent of the weight quantization.** The FP8 path +uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be +tried later for a speed bump (`R4D_ATTN=1`), but the MXFP4 path is the one +tested with it. + +## The quadlet contract + +A swap edits exactly four env vars in +`dankserver/frank-medium/quadlet/frank-medium.container`: + +- `SNAP=/models/` — the container path (the host dir + `/home/dank/models/` is mounted at `/models`). +- `SPEC_METHOD=mtp|dflash` — the speculative-decoding method. +- `SPEC=8|7` — the speculative depth. +- `KV_MEM=0` — the KV pool pin in MiB; `0` = auto-size (the documented + plain-FP8 default). After the first healthy start, recalibrate with + `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards. + +The deploy handler (`deploy-remote.sh deploy_llm_medium`) reads `SNAP` and +`SPEC_METHOD` from the quadlet (the source of truth), maps `/models/` → +`/home/dank/models/`, and gates on the right assets: the drafter only for +`dflash`, `r4d.so` only for `mxfp4`/`quark`. So a swap that points the quadlet +at a new checkpoint is enough — the handler follows. + +## Verify (capability battery) + +After the deploy is green, run the battery (read-only — it does not stop the +service): + +``` +python3 scripts/bench-conversation.py http://192.168.0.235:7770 \ + --model frank-medium --salt --max-tokens 512 --burst 4 +``` + +Then the capability checks (all must pass): + +- `/health` → 200, `/v1/models` → `frank-medium`. +- A chat completion returns `content` **and** reasoning (vLLM emits the + `reasoning` field; the harness auto-detects it). +- MTP speculation: the boot log's `SpecDecoding metrics` line shows a mean + acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2% + draft acceptance). +- Tool calls: a `tools` request returns `tool_calls`. +- Vision: an image request is answered (the FP8 checkpoint serves vision + despite `language_model_only: true` in `config.json`). + +## Rollback + +The previous checkpoint stays on disk (a swap never deletes it). To roll +back: + +1. Point the quadlet at the old checkpoint (`SNAP`, `SPEC_METHOD`, `SPEC` — + the old values are in git history). +2. `deploy.sh dankserver --service frank-medium --skip-e2e`. +3. Re-run the capability battery. + +The 2026-09-20 rollback target is the MXFP4 checkpoint at +`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single +safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`. + +## 2026-09-20 swap record (MXFP4 → FP8) + +- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4, + 19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise + 128×128, dynamic activation quant, 30.9 GB / 7 shards). +- **Serving path**: DFlash2 external drafter → MTP self-speculation + (`SPEC_METHOD=mtp`, `SPEC=8`); `R4D` → `ROCM_AITER_UNIFIED_ATTN`. +- **MTP**: mean acceptance length 5.66/8, 66.2% draft acceptance, + per-position rates 0.89 → 0.41. +- **KV pool**: auto-sized to 461,540 tokens (1.76× full-window) — smaller than + MXFP4's 943,581 because FP8 weights are bigger (30.9 vs 19.4 GiB). +- **Battery** (vs MXFP4 baseline): geomean decode 85.9 vs 92.4 t/s (−7%), + mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s + (−27%), burst-4 aggregate 179.7 vs 248.9 t/s (−28%). The trade-off is the + expected one: higher precision (8-bit vs 4-bit weights), decode within 7%, + TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path + vs R4D). diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 729631a..e514d83 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-medium vllm-radiance MXFP4 Runbook -description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201), with a DFlash2-FP8 drafter. Vision is served by the loaded checkpoint (verified on synthetic images 2026-09-20). The sections below describe the 2026-09-16 two-replica era. +description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era. tags: - frank-medium - dankserver @@ -28,6 +28,16 @@ It replaced the mainline-llama.cpp layer-split deployment model had earlier run here as two single-GPU replicas, which is the shape most of this page describes. +**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves +`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7 +shards) with MTP self-speculation — the quantization-aware entrypoint reads +`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4 +patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4 +checkpoint above remains on disk as the rollback target. The swap process, +the toolchain, and the measured trade-off are in [[frank-medium-model-swap]]; +the "Deployed flags" table below still shows the MXFP4 shape (the rollback +target). + **Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.** The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every @@ -43,8 +53,8 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Setting | Value | |---|---| -| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` | -| Speculative | `--speculative-config {"method":"dflash","model":"/models/draft","num_speculative_tokens":7}` (z-lab DFlash2-FP8) | +| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) | +| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) | | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) | | Context | `--max-model-len 262144` | | Concurrency | `--max-num-seqs 8`, chunk 8192 | @@ -56,17 +66,17 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Multimodal | `--limit-mm-per-prompt '{"image": 8}'` | | Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) | -**The KV pool is auto-sized, not pinned.** The arm-R benchmark run used -`KV_CACHE_MEMORY_MB=18563072000`, which bought exactly one 262,144-token -sequence (262,720 tokens at 70,656 B/token — a profile inflated by the -8192-chunk activation peak). Production ships no such pin: the pool came up at -**943,581 tokens = 3.59× the full window**, and the boot line read -`Available KV cache memory: 18.17 GiB → GPU KV cache size: 943,581 tokens`. +**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin +(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8 +checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it +leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540 +tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`, +`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4 +rollback target auto-sized to 943,581 tokens (3.59×) on the same cards. Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16 -full-attention layers plus the drafter's. Container RSS measured 28.4 GiB -against the host's 62 GiB and stayed flat through a 4-way burst at 512-token -max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the -two-replica vision shape, not this one. +full-attention layers plus the MTP drafter's. After the first healthy start, +recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB +cards. ## Prefix caching — measured granularity (2026-09-20) diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index e365f87..a837fb6 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -58,13 +58,20 @@ frank-small, port 7771] ## Active Deployment - [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since - 2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 + - ggz14/radiance-vllm-mxfp4 patches) serving - just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8 - drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm - battery that chose it, the auto-sized 943,581-token KV pool, verified - capabilities (tools, reasoning, vision), and the record of the 2026-09-16 - two-replica era + 2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving + orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 + shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP + self-speculation (the model's own single-layer MTP head). The quantization- + aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint + (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 + drafter) remains on disk as the rollback target. The three-arm battery that + chose TP=2, the auto-sized KV pool, and verified capabilities (tools, + reasoning, vision) +- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium: + the streamlining toolchain (hf-manifest.py, hf-fetch.sh, + swap-frank-medium.sh), the quantization-aware entrypoint contract, the + quadlet edit, the deploy, the capability battery, and the rollback path. + The 2026-09-20 MXFP4 → FP8 swap is the worked example - [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 8d1e2cd..9da7dd7 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -116,11 +116,16 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since - 2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the - R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq - with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700 - behind the :7770 endpoint — deployed flags, the three-arm battery that chose - it, verified capabilities, and the record of the 2026-09-16 two-replica era + 2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D + one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 + block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700 + behind the :7770 endpoint — the quantization-aware entrypoint, the + three-arm battery that chose TP=2, verified capabilities, and the record of + the 2026-09-16 two-replica era +- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium: + the streamlining toolchain (hf-manifest.py, hf-fetch.sh, + swap-frank-medium.sh), the quantization-aware entrypoint contract, the + quadlet edit, the deploy, the capability battery, and the rollback path - [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index f4ceac0..83b6311 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,51 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command + +- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4, + 19.37 GiB single shard, DFlash2 external drafter) → + `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, + dynamic activation quant, 30.9 GB / 7 shards, its own single-layer MTP head). + The quantization-aware entrypoint (`radiance-entrypoint.sh`) reads + `quant_method` from `config.json` and serves the plain FP8 path — no MXFP4 + patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`, MTP + self-speculation (`SPEC_METHOD=mtp`, `SPEC=8`). The quadlet edit was four + env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`); the deploy handler + (`deploy-remote.sh`) now reads the serving shape from the quadlet and gates + on the right assets (drafter only for `dflash`, `r4d.so` only for + `mxfp4`/`quark`). +- **MTP measured, not inferred.** Boot log `SpecDecoding metrics`: mean + acceptance length **5.66/8**, 66.2 % draft acceptance, per-position rates + 0.89 → 0.41. The FP8 model carries its own single-layer MTP head (22 + tensors), so self-speculation needs no external drafter and no `fp8_mtp.py` + rewrite. +- **KV pool auto-sized to 461,540 tokens (1.76× the window)** — smaller than + MXFP4's 943,581 (3.59×) because the FP8 weights are bigger (30.9 vs 19.4 + GiB). `KV_MEM=0` (the documented plain-FP8 default); recalibrate with + `calibrate-kv.sh` after the first healthy start. +- **Battery (vs the MXFP4 baseline).** geomean decode 85.9 vs 92.4 t/s (−7 %), + mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s + (−27 %), burst-4 aggregate 179.7 vs 248.9 t/s (−28 %). The trade-off is the + expected one: higher precision (8-bit vs 4-bit weights), decode within 7 %, + TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path + vs R4D). Vision, tools, and reasoning all verified live. +- **Streamlining — the long pole was the download, not the GPU.** Three tools + make a swap one command: `scripts/hf-manifest.py` (generate a + ` ` manifest from the live repo tree — LFS sha256s + from the resolve endpoint's `X-Linked-ETag`, sizes from `Content-Length`), + `scripts/hf-fetch.sh` (resumable, byte + sha256 verified, idempotent, token + out-of-band via a mode-600 `curl -K` file), and + `scripts/swap-frank-medium.sh` (generate manifest → download → detect + quantization → edit the quadlet → deploy). A swap is now + `swap-frank-medium.sh --repo --target /home/dank/models/ + --generate-manifest --apply --deploy`. Full process: [[frank-medium-model-swap]]. +- **Docs.** [[frank-medium-radiance-mxfp4]] description + deployed-flags table + + KV-pool note flipped to the FP8 shape (the MXFP4 checkpoint is the + rollback target); [[frank-medium-model-swap]] is the new swap runbook; the + LLM index's frank-medium entry flipped. The MXFP4 checkpoint stays on disk + at `/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq`. + ## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime - **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`