feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain

Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq ->
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
shards, its own single-layer MTP head). The quantization-aware entrypoint
serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP
self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8
(66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x),
battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/
248.9). MXFP4 checkpoint stays on disk as the rollback target.

Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256>
manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte +
sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate
manifest -> download -> detect quantization -> edit the quadlet -> deploy).

Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes,
AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
Frank committed 2026-09-20 21:53:00 -07:00
1 parent 2458d754ce
commit c1a4ccbbcd
8 files changed
+534 -45

No files matched your search

+42 -20
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | Qwen3.8-27B-Uncensored MXFP4-awq with the z-lab DFlash2-FP8 drafter (depth 7, live mean accept length ~4.1), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 943,581-token KV pool = 3.59x the window), fp8 KV, prefix caching, vision | 7770 |
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | orcarouter Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards) with MTP self-speculation (depth 8, live mean accept length 5.66/8), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 461,540-token KV pool = 1.76x the window), fp8 KV, prefix caching, vision | 7770 |
| Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
@@ -334,18 +334,31 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-medium — radiance TP=2, DFlash2-FP8 drafter (Dankserver, ACTIVE)
### frank-medium — radiance TP=2, MTP self-speculation (Dankserver, ACTIVE)
vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
`dankserver/frank-medium/quadlet/frank-medium.container`) serving
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4) with the
**z-lab DFlash2 FP8 drafter** (`num_speculative_tokens: 7`, live mean accept
~4.1) in **one tensor-parallel process across both R9700s**
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, dynamic
activation quant, 30.9 GB / 7 shards) with **MTP self-speculation**
(`num_speculative_tokens: 8`, the model's own single-layer MTP head, live mean
accept 5.66/8) in **one tensor-parallel process across both R9700s**
(`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8
KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized
to 943,581 tokens (3.59× the full window). Tools via `qwen3_coder`, reasoning
via `qwen3` (the field is `reasoning`, not llama.cpp's `reasoning_content`),
vision through the Gemma4 merge path with `MM_MERGE_ENGINE=AXROW_SEPARATE`.
to 461,540 tokens (1.76× the full window; smaller than the MXFP4 rollback
target's 943,581 because the FP8 weights are bigger). Tools via `qwen3_coder`,
reasoning via `qwen3` (the field is `reasoning`, not llama.cpp's
`reasoning_content`), vision through the Gemma4 merge path.
**The entrypoint is quantization-aware.** `radiance-entrypoint.sh` reads
`quantization_config.quant_method` from the checkpoint's `config.json` and
serves the plain FP8 path (no MXFP4 patches, `--quantization fp8`,
`ROCM_AITER_UNIFIED_ATTN`, MTP self-speculation) or the MXFP4 patch-and-serve
path (the DFlash2 external drafter). A checkpoint swap is therefore four quadlet
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`) — the deploy handler reads
the serving shape from the quadlet and gates on the right assets. The MXFP4
checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` + the z-lab
DFlash2-FP8 drafter) remains on disk as the rollback target. Full swap
process + toolchain: [[frank-medium-model-swap]].
**TP=2 runs because the board changed.** The predecessor board (TUF GAMING
B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-`
@@ -365,18 +378,26 @@ vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured
against production after deploy: 92.4 t/s, 0.22 s, 248.9.
Files: `dankserver/frank-medium/quadlet/frank-medium.container`,
`dankserver/frank-medium/radiance-entrypoint.sh`,
`dankserver/frank-medium/frank-medium-healthcheck.py`. The llama.cpp layer-split
trio (`scripts/frank-medium.service`, `.env`, `serve-frank-medium.sh`) is
archived in `scripts/.AGENTTRASH/20260920-llama-layersplit/` and is the
rollback target; `deploy-remote.sh`'s `retire_llama_medium_unit` moved the
installed unit symlink there rather than `systemctl disable`, which would delete
it. The deploy handler gates on the image tag, both model configs, the patch
clone + jinja template, and `libr4d`, then waits for :7770 and gates on
`/health`.
`dankserver/frank-medium/radiance-entrypoint.sh` (quantization-aware),
`dankserver/frank-medium/frank-medium-healthcheck.py`. Swap toolchain:
`scripts/hf-manifest.py` (generate a `<filename> <size> <sha256>` manifest
from the live repo tree), `scripts/hf-fetch.sh` (resumable, byte + sha256
verified, idempotent fetch), `scripts/swap-frank-medium.sh` (generate
manifest → download → detect quantization → edit the quadlet → deploy). The
llama.cpp layer-split trio (`scripts/frank-medium.service`, `.env`,
`serve-frank-medium.sh`) is archived in `scripts/.AGENTTRASH/
20260920-llama-layersplit/`; `deploy-remote.sh`'s `retire_llama_medium_unit`
moved the installed unit symlink there rather than `systemctl disable`, which
would delete it. The deploy handler reads `SNAP`/`SPEC_METHOD` from the
quadlet, gates on the image tag + the checkpoint's `config.json` (drafter only
for `dflash`, `libr4d` only for `mxfp4`/`quark`), then waits for :7770 and
gates on `/health`.
Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
Rollback: [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).
Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` (serving
details); swap process + toolchain: `serverAppConfigs-docs/LLM/
frank-medium-model-swap.md`.
Rollback: the MXFP4 checkpoint on disk (`SPEC_METHOD=dflash`, `SPEC=7`);
before that, [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -721,7 +742,8 @@ Active:
|---------|---------|
| frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-medium (radiance TP=2 + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| frank-medium (radiance TP=2 + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| frank-medium model swap (toolchain + process) | `serverAppConfigs-docs/LLM/frank-medium-model-swap.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |