feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain
Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256> manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
Frank
@@ -0,0 +1,173 @@
|
||||
---
|
||||
type: runbook
|
||||
title: frank-medium Model Swap
|
||||
description: How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
- model-swap
|
||||
- vllm-radiance
|
||||
- huggingface
|
||||
- runbook
|
||||
timestamp: 2026-09-20
|
||||
---
|
||||
|
||||
# frank-medium Model Swap
|
||||
|
||||
Swapping the checkpoint behind **frank-medium** (Dankserver `:7770`,
|
||||
vllm-radiance TP=2) is a four-step process, and the toolchain below drives all
|
||||
of it. The design goal (2026-09-20, after the MXFP4 → FP8 swap) is that a swap
|
||||
is **one command**: generate a manifest, download it verified, point the
|
||||
quadlet at it, deploy. The entrypoint is quantization-aware, so the only repo
|
||||
change a swap makes is the quadlet's four env vars.
|
||||
|
||||
## Why this exists
|
||||
|
||||
The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not
|
||||
the GPU — it was the **download** (30.9 GB across 7 shards, hand-verified) and
|
||||
the **serving-path decision** (which attention backend, which drafter, which
|
||||
quantization flag). Both are now mechanical:
|
||||
|
||||
- The download is `hf-fetch.sh` — resumable, byte-count + sha256-verified,
|
||||
idempotent (a re-run only fetches what is missing or corrupt).
|
||||
- The serving path is the **quantization-aware entrypoint**
|
||||
(`dankserver/frank-medium/radiance-entrypoint.sh`) — it reads
|
||||
`quantization_config.quant_method` from the checkpoint's `config.json` and
|
||||
picks the path. The swap operator never chooses it by hand.
|
||||
|
||||
## The toolchain
|
||||
|
||||
| Tool | Role |
|
||||
|---|---|
|
||||
| `scripts/hf-manifest.py` | Generate a fetch manifest from a live HF repo tree: `<filename> <size> <sha256-or-blank>` per line. LFS sha256s come from the resolve endpoint's `X-Linked-ETag`; non-LFS files get a blank hash (size-only check). |
|
||||
| `scripts/hf-fetch.sh` | Fetch a manifest to a dir, resumable (`curl -C -`), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 `curl -K` file. |
|
||||
| `scripts/swap-frank-medium.sh` | The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy. |
|
||||
|
||||
The HF token is resolved on the **host** (dankserver) from `$HF_TOKEN`,
|
||||
`~/.cache/huggingface/token`, or the llamaswap conf. Gated repos need a token
|
||||
that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant
|
||||
for `orcarouter/Qwen3.8-27B-Uncensored-FP8`).
|
||||
|
||||
## The swap (one command)
|
||||
|
||||
```
|
||||
scripts/swap-frank-medium.sh \
|
||||
--repo <owner/repo> \
|
||||
--target /home/dank/models/<name> \
|
||||
--generate-manifest \
|
||||
--apply \
|
||||
--deploy
|
||||
```
|
||||
|
||||
What it does, in order:
|
||||
|
||||
1. **Manifest** — `hf-manifest.py` lists the repo tree and writes
|
||||
`scripts/frank-medium-<name>.manifest` (committed, so the payload is
|
||||
reproducible). Pass `--manifest <path>` to reuse an existing one.
|
||||
2. **Download** — stages the manifest + `hf-fetch.sh` on the host and fetches
|
||||
to `--target`. Resumable: an interrupted swap re-runs and skips what is
|
||||
already verified.
|
||||
3. **Detect** — reads `quantization_config.quant_method` from the downloaded
|
||||
`config.json` and maps it to the serving shape (below).
|
||||
4. **Apply** — edits the quadlet's four env vars (`SNAP`, `SPEC_METHOD`,
|
||||
`SPEC`, `KV_MEM`).
|
||||
5. **Deploy** — `deploy.sh dankserver --service frank-medium --skip-e2e`
|
||||
(pull, restart, health gate, up to 1800 s — the first start compiles
|
||||
kernels).
|
||||
|
||||
Without `--apply` the script stops after the download + detection and prints
|
||||
the quadlet lines to set, so you can eyeball them before committing.
|
||||
|
||||
## Quantization → serving path
|
||||
|
||||
The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches:
|
||||
|
||||
| `quant_method` | Path | Attention | Speculation | Drafter |
|
||||
|---|---|---|---|---|
|
||||
| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | MTP self-speculation (`--quantization fp8`) | the model's own MTP head (`mtp.layers.0`) |
|
||||
| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` |
|
||||
|
||||
The quadlet `SPEC_METHOD`/`SPEC` the swap script sets must match the path:
|
||||
`fp8 → mtp/8`, `mxfp4|quark → dflash/7`. The FP8 model carries its own
|
||||
single-layer MTP head (22 tensors), so MTP self-speculation needs no external
|
||||
drafter and no `fp8_mtp.py` rewrite. The MXFP4 path applies the
|
||||
`radiance-vllm-mxfp4` patches at every container start and uses the DFlash2
|
||||
drafter.
|
||||
|
||||
**R4D allreduce is independent of the weight quantization.** The FP8 path
|
||||
uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be
|
||||
tried later for a speed bump (`R4D_ATTN=1`), but the MXFP4 path is the one
|
||||
tested with it.
|
||||
|
||||
## The quadlet contract
|
||||
|
||||
A swap edits exactly four env vars in
|
||||
`dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
|
||||
- `SNAP=/models/<name>` — the container path (the host dir
|
||||
`/home/dank/models/<name>` is mounted at `/models`).
|
||||
- `SPEC_METHOD=mtp|dflash` — the speculative-decoding method.
|
||||
- `SPEC=8|7` — the speculative depth.
|
||||
- `KV_MEM=0` — the KV pool pin in MiB; `0` = auto-size (the documented
|
||||
plain-FP8 default). After the first healthy start, recalibrate with
|
||||
`calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards.
|
||||
|
||||
The deploy handler (`deploy-remote.sh deploy_llm_medium`) reads `SNAP` and
|
||||
`SPEC_METHOD` from the quadlet (the source of truth), maps `/models/` →
|
||||
`/home/dank/models/`, and gates on the right assets: the drafter only for
|
||||
`dflash`, `r4d.so` only for `mxfp4`/`quark`. So a swap that points the quadlet
|
||||
at a new checkpoint is enough — the handler follows.
|
||||
|
||||
## Verify (capability battery)
|
||||
|
||||
After the deploy is green, run the battery (read-only — it does not stop the
|
||||
service):
|
||||
|
||||
```
|
||||
python3 scripts/bench-conversation.py http://192.168.0.235:7770 \
|
||||
--model frank-medium --salt <name> --max-tokens 512 --burst 4
|
||||
```
|
||||
|
||||
Then the capability checks (all must pass):
|
||||
|
||||
- `/health` → 200, `/v1/models` → `frank-medium`.
|
||||
- A chat completion returns `content` **and** reasoning (vLLM emits the
|
||||
`reasoning` field; the harness auto-detects it).
|
||||
- MTP speculation: the boot log's `SpecDecoding metrics` line shows a mean
|
||||
acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2%
|
||||
draft acceptance).
|
||||
- Tool calls: a `tools` request returns `tool_calls`.
|
||||
- Vision: an image request is answered (the FP8 checkpoint serves vision
|
||||
despite `language_model_only: true` in `config.json`).
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous checkpoint stays on disk (a swap never deletes it). To roll
|
||||
back:
|
||||
|
||||
1. Point the quadlet at the old checkpoint (`SNAP`, `SPEC_METHOD`, `SPEC` —
|
||||
the old values are in git history).
|
||||
2. `deploy.sh dankserver --service frank-medium --skip-e2e`.
|
||||
3. Re-run the capability battery.
|
||||
|
||||
The 2026-09-20 rollback target is the MXFP4 checkpoint at
|
||||
`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single
|
||||
safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`.
|
||||
|
||||
## 2026-09-20 swap record (MXFP4 → FP8)
|
||||
|
||||
- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||||
19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise
|
||||
128×128, dynamic activation quant, 30.9 GB / 7 shards).
|
||||
- **Serving path**: DFlash2 external drafter → MTP self-speculation
|
||||
(`SPEC_METHOD=mtp`, `SPEC=8`); `R4D` → `ROCM_AITER_UNIFIED_ATTN`.
|
||||
- **MTP**: mean acceptance length 5.66/8, 66.2% draft acceptance,
|
||||
per-position rates 0.89 → 0.41.
|
||||
- **KV pool**: auto-sized to 461,540 tokens (1.76× full-window) — smaller than
|
||||
MXFP4's 943,581 because FP8 weights are bigger (30.9 vs 19.4 GiB).
|
||||
- **Battery** (vs MXFP4 baseline): geomean decode 85.9 vs 92.4 t/s (−7%),
|
||||
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
|
||||
(−27%), burst-4 aggregate 179.7 vs 248.9 t/s (−28%). The trade-off is the
|
||||
expected one: higher precision (8-bit vs 4-bit weights), decode within 7%,
|
||||
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
|
||||
vs R4D).
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-medium vllm-radiance MXFP4 Runbook
|
||||
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201), with a DFlash2-FP8 drafter. Vision is served by the loaded checkpoint (verified on synthetic images 2026-09-20). The sections below describe the 2026-09-16 two-replica era.
|
||||
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
@@ -28,6 +28,16 @@ It replaced the mainline-llama.cpp layer-split deployment
|
||||
model had earlier run here as two single-GPU replicas, which is the shape most
|
||||
of this page describes.
|
||||
|
||||
**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7
|
||||
shards) with MTP self-speculation — the quantization-aware entrypoint reads
|
||||
`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4
|
||||
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4
|
||||
checkpoint above remains on disk as the rollback target. The swap process,
|
||||
the toolchain, and the measured trade-off are in [[frank-medium-model-swap]];
|
||||
the "Deployed flags" table below still shows the MXFP4 shape (the rollback
|
||||
target).
|
||||
|
||||
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
|
||||
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
|
||||
ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every
|
||||
@@ -43,8 +53,8 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` |
|
||||
| Speculative | `--speculative-config {"method":"dflash","model":"/models/draft","num_speculative_tokens":7}` (z-lab DFlash2-FP8) |
|
||||
| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
|
||||
| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) |
|
||||
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
|
||||
| Context | `--max-model-len 262144` |
|
||||
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
|
||||
@@ -56,17 +66,17 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
|
||||
| Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) |
|
||||
|
||||
**The KV pool is auto-sized, not pinned.** The arm-R benchmark run used
|
||||
`KV_CACHE_MEMORY_MB=18563072000`, which bought exactly one 262,144-token
|
||||
sequence (262,720 tokens at 70,656 B/token — a profile inflated by the
|
||||
8192-chunk activation peak). Production ships no such pin: the pool came up at
|
||||
**943,581 tokens = 3.59× the full window**, and the boot line read
|
||||
`Available KV cache memory: 18.17 GiB → GPU KV cache size: 943,581 tokens`.
|
||||
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
|
||||
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8
|
||||
checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it
|
||||
leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540
|
||||
tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`,
|
||||
`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4
|
||||
rollback target auto-sized to 943,581 tokens (3.59×) on the same cards.
|
||||
Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16
|
||||
full-attention layers plus the drafter's. Container RSS measured 28.4 GiB
|
||||
against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
|
||||
max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
|
||||
two-replica vision shape, not this one.
|
||||
full-attention layers plus the MTP drafter's. After the first healthy start,
|
||||
recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
|
||||
cards.
|
||||
|
||||
## Prefix caching — measured granularity (2026-09-20)
|
||||
|
||||
|
||||
@@ -58,13 +58,20 @@ frank-small, port 7771]
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 +
|
||||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8
|
||||
drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm
|
||||
battery that chose it, the auto-sized 943,581-token KV pool, verified
|
||||
capabilities (tools, reasoning, vision), and the record of the 2026-09-16
|
||||
two-replica era
|
||||
2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
|
||||
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
|
||||
shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP
|
||||
self-speculation (the model's own single-layer MTP head). The quantization-
|
||||
aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint
|
||||
(just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8
|
||||
drafter) remains on disk as the rollback target. The three-arm battery that
|
||||
chose TP=2, the auto-sized KV pool, and verified capabilities (tools,
|
||||
reasoning, vision)
|
||||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||||
quadlet edit, the deploy, the capability battery, and the rollback path.
|
||||
The 2026-09-20 MXFP4 → FP8 swap is the worked example
|
||||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||||
|
||||
@@ -116,11 +116,16 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the
|
||||
R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700
|
||||
behind the :7770 endpoint — deployed flags, the three-arm battery that chose
|
||||
it, verified capabilities, and the record of the 2026-09-16 two-replica era
|
||||
2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D
|
||||
one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3
|
||||
block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700
|
||||
behind the :7770 endpoint — the quantization-aware entrypoint, the
|
||||
three-arm battery that chose TP=2, verified capabilities, and the record of
|
||||
the 2026-09-16 two-replica era
|
||||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||||
quadlet edit, the deploy, the capability battery, and the rollback path
|
||||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
|
||||
mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
|
||||
Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
|
||||
|
||||
@@ -11,6 +11,51 @@ timestamp: 2026-09-14
|
||||
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
|
||||
|
||||
- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||||
19.37 GiB single shard, DFlash2 external drafter) →
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128,
|
||||
dynamic activation quant, 30.9 GB / 7 shards, its own single-layer MTP head).
|
||||
The quantization-aware entrypoint (`radiance-entrypoint.sh`) reads
|
||||
`quant_method` from `config.json` and serves the plain FP8 path — no MXFP4
|
||||
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`, MTP
|
||||
self-speculation (`SPEC_METHOD=mtp`, `SPEC=8`). The quadlet edit was four
|
||||
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`); the deploy handler
|
||||
(`deploy-remote.sh`) now reads the serving shape from the quadlet and gates
|
||||
on the right assets (drafter only for `dflash`, `r4d.so` only for
|
||||
`mxfp4`/`quark`).
|
||||
- **MTP measured, not inferred.** Boot log `SpecDecoding metrics`: mean
|
||||
acceptance length **5.66/8**, 66.2 % draft acceptance, per-position rates
|
||||
0.89 → 0.41. The FP8 model carries its own single-layer MTP head (22
|
||||
tensors), so self-speculation needs no external drafter and no `fp8_mtp.py`
|
||||
rewrite.
|
||||
- **KV pool auto-sized to 461,540 tokens (1.76× the window)** — smaller than
|
||||
MXFP4's 943,581 (3.59×) because the FP8 weights are bigger (30.9 vs 19.4
|
||||
GiB). `KV_MEM=0` (the documented plain-FP8 default); recalibrate with
|
||||
`calibrate-kv.sh` after the first healthy start.
|
||||
- **Battery (vs the MXFP4 baseline).** geomean decode 85.9 vs 92.4 t/s (−7 %),
|
||||
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
|
||||
(−27 %), burst-4 aggregate 179.7 vs 248.9 t/s (−28 %). The trade-off is the
|
||||
expected one: higher precision (8-bit vs 4-bit weights), decode within 7 %,
|
||||
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
|
||||
vs R4D). Vision, tools, and reasoning all verified live.
|
||||
- **Streamlining — the long pole was the download, not the GPU.** Three tools
|
||||
make a swap one command: `scripts/hf-manifest.py` (generate a
|
||||
`<filename> <size> <sha256>` manifest from the live repo tree — LFS sha256s
|
||||
from the resolve endpoint's `X-Linked-ETag`, sizes from `Content-Length`),
|
||||
`scripts/hf-fetch.sh` (resumable, byte + sha256 verified, idempotent, token
|
||||
out-of-band via a mode-600 `curl -K` file), and
|
||||
`scripts/swap-frank-medium.sh` (generate manifest → download → detect
|
||||
quantization → edit the quadlet → deploy). A swap is now
|
||||
`swap-frank-medium.sh --repo <owner/repo> --target /home/dank/models/<name>
|
||||
--generate-manifest --apply --deploy`. Full process: [[frank-medium-model-swap]].
|
||||
- **Docs.** [[frank-medium-radiance-mxfp4]] description + deployed-flags table
|
||||
+ KV-pool note flipped to the FP8 shape (the MXFP4 checkpoint is the
|
||||
rollback target); [[frank-medium-model-swap]] is the new swap runbook; the
|
||||
LLM index's frank-medium entry flipped. The MXFP4 checkpoint stays on disk
|
||||
at `/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq`.
|
||||
|
||||
## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime
|
||||
|
||||
- **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`
|
||||
|
||||
Reference in New Issue
Block a user