feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain
Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256> manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
Frank committed
1 parent
2458d754ce
commit
c1a4ccbbcd
8 files changed
+534
-45
No files matched your search
@@ -0,0 +1,173 @@
|
||||
---
|
||||
type: runbook
|
||||
title: frank-medium Model Swap
|
||||
description: How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
- model-swap
|
||||
- vllm-radiance
|
||||
- huggingface
|
||||
- runbook
|
||||
timestamp: 2026-09-20
|
||||
---
|
||||
|
||||
# frank-medium Model Swap
|
||||
|
||||
Swapping the checkpoint behind **frank-medium** (Dankserver `:7770`,
|
||||
vllm-radiance TP=2) is a four-step process, and the toolchain below drives all
|
||||
of it. The design goal (2026-09-20, after the MXFP4 → FP8 swap) is that a swap
|
||||
is **one command**: generate a manifest, download it verified, point the
|
||||
quadlet at it, deploy. The entrypoint is quantization-aware, so the only repo
|
||||
change a swap makes is the quadlet's four env vars.
|
||||
|
||||
## Why this exists
|
||||
|
||||
The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not
|
||||
the GPU — it was the **download** (30.9 GB across 7 shards, hand-verified) and
|
||||
the **serving-path decision** (which attention backend, which drafter, which
|
||||
quantization flag). Both are now mechanical:
|
||||
|
||||
- The download is `hf-fetch.sh` — resumable, byte-count + sha256-verified,
|
||||
idempotent (a re-run only fetches what is missing or corrupt).
|
||||
- The serving path is the **quantization-aware entrypoint**
|
||||
(`dankserver/frank-medium/radiance-entrypoint.sh`) — it reads
|
||||
`quantization_config.quant_method` from the checkpoint's `config.json` and
|
||||
picks the path. The swap operator never chooses it by hand.
|
||||
|
||||
## The toolchain
|
||||
|
||||
| Tool | Role |
|
||||
|---|---|
|
||||
| `scripts/hf-manifest.py` | Generate a fetch manifest from a live HF repo tree: `<filename> <size> <sha256-or-blank>` per line. LFS sha256s come from the resolve endpoint's `X-Linked-ETag`; non-LFS files get a blank hash (size-only check). |
|
||||
| `scripts/hf-fetch.sh` | Fetch a manifest to a dir, resumable (`curl -C -`), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 `curl -K` file. |
|
||||
| `scripts/swap-frank-medium.sh` | The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy. |
|
||||
|
||||
The HF token is resolved on the **host** (dankserver) from `$HF_TOKEN`,
|
||||
`~/.cache/huggingface/token`, or the llamaswap conf. Gated repos need a token
|
||||
that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant
|
||||
for `orcarouter/Qwen3.8-27B-Uncensored-FP8`).
|
||||
|
||||
## The swap (one command)
|
||||
|
||||
```
|
||||
scripts/swap-frank-medium.sh \
|
||||
--repo <owner/repo> \
|
||||
--target /home/dank/models/<name> \
|
||||
--generate-manifest \
|
||||
--apply \
|
||||
--deploy
|
||||
```
|
||||
|
||||
What it does, in order:
|
||||
|
||||
1. **Manifest** — `hf-manifest.py` lists the repo tree and writes
|
||||
`scripts/frank-medium-<name>.manifest` (committed, so the payload is
|
||||
reproducible). Pass `--manifest <path>` to reuse an existing one.
|
||||
2. **Download** — stages the manifest + `hf-fetch.sh` on the host and fetches
|
||||
to `--target`. Resumable: an interrupted swap re-runs and skips what is
|
||||
already verified.
|
||||
3. **Detect** — reads `quantization_config.quant_method` from the downloaded
|
||||
`config.json` and maps it to the serving shape (below).
|
||||
4. **Apply** — edits the quadlet's four env vars (`SNAP`, `SPEC_METHOD`,
|
||||
`SPEC`, `KV_MEM`).
|
||||
5. **Deploy** — `deploy.sh dankserver --service frank-medium --skip-e2e`
|
||||
(pull, restart, health gate, up to 1800 s — the first start compiles
|
||||
kernels).
|
||||
|
||||
Without `--apply` the script stops after the download + detection and prints
|
||||
the quadlet lines to set, so you can eyeball them before committing.
|
||||
|
||||
## Quantization → serving path
|
||||
|
||||
The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches:
|
||||
|
||||
| `quant_method` | Path | Attention | Speculation | Drafter |
|
||||
|---|---|---|---|---|
|
||||
| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | MTP self-speculation (`--quantization fp8`) | the model's own MTP head (`mtp.layers.0`) |
|
||||
| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` |
|
||||
|
||||
The quadlet `SPEC_METHOD`/`SPEC` the swap script sets must match the path:
|
||||
`fp8 → mtp/8`, `mxfp4|quark → dflash/7`. The FP8 model carries its own
|
||||
single-layer MTP head (22 tensors), so MTP self-speculation needs no external
|
||||
drafter and no `fp8_mtp.py` rewrite. The MXFP4 path applies the
|
||||
`radiance-vllm-mxfp4` patches at every container start and uses the DFlash2
|
||||
drafter.
|
||||
|
||||
**R4D allreduce is independent of the weight quantization.** The FP8 path
|
||||
uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be
|
||||
tried later for a speed bump (`R4D_ATTN=1`), but the MXFP4 path is the one
|
||||
tested with it.
|
||||
|
||||
## The quadlet contract
|
||||
|
||||
A swap edits exactly four env vars in
|
||||
`dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
|
||||
- `SNAP=/models/<name>` — the container path (the host dir
|
||||
`/home/dank/models/<name>` is mounted at `/models`).
|
||||
- `SPEC_METHOD=mtp|dflash` — the speculative-decoding method.
|
||||
- `SPEC=8|7` — the speculative depth.
|
||||
- `KV_MEM=0` — the KV pool pin in MiB; `0` = auto-size (the documented
|
||||
plain-FP8 default). After the first healthy start, recalibrate with
|
||||
`calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards.
|
||||
|
||||
The deploy handler (`deploy-remote.sh deploy_llm_medium`) reads `SNAP` and
|
||||
`SPEC_METHOD` from the quadlet (the source of truth), maps `/models/` →
|
||||
`/home/dank/models/`, and gates on the right assets: the drafter only for
|
||||
`dflash`, `r4d.so` only for `mxfp4`/`quark`. So a swap that points the quadlet
|
||||
at a new checkpoint is enough — the handler follows.
|
||||
|
||||
## Verify (capability battery)
|
||||
|
||||
After the deploy is green, run the battery (read-only — it does not stop the
|
||||
service):
|
||||
|
||||
```
|
||||
python3 scripts/bench-conversation.py http://192.168.0.235:7770 \
|
||||
--model frank-medium --salt <name> --max-tokens 512 --burst 4
|
||||
```
|
||||
|
||||
Then the capability checks (all must pass):
|
||||
|
||||
- `/health` → 200, `/v1/models` → `frank-medium`.
|
||||
- A chat completion returns `content` **and** reasoning (vLLM emits the
|
||||
`reasoning` field; the harness auto-detects it).
|
||||
- MTP speculation: the boot log's `SpecDecoding metrics` line shows a mean
|
||||
acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2%
|
||||
draft acceptance).
|
||||
- Tool calls: a `tools` request returns `tool_calls`.
|
||||
- Vision: an image request is answered (the FP8 checkpoint serves vision
|
||||
despite `language_model_only: true` in `config.json`).
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous checkpoint stays on disk (a swap never deletes it). To roll
|
||||
back:
|
||||
|
||||
1. Point the quadlet at the old checkpoint (`SNAP`, `SPEC_METHOD`, `SPEC` —
|
||||
the old values are in git history).
|
||||
2. `deploy.sh dankserver --service frank-medium --skip-e2e`.
|
||||
3. Re-run the capability battery.
|
||||
|
||||
The 2026-09-20 rollback target is the MXFP4 checkpoint at
|
||||
`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single
|
||||
safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`.
|
||||
|
||||
## 2026-09-20 swap record (MXFP4 → FP8)
|
||||
|
||||
- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||||
19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise
|
||||
128×128, dynamic activation quant, 30.9 GB / 7 shards).
|
||||
- **Serving path**: DFlash2 external drafter → MTP self-speculation
|
||||
(`SPEC_METHOD=mtp`, `SPEC=8`); `R4D` → `ROCM_AITER_UNIFIED_ATTN`.
|
||||
- **MTP**: mean acceptance length 5.66/8, 66.2% draft acceptance,
|
||||
per-position rates 0.89 → 0.41.
|
||||
- **KV pool**: auto-sized to 461,540 tokens (1.76× full-window) — smaller than
|
||||
MXFP4's 943,581 because FP8 weights are bigger (30.9 vs 19.4 GiB).
|
||||
- **Battery** (vs MXFP4 baseline): geomean decode 85.9 vs 92.4 t/s (−7%),
|
||||
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
|
||||
(−27%), burst-4 aggregate 179.7 vs 248.9 t/s (−28%). The trade-off is the
|
||||
expected one: higher precision (8-bit vs 4-bit weights), decode within 7%,
|
||||
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
|
||||
vs R4D).
|
||||
Reference in new issue
Block a user