serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-model-swap.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
runbook	frank-medium Model Swap	How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path.	frank-medium dankserver model-swap vllm-radiance huggingface runbook	2026-09-20

frank-medium Model Swap

Swapping the checkpoint behind frank-medium (Dankserver :7770, vllm-radiance TP=2) is a four-step process, and the toolchain below drives all of it. The design goal (2026-09-20, after the MXFP4 → FP8 swap) is that a swap is one command: generate a manifest, download it verified, point the quadlet at it, deploy. The entrypoint is quantization-aware, so the only repo change a swap makes is the quadlet's four env vars.

Why this exists

The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not the GPU — it was the download (30.9 GB across 7 shards, hand-verified) and the serving-path decision (which attention backend, which drafter, which quantization flag). Both are now mechanical:

• The download is hf-fetch.sh — resumable, byte-count + sha256-verified, idempotent (a re-run only fetches what is missing or corrupt).
• The serving path is the quantization-aware entrypoint (dankserver/frank-medium/radiance-entrypoint.sh) — it reads quantization_config.quant_method from the checkpoint's config.json and picks the path. The swap operator never chooses it by hand.

The toolchain

Tool	Role
scripts/hf-manifest.py	Generate a fetch manifest from a live HF repo tree: <filename> <size> <sha256-or-blank> per line. LFS sha256s come from the resolve endpoint's X-Linked-ETag; non-LFS files get a blank hash (size-only check).
scripts/hf-fetch.sh	Fetch a manifest to a dir, resumable (curl -C -), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 curl -K file.
scripts/swap-frank-medium.sh	The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy.

The HF token is resolved on the host (dankserver) from $HF_TOKEN, ~/.cache/huggingface/token, or the llamaswap conf. Gated repos need a token that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant for orcarouter/Qwen3.8-27B-Uncensored-FP8).

The swap (one command)

scripts/swap-frank-medium.sh \
    --repo <owner/repo> \
    --target /home/dank/models/<name> \
    --generate-manifest \
    --apply \
    --deploy

What it does, in order:

1. Manifest — hf-manifest.py lists the repo tree and writes scripts/frank-medium-<name>.manifest (committed, so the payload is reproducible). Pass --manifest <path> to reuse an existing one.
2. Download — stages the manifest + hf-fetch.sh on the host and fetches to --target. Resumable: an interrupted swap re-runs and skips what is already verified.
3. Detect — reads quantization_config.quant_method from the downloaded config.json and maps it to the serving shape (below).
4. Apply — edits the quadlet's four env vars (SNAP, SPEC_METHOD, SPEC, KV_MEM).
5. Deploy — deploy.sh dankserver --service frank-medium --skip-e2e (pull, restart, health gate, up to 1800 s — the first start compiles kernels).

Without --apply the script stops after the download + detection and prints the quadlet lines to set, so you can eyeball them before committing.

Quantization → serving path

The entrypoint (radiance-entrypoint.sh) reads quant_method and branches:

quant_method	Path	Attention	Speculation	Drafter
fp8	plain FP8	ROCM_AITER_UNIFIED_ATTN	DFlash2 external drafter (A/B winner)	/models/Qwen3.8-27B-DFlash2-FP8
mxfp4 / quark / empty	MXFP4	R4D (if R4D_ATTN=1)	DFlash2 external drafter	/models/Qwen3.8-27B-DFlash2-FP8

The quadlet SPEC_METHOD/SPEC the swap script sets: both paths use dflash/7 (the DFlash2 drafter is quantization-agnostic — it proposes tokens, the target verifies, so it works with any Qwen3.8-27B target). The FP8 model also carries its own single-layer MTP head (22 tensors), so MTP self-speculation (mtp/8) is available as an alternative — but A/B tested on this checkpoint, DFlash2 wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4) despite a lower accept length (4.24 vs 5.66), because the small separate drafter steps faster than the target's own MTP layer applied autoregressively. The MXFP4 path additionally applies the radiance-vllm-mxfp4 patches at every container start.

R4D allreduce is independent of the weight quantization. The FP8 path uses ROCM_AITER_UNIFIED_ATTN (the documented plain-FP8 default); R4D can be tried later for a speed bump (R4D_ATTN=1), but the MXFP4 path is the one tested with it.

The quadlet contract

A swap edits exactly four env vars in dankserver/frank-medium/quadlet/frank-medium.container:

• SNAP=/models/<name> — the container path (the host dir /home/dank/models/<name> is mounted at /models).
• SPEC_METHOD=mtp|dflash — the speculative-decoding method.
• SPEC=8|7 — the speculative depth.
• KV_MEM=0 — the KV pool pin in MiB; 0 = auto-size (the documented plain-FP8 default). After the first healthy start, recalibrate with calibrate-kv.sh to use the extra VRAM on the 34.2 GiB cards.

The deploy handler (deploy-remote.sh deploy_llm_medium) reads SNAP and SPEC_METHOD from the quadlet (the source of truth), maps /models/ → /home/dank/models/, and gates on the right assets: the drafter only for dflash, r4d.so only for mxfp4/quark. So a swap that points the quadlet at a new checkpoint is enough — the handler follows.

Verify (capability battery)

After the deploy is green, run the battery (read-only — it does not stop the service):

python3 scripts/bench-conversation.py http://192.168.0.235:7770 \
    --model frank-medium --salt <name> --max-tokens 512 --burst 4

Then the capability checks (all must pass):

• /health → 200, /v1/models → frank-medium.
• A chat completion returns content and reasoning (vLLM emits the reasoning field; the harness auto-detects it).
• MTP speculation: the boot log's SpecDecoding metrics line shows a mean acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2% draft acceptance).
• Tool calls: a tools request returns tool_calls.
• Vision: an image request is answered (the FP8 checkpoint serves vision despite language_model_only: true in config.json).

Rollback

The previous checkpoint stays on disk (a swap never deletes it). To roll back:

1. Point the quadlet at the old checkpoint (SNAP, SPEC_METHOD, SPEC — the old values are in git history).
2. deploy.sh dankserver --service frank-medium --skip-e2e.
3. Re-run the capability battery.

The 2026-09-20 rollback target is the MXFP4 checkpoint at /home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq (19.37 GiB single safetensors) with SPEC_METHOD=dflash, SPEC=7.

2026-09-20 swap record (MXFP4 → FP8)

• Model: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4, 19.37 GiB) → orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise 128×128, dynamic activation quant, 30.9 GB / 7 shards).
• Serving path: DFlash2 external drafter → DFlash2 external drafter (kept; R4D → ROCM_AITER_UNIFIED_ATTN). After the swap, MTP self-speculation (mtp/8) was A/B tested against the DFlash2 drafter (dflash/7) on the FP8 target.
• DFlash2 vs MTP A/B (the deployed shape is DFlash2):
  • DFlash2: mean accept 4.24/7, geomean decode 100.0 t/s, TTFT 0.19 s, prefill 923.0 t/s, burst-4 287.1 t/s, KV pool 384,238 (1.47×).
  • MTP: mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s, prefill 1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
  • DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on TTFT and prefill. Accept length is a proxy, not the metric — the small separate drafter steps faster than the target's own MTP layer applied autoregressively, and the faster step more than makes up the lower accept length. FP8 + DFlash2 (100.0 t/s) also beats the original MXFP4 + DFlash2 (92.4 t/s) — the higher-precision FP8 target verifies more of the drafter's proposals.
• KV pool: auto-sized to 384,238 tokens (1.47× full-window) — the DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540 (1.76×); both smaller than MXFP4's 943,581 (3.59×) because FP8 weights are bigger (30.9 vs 19.4 GiB).