serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-model-swap.md

---
type: runbook
title: frank-medium Model Swap
description: RETIRED-era procedure (frank-medium is a retired endpoint; `:7770` on Dankserver is [[frank-archon]] today and the live swap law is [[Model Swap and Rollback]]) — how a checkpoint swap behind frank-medium worked: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path.
tags:
  - frank-medium
  - dankserver
  - model-swap
  - vllm-radiance
  - huggingface
  - runbook
timestamp: 2026-09-20
---

# frank-medium Model Swap

**Historical.** frank-medium is a retired endpoint — `:7770` on Dankserver is
[[frank-archon]] today, and the live procedure is [[Model Swap and Rollback]].
This page records how a swap worked in that era: a four-step process whose design
goal (2026-09-20, after the MXFP4 → FP8 swap) was that a swap be **one command** —
generate a manifest, download it verified, point the quadlet at it, deploy. The
entrypoint was quantization-aware, so the only repo change a swap made was the
quadlet's four env vars.

## Why this exists

The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not
the GPU — it was the **download** (30.9 GB across 7 shards, hand-verified) and
the **serving-path decision** (which attention backend, which drafter, which
quantization flag). Both are now mechanical:

- The download is `hf-fetch.sh` — resumable, byte-count + sha256-verified,
  idempotent (a re-run only fetches what is missing or corrupt).
- The serving path is the **quantization-aware entrypoint**
  (`dankserver/frank-medium/radiance-entrypoint.sh`) — it reads
  `quantization_config.quant_method` from the checkpoint's `config.json` and
  picks the path. The swap operator never chooses it by hand.

## The toolchain

| Tool | Role |
|---|---|
| `scripts/hf-manifest.py` | Generate a fetch manifest from a live HF repo tree: `<filename> <size> <sha256-or-blank>` per line. LFS sha256s come from the resolve endpoint's `X-Linked-ETag`; non-LFS files get a blank hash (size-only check). |
| `scripts/hf-fetch.sh` | Fetch a manifest to a dir, resumable (`curl -C -`), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 `curl -K` file. |
| `scripts/swap-frank-medium.sh` | The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy. |

The HF token is resolved on the **host** (dankserver) from `$HF_TOKEN`,
`~/.cache/huggingface/token`, or the llamaswap conf. Gated repos need a token
that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant
for `orcarouter/Qwen3.8-27B-Uncensored-FP8`).

## The swap (one command)

```
scripts/swap-frank-medium.sh \
    --repo <owner/repo> \
    --target /home/dank/models/<name> \
    --generate-manifest \
    --apply \
    --deploy
```

What it does, in order:

1. **Manifest** — `hf-manifest.py` lists the repo tree and writes
   `scripts/frank-medium-<name>.manifest` (committed, so the payload is
   reproducible). Pass `--manifest <path>` to reuse an existing one.
2. **Download** — stages the manifest + `hf-fetch.sh` on the host and fetches
   to `--target`. Resumable: an interrupted swap re-runs and skips what is
   already verified.
3. **Detect** — reads `quantization_config.quant_method` from the downloaded
   `config.json` and maps it to the serving shape (below).
4. **Apply** — edits the quadlet's four env vars (`SNAP`, `SPEC_METHOD`,
   `SPEC`, `KV_MEM`).
5. **Deploy** — `deploy.sh dankserver --service frank-medium --skip-e2e`
   (pull, restart, health gate, up to 1800 s — the first start compiles
   kernels).

Without `--apply` the script stops after the download + detection and prints
the quadlet lines to set, so you can eyeball them before committing.

## Quantization → serving path

The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches:

| `quant_method` | Path | Attention | Speculation | Drafter |
|---|---|---|---|---|
| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | **DFlash2 external drafter** (A/B winner) | `/models/Qwen3.8-27B-DFlash2-FP8` |
| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` |

The quadlet `SPEC_METHOD`/`SPEC` the swap script sets: **both paths use
`dflash/7`** (the DFlash2 drafter is quantization-agnostic — it proposes
tokens, the target verifies, so it works with any Qwen3.8-27B target). The
FP8 model also carries its own single-layer MTP head (22 tensors), so MTP
self-speculation (`mtp/8`) is available as an alternative — but A/B tested on
this checkpoint, **DFlash2 wins on decode** (100.0 vs 85.9 t/s geomean, 287.1
vs 179.7 t/s burst-4) despite a lower accept length (4.24 vs 5.66), because
the small separate drafter steps faster than the target's own MTP layer
applied autoregressively. The MXFP4 path additionally applies the
`radiance-vllm-mxfp4` patches at every container start.

**R4D allreduce is independent of the weight quantization.** The FP8 path
uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be
tried later for a speed bump (`R4D_ATTN=1`), but the MXFP4 path is the one
tested with it.

## The quadlet contract

A swap edits exactly four env vars in
`dankserver/frank-medium/quadlet/frank-medium.container`:

- `SNAP=/models/<name>` — the container path (the host dir
  `/home/dank/models/<name>` is mounted at `/models`).
- `SPEC_METHOD=mtp|dflash` — the speculative-decoding method.
- `SPEC=8|7` — the speculative depth.
- `KV_MEM=0` — the KV pool pin in MiB; `0` = auto-size (the documented
  plain-FP8 default). After the first healthy start, recalibrate with
  `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards.

The deploy handler (`deploy-remote.sh deploy_llm_medium`) reads `SNAP` and
`SPEC_METHOD` from the quadlet (the source of truth), maps `/models/` →
`/home/dank/models/`, and gates on the right assets: the drafter only for
`dflash`, `r4d.so` only for `mxfp4`/`quark`. So a swap that points the quadlet
at a new checkpoint is enough — the handler follows.

## Verify (capability battery)

After the deploy is green, run the battery (read-only — it does not stop the
service):

```
python3 scripts/bench-conversation.py http://192.168.0.235:7770 \
    --model frank-medium --salt <name> --max-tokens 512 --burst 4
```

Then the capability checks (all must pass):

- `/health` → 200, `/v1/models` → `frank-medium`.
- A chat completion returns `content` **and** reasoning (vLLM emits the
  `reasoning` field; the harness auto-detects it).
- MTP speculation: the boot log's `SpecDecoding metrics` line shows a mean
  acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2%
  draft acceptance).
- Tool calls: a `tools` request returns `tool_calls`.
- Vision: an image request is answered (the FP8 checkpoint serves vision
  despite `language_model_only: true` in `config.json`).

## Rollback

The previous checkpoint stays on disk (a swap never deletes it). To roll
back:

1. Point the quadlet at the old checkpoint (`SNAP`, `SPEC_METHOD`, `SPEC` —
   the old values are in git history).
2. `deploy.sh dankserver --service frank-medium --skip-e2e`.
3. Re-run the capability battery.

The 2026-09-20 rollback target is the MXFP4 checkpoint at
`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single
safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`.

## 2026-09-20 swap record (MXFP4 → FP8)

- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
  19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise
  128×128, dynamic activation quant, 30.9 GB / 7 shards).
- **Serving path**: DFlash2 external drafter → DFlash2 external drafter
  (kept; `R4D` → `ROCM_AITER_UNIFIED_ATTN`). After the swap, MTP
  self-speculation (`mtp/8`) was A/B tested against the DFlash2 drafter
  (`dflash/7`) on the FP8 target.
- **DFlash2 vs MTP A/B** (the deployed shape is DFlash2):
  - DFlash2: mean accept 4.24/7, geomean decode **100.0 t/s**, TTFT 0.19 s,
    prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×).
  - MTP: mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s, prefill
    1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
  - **DFlash2 wins on decode (+16 %) and burst-4 (+60 %)**; MTP wins on TTFT
    and prefill. Accept length is a proxy, not the metric — the small separate
    drafter steps faster than the target's own MTP layer applied
    autoregressively, and the faster step more than makes up the lower accept
    length. FP8 + DFlash2 (100.0 t/s) also beats the original MXFP4 + DFlash2
    (92.4 t/s) — the higher-precision FP8 target verifies more of the drafter's
    proposals.
- **KV pool**: auto-sized to 384,238 tokens (1.47× full-window) — the
  DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540 (1.76×); both
  smaller than MXFP4's 943,581 (3.59×) because FP8 weights are bigger (30.9
  vs 19.4 GiB).