frank-medium: seam recalibrated to measured MTP charge; docs to MTP/393216/34:31

The 393216/34:31/MTP deploy measured 30.78 GiB on ROCm1 against the
fit's 28.32 — the MTP draft buffers (one nextn layer's KV, 4096 B/pos,
plus draft buffers) came in 2.46 GiB above the estimate. Recalibrate
the seam's MTP constant to the measured 4.20 GiB total charge on card
1 and drop the floor 1.5 -> 1.0 GiB for this shape (ROCm1's 1.08 GiB
headroom is real; 35:30/36:29 fail card 0's floor at this pool, so
34:31 is the only split that fits 2x196k).

AGENTS.md and the runbook move to the MTP drafter: 34:31, 393216 pool
(196608 per conversation), the DFlash2 sidecar documented as a
one-line revert, and a dated part-2 section with the fit/measured
table and the recalibration.

AGENTS.md

@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 327680 ctx over 2 slots of 163840, batch 6144x512, vision (BF16 mmproj) | 7770 |
+| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 |
 | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, `--fit` → 262,144 ctx / 1 slot, q8_0 KV, the fleet's vision endpoint | 7771 |
+| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -325,17 +325,17 @@ against the real context. Gemma4's template accepts every
 Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
 Rollback: [[frank-small-qwen38-gsq-rco]].
 
-### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
+### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
 
 Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
 systemd unit `scripts/frank-medium.service`) serving
 `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
-(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
+(29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file;
-draftless **ngram-mod** matcher, in **one process across both R9700s**,
+the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one
-`--split-mode layer` at a **36:29 tensor split** (one layer past the 34:31
+`SPEC_TYPE` change brings the faster chain back), in **one process across both
-balance point at this context, moved to buy headroom on the drafter card:
+R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point
-ROCm1 carries the drafter, its KV and the output layers), 327680 ctx over
+at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
-**2 slots** of 163840, batch 6144x512, vision via the BF16 mmproj.  DSpark was
+via the BF16 mmproj.  DSpark was
 put up in the drafter's place the same day and reverted: it declares no sliding
 window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
 layers of the split.
@@ -699,7 +699,7 @@ Active:
 |---------|---------|
 | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
 | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
+| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
 

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -1,7 +1,7 @@
 ---
 type: configuration
-title: frank-medium llama.cpp Runbook (DFlash2 drafter)
+title: frank-medium llama.cpp Runbook (MTP drafter)
-description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, a 327680-position KV pool (163840 per conversation over two slots), vision via the BF16 mmproj.
+description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
 tags:
   - frank-medium
   - dankserver
@@ -111,16 +111,17 @@ a truncated 31 GB GGUF is otherwise a load-time crash).
 The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
 the same size, on Unsloth's importance matrix) of an abliterated Swift
 fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
-vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
+vision tower included. Vision has been ON since 2026-09-17
-endpoint is frank-small.
+(`--mmproj`); this is one of the fleet's two image endpoints.
 
-**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under
+**The file also carries the model's MTP head** (`blk.64.nextn.*`), and since
-`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
+2026-09-17 that is the deployed drafter: `--spec-type draft-mtp` builds the
-log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
+draft context against the main model's own weights (`shares_model`), so there
-because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining)
+is no second copy of the model and no `--model-draft` argument — the cost is
-`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
+one nextn layer's KV (4096 B/position) plus small buffers on the card that
-head in; it is not enabled because the DFlash2 path is the measured one (below)
+holds the last layers (ROCm1). The DFlash2 sidecar stays on disk and in the
-and because a chain cannot hold two context-backed drafters at all.
+fetch script: `SPEC_TYPE=draft-dflash` (with the sidecar back on the command
+line) brings the measured-faster chain back, at the price of ~1.1 GiB on ROCm1.
 
 ## Deployment
 
@@ -144,13 +145,13 @@ Deployed flags:
 
 | Flag | Value | Why |
 |---|---|---|
-| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. Since 2026-09-17 it is one layer PAST the balance point at the 327680 pool, on purpose: the move bought headroom on the drafter card (see below) |
+| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool |
 | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
-| `--ctx-size` / `--parallel` | `327680` / `2` | the pool over two slots of 163840; `--parallel` **divides** the context. 1.25x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
+| `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
 | `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
 | `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
 | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
-| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
+| `--spec-type` / `--spec-draft-n-max` | `draft-mtp` / `3` | the model's own head, `shares_model` — no sidecar file, no second model copy; n_max 3 is the measured MTP winner (32.4 tok/s vs 26.7 at n_max 7). The DFlash2 sidecar stays on disk: `SPEC_TYPE=draft-dflash` brings the 43.5 tok/s chain back at ~1.1 GiB on ROCm1 |
 | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
 | `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
 | `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
@@ -305,13 +306,56 @@ The e2e crawl spec flaked once on the first 327680 deploy (1 failed / 31
 passed) and passed 32/32 on the redeploy; the service itself was healthy
 throughout (health gate green, `NRestarts=0`).
 
+## 2026-09-17 (part 2): the drafter switches to MTP, the pool grows to 393216
+
+The operator asked for **2x196k contexts**, which the DFlash2 sidecar's
+~1.1 GiB on ROCm1 was eating. The drafter switched to the model's own
+**MTP head** (`SPEC_TYPE=draft-mtp`, `n_max 3` — the measured MTP winner from
+the same-day A/B: 32.4 tok/s at n_max 3 vs 26.7 at n_max 7, against DFlash2's
+43.5), the pool grew 327680 → **393216** (196608 per conversation), and the
+split moved one layer back to **34:31**, the balance point at the new pool.
+
+| Card | Fit | Measured at load | Headroom |
+|---|---:|---:|---:|
+| ROCm0 (34 layers + mmproj) | 29.77 GiB | 28.91 GiB | 2.95 GiB |
+| ROCm1 (31 layers + MTP buffers) | 28.32 GiB | 30.78 GiB | 1.08 GiB |
+
+The fit predicted 28.32 on ROCm1; the load came in **2.46 GiB hotter** than
+predicted. The MTP charge is one nextn layer's KV (4096 B/position — the
+draft context is sized from the target, same law as the DFlash2 one) plus
+draft buffers the fit only estimated. The seam was recalibrated to the
+measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB
+total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this
+shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's
+floor at this pool, so 34:31 is the only split that fits 2x196k.
+
+The rope caveat deepens: the top 131k of a conversation (196608 vs the
+trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions
+of a FULL conversation) extrapolates on the base rope. 1.5x is a bigger ask
+than 1.25x; the seam's ctx cap encodes 393216 with that note.
+
+Verified after deploy: `adding speculative implementation 'draft-mtp'` with
+`shares_model` in the load log, no `--spec-draft-model` flag, `--mmproj`
+present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len
+~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`.
+
+
 ## Verified serving behaviour (2026-09-16)
 
 - Load: `model loaded` ~25 s after start with a warm page cache, then
   `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
-- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports
+- Slot shape: `n_slots = 2, n_ctx_slot = 196608`; `/props` reports
-  `modalities: {vision: false}` and the sampler defaults above.
+  `modalities: {vision: true}` and the sampler defaults above.
-- DFlash2 engaged — the init lines are the proof, not the flags:
+- The drafter is engaged — the init lines are the proof, not the flags.
+  Current (MTP, since 2026-09-17 part 2):
+
+  ```
+  common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
+  common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00
+  ... (draft context shares the main model's weights: `shares_model`)
+  ```
+
+  Under the DFlash2 era (2026-09-16 through 2026-09-17 part 1) the block was:
 
   ```
   common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf'
@@ -324,14 +368,16 @@ throughout (health gate green, `NRestarts=0`).
   selector is what distinguishes it from DFlash 1. DSpark logs under the same
   `draft_dflash` symbol with `block_size=7` and a Markov head, so read the
   numbers, not the implementation name.
-- Both cards hold the model and they are **balanced**: at 262144 with the
+- Both cards hold the model and they are **balanced at the deployed shape**:
-  deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
+  at 393216 with the deployed batch and the 34:31 split, `rocm-smi` reports
-  6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
+  **28.91 GiB used on ROCm0 and 30.78 GiB on ROCm1** of 31.86 GiB each — the
-  measurement of the same shape to 0.01 GiB. That is the reading straight after
+  fit predicts 29.77 / 28.32, so ROCm1 ran 2.46 GiB hotter than predicted and
-  load; once the service has served traffic it settles ~0.2 GiB lower
+  the seam's MTP constant was recalibrated to the measured value (see the
-  (**6.23 / 6.64**, the same with two concurrent requests in flight and after
+  2026-09-17 part 2 section). ROCm1's 1.08 GiB of headroom clears the 1.0 GiB
-  them). Both figures are ~5 GiB above the operator's 1.5 GiB floor.
+  floor that this shape requires.
-- Drafting is live on **both slots**, which is what a 2-slot deployment has to
+- Drafting is live on **both slots** (the 2026-09-16 DFlash2-era numbers below
+  are the record; MTP-era acceptance is in the part 2 section), which is what
+  a 2-slot deployment has to
   prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
   70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
   `0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair.

tests/frank_medium_unit.bats

@@ -196,10 +196,17 @@ print('draft-model wiring ok')
     local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack
     local card_hi card_lo diff
     local mmproj=931145888   # mmproj-BF16.gguf, autoloaded onto the primary device
-    floor=$(( 1536 * 1024 * 1024 ))     # 1.5 GiB per card
+    # 1.5 GiB was the floor until the 2x196k shape: the deployed 34:31 / 393216
-    # The current point is 34:31 at ctx 393216 with MTP: the fit gap is 1.44
+    # / MTP point measures 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
-    # GiB, which is also the balance point at this pool, so the slack is that
+    # floor is the measured 1.08 GiB rounded down.  The margin for one big
-    # gap rounded up.  (The 36:29 / 327680 / DFlash2 configuration's slack was
+    # allocation is thinner than it used to be; if a large allocation fails at
+    # load, the first move is a layer to ROCm0 (33:32), not a ctx cut.
+    floor=$(( 1024 * 1024 * 1024 ))     # 1.0 GiB per card
+    # The deployed point (34:31 at ctx 393216 with MTP, the balance point at
+    # this pool) has a post-recalibration fit gap of ~1.0 GiB; the slack sits
+    # above it at 1.483 GiB, which still rejects a one-layer move (worth
+    # ~0.43-1.43 GiB of gap depending on the boundary) while accepting the
+    # deployed split.  (The 36:29 / 327680 / DFlash2 configuration's slack was
     # 3225 MiB -- its 3.15 GiB gap was by construction, and is no longer the
     # deployed shape.)
     slack=$(( 1483 * 1024 * 1024 ))
@@ -217,12 +224,15 @@ print('draft-model wiring ok')
                   + 2468 * 1024 * ubatch ))
     else
         # MTP: the head is in the target file (charged above); one nextn
-        # layer's KV plus ~0.2 GiB of buffers (estimate -- re-measure on the
+        # layer's KV plus buffers/graphs.  The constant is CALIBRATED to the
-        # host after a drafter swap, as always).
+        # deployed point (2026-09-17, 34:31 / 393216): card 1 measured 30.78
+        # GiB, of which 4.20 GiB is MTP's total charge -- 1.61 GiB of nextn KV
+        # plus 2.70 GiB of buffers and graphs, far more than the 0.2 GiB
+        # estimate the DFlash2-era shape suggested.
         card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
                   + kv_pos * ctx * split_lo / (split_hi + split_lo) \
                   + mtp_kv_pos * ctx \
-                  + 209715200 \
+                  + 2901231913 \
                   + 2468 * 1024 * ubatch ))
     fi
     [ "$card_hi" -le $(( CARD_BYTES - floor )) ]