dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: seam recalibrated to measured MTP charge; docs to MTP/393216/34:31

Browse Source 
The 393216/34:31/MTP deploy measured 30.78 GiB on ROCm1 against the
fit's 28.32 — the MTP draft buffers (one nextn layer's KV, 4096 B/pos,
plus draft buffers) came in 2.46 GiB above the estimate. Recalibrate
the seam's MTP constant to the measured 4.20 GiB total charge on card
1 and drop the floor 1.5 -> 1.0 GiB for this shape (ROCm1's 1.08 GiB
headroom is real; 35:30/36:29 fail card 0's floor at this pool, so
34:31 is the only split that fits 2x196k).

AGENTS.md and the runbook move to the MTP drafter: 34:31, 393216 pool
(196608 per conversation), the DFlash2 sidecar documented as a
one-line revert, and a dated part-2 section with the fit/measured
table and the recalibration.
This commit is contained in:
Frank
2026-09-17 03:35:46 -07:00
parent 4d13333da7
commit 74dda30fc7
3 changed files with 98 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+71 -25
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, a 327680-position KV pool (163840 per conversation over two slots), vision via the BF16 mmproj.
title: frank-medium llama.cpp Runbook (MTP drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
tags:
- frank-medium
- dankserver
@@ -111,16 +111,17 @@ a truncated 31 GB GGUF is otherwise a load-time crash).
The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
the same size, on Unsloth's importance matrix) of an abliterated Swift
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.
vision tower included. Vision has been ON since 2026-09-17
(`--mmproj`); this is one of the fleet's two image endpoints.
**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled because the DFlash2 path is the measured one (below)
and because a chain cannot hold two context-backed drafters at all.
**The file also carries the model's MTP head** (`blk.64.nextn.*`), and since
2026-09-17 that is the deployed drafter: `--spec-type draft-mtp` builds the
draft context against the main model's own weights (`shares_model`), so there
is no second copy of the model and no `--model-draft` argument — the cost is
one nextn layer's KV (4096 B/position) plus small buffers on the card that
holds the last layers (ROCm1). The DFlash2 sidecar stays on disk and in the
fetch script: `SPEC_TYPE=draft-dflash` (with the sidecar back on the command
line) brings the measured-faster chain back, at the price of ~1.1 GiB on ROCm1.
## Deployment
@@ -144,13 +145,13 @@ Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. Since 2026-09-17 it is one layer PAST the balance point at the 327680 pool, on purpose: the move bought headroom on the drafter card (see below) |
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `327680` / `2` | the pool over two slots of 163840; `--parallel` **divides** the context. 1.25x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
| `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
| `--spec-type` / `--spec-draft-n-max` | `draft-mtp` / `3` | the model's own head, `shares_model` — no sidecar file, no second model copy; n_max 3 is the measured MTP winner (32.4 tok/s vs 26.7 at n_max 7). The DFlash2 sidecar stays on disk: `SPEC_TYPE=draft-dflash` brings the 43.5 tok/s chain back at ~1.1 GiB on ROCm1 |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
@@ -305,13 +306,56 @@ The e2e crawl spec flaked once on the first 327680 deploy (1 failed / 31
passed) and passed 32/32 on the redeploy; the service itself was healthy
throughout (health gate green, `NRestarts=0`).
## 2026-09-17 (part 2): the drafter switches to MTP, the pool grows to 393216
The operator asked for **2x196k contexts**, which the DFlash2 sidecar's
~1.1 GiB on ROCm1 was eating. The drafter switched to the model's own
**MTP head** (`SPEC_TYPE=draft-mtp`, `n_max 3` — the measured MTP winner from
the same-day A/B: 32.4 tok/s at n_max 3 vs 26.7 at n_max 7, against DFlash2's
43.5), the pool grew 327680 → **393216** (196608 per conversation), and the
split moved one layer back to **34:31**, the balance point at the new pool.
| Card | Fit | Measured at load | Headroom |
|---|---:|---:|---:|
| ROCm0 (34 layers + mmproj) | 29.77 GiB | 28.91 GiB | 2.95 GiB |
| ROCm1 (31 layers + MTP buffers) | 28.32 GiB | 30.78 GiB | 1.08 GiB |
The fit predicted 28.32 on ROCm1; the load came in **2.46 GiB hotter** than
predicted. The MTP charge is one nextn layer's KV (4096 B/position — the
draft context is sized from the target, same law as the DFlash2 one) plus
draft buffers the fit only estimated. The seam was recalibrated to the
measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB
total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this
shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's
floor at this pool, so 34:31 is the only split that fits 2x196k.
The rope caveat deepens: the top 131k of a conversation (196608 vs the
trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions
of a FULL conversation) extrapolates on the base rope. 1.5x is a bigger ask
than 1.25x; the seam's ctx cap encodes 393216 with that note.
Verified after deploy: `adding speculative implementation 'draft-mtp'` with
`shares_model` in the load log, no `--spec-draft-model` flag, `--mmproj`
present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len
~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`.
## Verified serving behaviour (2026-09-16)
- Load: `model loaded` ~25 s after start with a warm page cache, then
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports
`modalities: {vision: false}` and the sampler defaults above.
- DFlash2 engaged — the init lines are the proof, not the flags:
- Slot shape: `n_slots = 2, n_ctx_slot = 196608`; `/props` reports
`modalities: {vision: true}` and the sampler defaults above.
- The drafter is engaged — the init lines are the proof, not the flags.
Current (MTP, since 2026-09-17 part 2):
```
common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00
... (draft context shares the main model's weights: `shares_model`)
```
Under the DFlash2 era (2026-09-16 through 2026-09-17 part 1) the block was:
```
common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf'
@@ -324,14 +368,16 @@ throughout (health gate green, `NRestarts=0`).
selector is what distinguishes it from DFlash 1. DSpark logs under the same
`draft_dflash` symbol with `block_size=7` and a Markov head, so read the
numbers, not the implementation name.
- Both cards hold the model and they are **balanced**: at 262144 with the
deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
measurement of the same shape to 0.01 GiB. That is the reading straight after
load; once the service has served traffic it settles ~0.2 GiB lower
(**6.23 / 6.64**, the same with two concurrent requests in flight and after
them). Both figures are ~5 GiB above the operator's 1.5 GiB floor.
- Drafting is live on **both slots**, which is what a 2-slot deployment has to
- Both cards hold the model and they are **balanced at the deployed shape**:
at 393216 with the deployed batch and the 34:31 split, `rocm-smi` reports
**28.91 GiB used on ROCm0 and 30.78 GiB on ROCm1** of 31.86 GiB each — the
fit predicts 29.77 / 28.32, so ROCm1 ran 2.46 GiB hotter than predicted and
the seam's MTP constant was recalibrated to the measured value (see the
2026-09-17 part 2 section). ROCm1's 1.08 GiB of headroom clears the 1.0 GiB
floor that this shape requires.
- Drafting is live on **both slots** (the 2026-09-16 DFlash2-era numbers below
are the record; MTP-era acceptance is in the part 2 section), which is what
a 2-slot deployment has to
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
`0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair.