dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: seam recalibrated to measured MTP charge; docs to MTP/393216/34:31

Browse Source 
The 393216/34:31/MTP deploy measured 30.78 GiB on ROCm1 against the
fit's 28.32 — the MTP draft buffers (one nextn layer's KV, 4096 B/pos,
plus draft buffers) came in 2.46 GiB above the estimate. Recalibrate
the seam's MTP constant to the measured 4.20 GiB total charge on card
1 and drop the floor 1.5 -> 1.0 GiB for this shape (ROCm1's 1.08 GiB
headroom is real; 35:30/36:29 fail card 0's floor at this pool, so
34:31 is the only split that fits 2x196k).

AGENTS.md and the runbook move to the MTP drafter: 34:31, 393216 pool
(196608 per conversation), the DFlash2 sidecar documented as a
one-line revert, and a dated part-2 section with the fit/measured
table and the recalibration.
This commit is contained in:
Frank
2026-09-17 03:35:46 -07:00
parent 4d13333da7
commit 74dda30fc7
3 changed files with 98 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+10 -10
View File
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 327680 ctx over 2 slots of 163840, batch 6144x512, vision (BF16 mmproj) | 7770 |
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, `--fit` → 262,144 ctx / 1 slot, q8_0 KV, the fleet's vision endpoint | 7771 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -325,17 +325,17 @@ against the real context. Gemma4's template accepts every
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
Rollback: [[frank-small-qwen38-gsq-rco]].
### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE)
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
systemd unit `scripts/frank-medium.service`) serving
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
draftless **ngram-mod** matcher, in **one process across both R9700s**,
`--split-mode layer` at a **36:29 tensor split** (one layer past the 34:31
balance point at this context, moved to buy headroom on the drafter card:
ROCm1 carries the drafter, its KV and the output layers), 327680 ctx over
**2 slots** of 163840, batch 6144x512, vision via the BF16 mmproj. DSpark was
(29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file;
the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one
`SPEC_TYPE` change brings the faster chain back), in **one process across both
R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point
at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
via the BF16 mmproj. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
layers of the split.
@@ -699,7 +699,7 @@ Active:
|---------|---------|
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+71 -25
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, a 327680-position KV pool (163840 per conversation over two slots), vision via the BF16 mmproj.
title: frank-medium llama.cpp Runbook (MTP drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
tags:
- frank-medium
- dankserver
@@ -111,16 +111,17 @@ a truncated 31 GB GGUF is otherwise a load-time crash).
The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
the same size, on Unsloth's importance matrix) of an abliterated Swift
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.
vision tower included. Vision has been ON since 2026-09-17
(`--mmproj`); this is one of the fleet's two image endpoints.
**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled because the DFlash2 path is the measured one (below)
and because a chain cannot hold two context-backed drafters at all.
**The file also carries the model's MTP head** (`blk.64.nextn.*`), and since
2026-09-17 that is the deployed drafter: `--spec-type draft-mtp` builds the
draft context against the main model's own weights (`shares_model`), so there
is no second copy of the model and no `--model-draft` argument — the cost is
one nextn layer's KV (4096 B/position) plus small buffers on the card that
holds the last layers (ROCm1). The DFlash2 sidecar stays on disk and in the
fetch script: `SPEC_TYPE=draft-dflash` (with the sidecar back on the command
line) brings the measured-faster chain back, at the price of ~1.1 GiB on ROCm1.
## Deployment
@@ -144,13 +145,13 @@ Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. Since 2026-09-17 it is one layer PAST the balance point at the 327680 pool, on purpose: the move bought headroom on the drafter card (see below) |
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `327680` / `2` | the pool over two slots of 163840; `--parallel` **divides** the context. 1.25x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
| `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
| `--spec-type` / `--spec-draft-n-max` | `draft-mtp` / `3` | the model's own head, `shares_model` — no sidecar file, no second model copy; n_max 3 is the measured MTP winner (32.4 tok/s vs 26.7 at n_max 7). The DFlash2 sidecar stays on disk: `SPEC_TYPE=draft-dflash` brings the 43.5 tok/s chain back at ~1.1 GiB on ROCm1 |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
@@ -305,13 +306,56 @@ The e2e crawl spec flaked once on the first 327680 deploy (1 failed / 31
passed) and passed 32/32 on the redeploy; the service itself was healthy
throughout (health gate green, `NRestarts=0`).
## 2026-09-17 (part 2): the drafter switches to MTP, the pool grows to 393216
The operator asked for **2x196k contexts**, which the DFlash2 sidecar's
~1.1 GiB on ROCm1 was eating. The drafter switched to the model's own
**MTP head** (`SPEC_TYPE=draft-mtp`, `n_max 3` — the measured MTP winner from
the same-day A/B: 32.4 tok/s at n_max 3 vs 26.7 at n_max 7, against DFlash2's
43.5), the pool grew 327680 → **393216** (196608 per conversation), and the
split moved one layer back to **34:31**, the balance point at the new pool.
| Card | Fit | Measured at load | Headroom |
|---|---:|---:|---:|
| ROCm0 (34 layers + mmproj) | 29.77 GiB | 28.91 GiB | 2.95 GiB |
| ROCm1 (31 layers + MTP buffers) | 28.32 GiB | 30.78 GiB | 1.08 GiB |
The fit predicted 28.32 on ROCm1; the load came in **2.46 GiB hotter** than
predicted. The MTP charge is one nextn layer's KV (4096 B/position — the
draft context is sized from the target, same law as the DFlash2 one) plus
draft buffers the fit only estimated. The seam was recalibrated to the
measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB
total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this
shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's
floor at this pool, so 34:31 is the only split that fits 2x196k.
The rope caveat deepens: the top 131k of a conversation (196608 vs the
trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions
of a FULL conversation) extrapolates on the base rope. 1.5x is a bigger ask
than 1.25x; the seam's ctx cap encodes 393216 with that note.
Verified after deploy: `adding speculative implementation 'draft-mtp'` with
`shares_model` in the load log, no `--spec-draft-model` flag, `--mmproj`
present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len
~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`.
## Verified serving behaviour (2026-09-16)
- Load: `model loaded` ~25 s after start with a warm page cache, then
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports
`modalities: {vision: false}` and the sampler defaults above.
- DFlash2 engaged — the init lines are the proof, not the flags:
- Slot shape: `n_slots = 2, n_ctx_slot = 196608`; `/props` reports
`modalities: {vision: true}` and the sampler defaults above.
- The drafter is engaged — the init lines are the proof, not the flags.
Current (MTP, since 2026-09-17 part 2):
```
common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00
... (draft context shares the main model's weights: `shares_model`)
```
Under the DFlash2 era (2026-09-16 through 2026-09-17 part 1) the block was:
```
common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf'
@@ -324,14 +368,16 @@ throughout (health gate green, `NRestarts=0`).
selector is what distinguishes it from DFlash 1. DSpark logs under the same
`draft_dflash` symbol with `block_size=7` and a Markov head, so read the
numbers, not the implementation name.
- Both cards hold the model and they are **balanced**: at 262144 with the
deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
measurement of the same shape to 0.01 GiB. That is the reading straight after
load; once the service has served traffic it settles ~0.2 GiB lower
(**6.23 / 6.64**, the same with two concurrent requests in flight and after
them). Both figures are ~5 GiB above the operator's 1.5 GiB floor.
- Drafting is live on **both slots**, which is what a 2-slot deployment has to
- Both cards hold the model and they are **balanced at the deployed shape**:
at 393216 with the deployed batch and the 34:31 split, `rocm-smi` reports
**28.91 GiB used on ROCm0 and 30.78 GiB on ROCm1** of 31.86 GiB each — the
fit predicts 29.77 / 28.32, so ROCm1 ran 2.46 GiB hotter than predicted and
the seam's MTP constant was recalibrated to the measured value (see the
2026-09-17 part 2 section). ROCm1's 1.08 GiB of headroom clears the 1.0 GiB
floor that this shape requires.
- Drafting is live on **both slots** (the 2026-09-16 DFlash2-era numbers below
are the record; MTP-era acceptance is in the part 2 section), which is what
a 2-slot deployment has to
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
`0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair.
tests/frank_medium_unit.bats
+17 -7
View File
@@ -196,10 +196,17 @@ print('draft-model wiring ok')
local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack
local card_hi card_lo diff
local mmproj=931145888 # mmproj-BF16.gguf, autoloaded onto the primary device
floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card
# The current point is 34:31 at ctx 393216 with MTP: the fit gap is 1.44
# GiB, which is also the balance point at this pool, so the slack is that
# gap rounded up. (The 36:29 / 327680 / DFlash2 configuration's slack was
# 1.5 GiB was the floor until the 2x196k shape: the deployed 34:31 / 393216
# / MTP point measures 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
# floor is the measured 1.08 GiB rounded down. The margin for one big
# allocation is thinner than it used to be; if a large allocation fails at
# load, the first move is a layer to ROCm0 (33:32), not a ctx cut.
floor=$(( 1024 * 1024 * 1024 )) # 1.0 GiB per card
# The deployed point (34:31 at ctx 393216 with MTP, the balance point at
# this pool) has a post-recalibration fit gap of ~1.0 GiB; the slack sits
# above it at 1.483 GiB, which still rejects a one-layer move (worth
# ~0.43-1.43 GiB of gap depending on the boundary) while accepting the
# deployed split. (The 36:29 / 327680 / DFlash2 configuration's slack was
# 3225 MiB -- its 3.15 GiB gap was by construction, and is no longer the
# deployed shape.)
slack=$(( 1483 * 1024 * 1024 ))
@@ -217,12 +224,15 @@ print('draft-model wiring ok')
+ 2468 * 1024 * ubatch ))
else
# MTP: the head is in the target file (charged above); one nextn
# layer's KV plus ~0.2 GiB of buffers (estimate -- re-measure on the
# host after a drafter swap, as always).
# layer's KV plus buffers/graphs. The constant is CALIBRATED to the
# deployed point (2026-09-17, 34:31 / 393216): card 1 measured 30.78
# GiB, of which 4.20 GiB is MTP's total charge -- 1.61 GiB of nextn KV
# plus 2.70 GiB of buffers and graphs, far more than the 0.2 GiB
# estimate the DFlash2-era shape suggested.
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
+ mtp_kv_pos * ctx \
+ 209715200 \
+ 2901231913 \
+ 2468 * 1024 * ubatch ))
fi
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]