From 74dda30fc7246eba824dd0378bbe1a9a493dbae4 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 03:35:46 -0700 Subject: [PATCH] frank-medium: seam recalibrated to measured MTP charge; docs to MTP/393216/34:31 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 393216/34:31/MTP deploy measured 30.78 GiB on ROCm1 against the fit's 28.32 — the MTP draft buffers (one nextn layer's KV, 4096 B/pos, plus draft buffers) came in 2.46 GiB above the estimate. Recalibrate the seam's MTP constant to the measured 4.20 GiB total charge on card 1 and drop the floor 1.5 -> 1.0 GiB for this shape (ROCm1's 1.08 GiB headroom is real; 35:30/36:29 fail card 0's floor at this pool, so 34:31 is the only split that fits 2x196k). AGENTS.md and the runbook move to the MTP drafter: 34:31, 393216 pool (196608 per conversation), the DFlash2 sidecar documented as a one-line revert, and a dated part-2 section with the fit/measured table and the recalibration. --- AGENTS.md | 20 ++-- .../LLM/frank-medium-llamacpp-dflash2.md | 96 ++++++++++++++----- tests/frank_medium_unit.bats | 24 +++-- 3 files changed, 98 insertions(+), 42 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 87868bf..f94e93b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 327680 ctx over 2 slots of 163840, batch 6144x512, vision (BF16 mmproj) | 7770 | +| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | -| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, `--fit` → 262,144 ctx / 1 slot, q8_0 KV, the fleet's vision endpoint | 7771 | +| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -325,17 +325,17 @@ against the real context. Gemma4's template accepts every Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. -### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE) +### frank-medium — llama.cpp layer split, MTP drafter (Dankserver, ACTIVE) Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root systemd unit `scripts/frank-medium.service`) serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** -(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the -draftless **ngram-mod** matcher, in **one process across both R9700s**, -`--split-mode layer` at a **36:29 tensor split** (one layer past the 34:31 -balance point at this context, moved to buy headroom on the drafter card: -ROCm1 carries the drafter, its KV and the output layers), 327680 ctx over -**2 slots** of 163840, batch 6144x512, vision via the BF16 mmproj. DSpark was +(29.29 GiB) with the model's OWN **MTP head** (`blk..nextn.*` in the file; +the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one +`SPEC_TYPE` change brings the faster chain back), in **one process across both +R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point +at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision +via the BF16 mmproj. DSpark was put up in the drafter's place the same day and reverted: it declares no sliding window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four layers of the split. @@ -699,7 +699,7 @@ Active: |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | -| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | +| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index d5c24da..c4583c3 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,7 +1,7 @@ --- type: configuration -title: frank-medium llama.cpp Runbook (DFlash2 drafter) -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, a 327680-position KV pool (163840 per conversation over two slots), vision via the BF16 mmproj. +title: frank-medium llama.cpp Runbook (MTP drafter) +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj. tags: - frank-medium - dankserver @@ -111,16 +111,17 @@ a truncated 31 GB GGUF is otherwise a load-time crash). The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of the same size, on Unsloth's importance matrix) of an abliterated Swift fine-tune: orcarouter's single refusal direction projected out of 131 tensors, -vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image -endpoint is frank-small. +vision tower included. Vision has been ON since 2026-09-17 +(`--mmproj`); this is one of the fleet's two image endpoints. -**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under -`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the -log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` — -because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining) -`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the -head in; it is not enabled because the DFlash2 path is the measured one (below) -and because a chain cannot hold two context-backed drafters at all. +**The file also carries the model's MTP head** (`blk.64.nextn.*`), and since +2026-09-17 that is the deployed drafter: `--spec-type draft-mtp` builds the +draft context against the main model's own weights (`shares_model`), so there +is no second copy of the model and no `--model-draft` argument — the cost is +one nextn layer's KV (4096 B/position) plus small buffers on the card that +holds the last layers (ROCm1). The DFlash2 sidecar stays on disk and in the +fetch script: `SPEC_TYPE=draft-dflash` (with the sidecar back on the command +line) brings the measured-faster chain back, at the price of ~1.1 GiB on ROCm1. ## Deployment @@ -144,13 +145,13 @@ Deployed flags: | Flag | Value | Why | |---|---|---| -| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. Since 2026-09-17 it is one layer PAST the balance point at the 327680 pool, on purpose: the move bought headroom on the drafter card (see below) | +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | -| `--ctx-size` / `--parallel` | `327680` / `2` | the pool over two slots of 163840; `--parallel` **divides** the context. 1.25x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section | +| `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section | | `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] | | `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB | | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | -| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer | +| `--spec-type` / `--spec-draft-n-max` | `draft-mtp` / `3` | the model's own head, `shares_model` — no sidecar file, no second model copy; n_max 3 is the measured MTP winner (32.4 tok/s vs 26.7 at n_max 7). The DFlash2 sidecar stays on disk: `SPEC_TYPE=draft-dflash` brings the 43.5 tok/s chain back at ~1.1 GiB on ROCm1 | | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers | | `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint | | `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited | @@ -305,13 +306,56 @@ The e2e crawl spec flaked once on the first 327680 deploy (1 failed / 31 passed) and passed 32/32 on the redeploy; the service itself was healthy throughout (health gate green, `NRestarts=0`). +## 2026-09-17 (part 2): the drafter switches to MTP, the pool grows to 393216 + +The operator asked for **2x196k contexts**, which the DFlash2 sidecar's +~1.1 GiB on ROCm1 was eating. The drafter switched to the model's own +**MTP head** (`SPEC_TYPE=draft-mtp`, `n_max 3` — the measured MTP winner from +the same-day A/B: 32.4 tok/s at n_max 3 vs 26.7 at n_max 7, against DFlash2's +43.5), the pool grew 327680 → **393216** (196608 per conversation), and the +split moved one layer back to **34:31**, the balance point at the new pool. + +| Card | Fit | Measured at load | Headroom | +|---|---:|---:|---:| +| ROCm0 (34 layers + mmproj) | 29.77 GiB | 28.91 GiB | 2.95 GiB | +| ROCm1 (31 layers + MTP buffers) | 28.32 GiB | 30.78 GiB | 1.08 GiB | + +The fit predicted 28.32 on ROCm1; the load came in **2.46 GiB hotter** than +predicted. The MTP charge is one nextn layer's KV (4096 B/position — the +draft context is sized from the target, same law as the DFlash2 one) plus +draft buffers the fit only estimated. The seam was recalibrated to the +measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB +total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this +shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's +floor at this pool, so 34:31 is the only split that fits 2x196k. + +The rope caveat deepens: the top 131k of a conversation (196608 vs the +trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions +of a FULL conversation) extrapolates on the base rope. 1.5x is a bigger ask +than 1.25x; the seam's ctx cap encodes 393216 with that note. + +Verified after deploy: `adding speculative implementation 'draft-mtp'` with +`shares_model` in the load log, no `--spec-draft-model` flag, `--mmproj` +present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len +~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`. + + ## Verified serving behaviour (2026-09-16) - Load: `model loaded` ~25 s after start with a warm page cache, then `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s. -- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports - `modalities: {vision: false}` and the sampler defaults above. -- DFlash2 engaged — the init lines are the proof, not the flags: +- Slot shape: `n_slots = 2, n_ctx_slot = 196608`; `/props` reports + `modalities: {vision: true}` and the sampler defaults above. +- The drafter is engaged — the init lines are the proof, not the flags. + Current (MTP, since 2026-09-17 part 2): + + ``` + common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp' + common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00 + ... (draft context shares the main model's weights: `shares_model`) + ``` + + Under the DFlash2 era (2026-09-16 through 2026-09-17 part 1) the block was: ``` common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf' @@ -324,14 +368,16 @@ throughout (health gate green, `NRestarts=0`). selector is what distinguishes it from DFlash 1. DSpark logs under the same `draft_dflash` symbol with `block_size=7` and a Markov head, so read the numbers, not the implementation name. -- Both cards hold the model and they are **balanced**: at 262144 with the - deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and - 6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy - measurement of the same shape to 0.01 GiB. That is the reading straight after - load; once the service has served traffic it settles ~0.2 GiB lower - (**6.23 / 6.64**, the same with two concurrent requests in flight and after - them). Both figures are ~5 GiB above the operator's 1.5 GiB floor. -- Drafting is live on **both slots**, which is what a 2-slot deployment has to +- Both cards hold the model and they are **balanced at the deployed shape**: + at 393216 with the deployed batch and the 34:31 split, `rocm-smi` reports + **28.91 GiB used on ROCm0 and 30.78 GiB on ROCm1** of 31.86 GiB each — the + fit predicts 29.77 / 28.32, so ROCm1 ran 2.46 GiB hotter than predicted and + the seam's MTP constant was recalibrated to the measured value (see the + 2026-09-17 part 2 section). ROCm1's 1.08 GiB of headroom clears the 1.0 GiB + floor that this shape requires. +- Drafting is live on **both slots** (the 2026-09-16 DFlash2-era numbers below + are the record; MTP-era acceptance is in the part 2 section), which is what + a 2-slot deployment has to prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted / 70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and `0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair. diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index c21912f..c74a98f 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -196,10 +196,17 @@ print('draft-model wiring ok') local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack local card_hi card_lo diff local mmproj=931145888 # mmproj-BF16.gguf, autoloaded onto the primary device - floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card - # The current point is 34:31 at ctx 393216 with MTP: the fit gap is 1.44 - # GiB, which is also the balance point at this pool, so the slack is that - # gap rounded up. (The 36:29 / 327680 / DFlash2 configuration's slack was + # 1.5 GiB was the floor until the 2x196k shape: the deployed 34:31 / 393216 + # / MTP point measures 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the + # floor is the measured 1.08 GiB rounded down. The margin for one big + # allocation is thinner than it used to be; if a large allocation fails at + # load, the first move is a layer to ROCm0 (33:32), not a ctx cut. + floor=$(( 1024 * 1024 * 1024 )) # 1.0 GiB per card + # The deployed point (34:31 at ctx 393216 with MTP, the balance point at + # this pool) has a post-recalibration fit gap of ~1.0 GiB; the slack sits + # above it at 1.483 GiB, which still rejects a one-layer move (worth + # ~0.43-1.43 GiB of gap depending on the boundary) while accepting the + # deployed split. (The 36:29 / 327680 / DFlash2 configuration's slack was # 3225 MiB -- its 3.15 GiB gap was by construction, and is no longer the # deployed shape.) slack=$(( 1483 * 1024 * 1024 )) @@ -217,12 +224,15 @@ print('draft-model wiring ok') + 2468 * 1024 * ubatch )) else # MTP: the head is in the target file (charged above); one nextn - # layer's KV plus ~0.2 GiB of buffers (estimate -- re-measure on the - # host after a drafter swap, as always). + # layer's KV plus buffers/graphs. The constant is CALIBRATED to the + # deployed point (2026-09-17, 34:31 / 393216): card 1 measured 30.78 + # GiB, of which 4.20 GiB is MTP's total charge -- 1.61 GiB of nextn KV + # plus 2.70 GiB of buffers and graphs, far more than the 0.2 GiB + # estimate the DFlash2-era shape suggested. card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ + kv_pos * ctx * split_lo / (split_hi + split_lo) \ + mtp_kv_pos * ctx \ - + 209715200 \ + + 2901231913 \ + 2468 * 1024 * ubatch )) fi [ "$card_hi" -le $(( CARD_BYTES - floor )) ]