From 0a25351caa53bfe7b2a22606526287f49d415d2a Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 02:08:14 -0700 Subject: [PATCH] docs: the 2026-09-17 frank-medium shape -- 36:29, 327680 pool, vision AGENTS.md topology row and section, and the runbook: the pool is 327680 positions (163840 per conversation over two slots), the split is 36:29 -- one layer past the 34:31 balance at this context, on purpose, for headroom on the drafter card (fit 29.19/26.04, measured 28.40/26.66 GiB at load, 3.46/5.20 headroom) -- and vision is on via the explicit --mmproj flag, because this build has no autoload (the first deploy came up text-only and 500'd image requests). The rope caveat is written down where the cap lives: the GGUF has no scaling keys, so the top 65k of the pool extrapolates on the base rope. --- AGENTS.md | 9 +-- .../LLM/frank-medium-llamacpp-dflash2.md | 59 ++++++++++++++++--- 2 files changed, 57 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index cb4c4dc..ead7ed7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (35:30), 262144 ctx over 2 slots, batch 6144x512, text only | 7770 | +| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 327680 ctx over 2 slots of 163840, batch 6144x512, vision (BF16 mmproj) | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 | @@ -321,9 +321,10 @@ systemd unit `scripts/frank-medium.service`) serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** (29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the draftless **ngram-mod** matcher, in **one process across both R9700s**, -`--split-mode layer` at a **35:30 tensor split** (the balance point for the -deployed batch: the drafter rides ROCm1, so it takes fewer target layers), -262144 ctx over **2 slots** of 131072, batch 6144x512, text only. DSpark was +`--split-mode layer` at a **36:29 tensor split** (one layer past the 34:31 +balance point at this context, moved to buy headroom on the drafter card: +ROCm1 carries the drafter, its KV and the output layers), 327680 ctx over +**2 slots** of 163840, batch 6144x512, vision via the BF16 mmproj. DSpark was put up in the drafter's place the same day and reverted: it declares no sliding window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four layers of the split. diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index fa9ec7a..d5c24da 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-medium llama.cpp Runbook (DFlash2 drafter) -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split, 262144 ctx over two slots. +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, a 327680-position KV pool (163840 per conversation over two slots), vision via the BF16 mmproj. tags: - frank-medium - dankserver @@ -12,7 +12,7 @@ tags: - gfx1201 - r9700 - layer-split -timestamp: 2026-09-16 +timestamp: 2026-09-17 --- # frank-medium — llama.cpp (DFlash2 drafter) @@ -144,9 +144,9 @@ Deployed flags: | Flag | Value | Why | |---|---|---| -| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one | +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. Since 2026-09-17 it is one layer PAST the balance point at the 327680 pool, on purpose: the move bought headroom on the drafter card (see below) | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | -| `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context | +| `--ctx-size` / `--parallel` | `327680` / `2` | the pool over two slots of 163840; `--parallel` **divides** the context. 1.25x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section | | `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] | | `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB | | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | @@ -154,7 +154,7 @@ Deployed flags: | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers | | `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint | | `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited | -| `--no-mmproj` | — | text only | +| `--mmproj` | `mmproj-BF16.gguf` (888 MiB, beside the target) | vision enabled since 2026-09-17; this build has NO autoload, so the flag is explicit and the serve script refuses to start without the file | ## The DSpark excursion (2026-09-16, tried and reverted) @@ -260,6 +260,51 @@ larger, so the question is closed rather than pending. > 39:26 itself was only *loaded* at the deployed batch, not exercised), and it > is **not** an OOM: no amount of headroom would prevent it. +## 2026-09-17: the pool grows to 327680, the split moves to 36:29, vision comes on + +The window grew from 262144 to a **327680-position pool** (163840 per +conversation over the two slots), the split moved one layer to ROCm0 +(35:30 → 36:29), and the **BF16 mmproj** (888 MiB) came on. + +**Why 36:29, not the balance.** At this context the fit model puts the balance +point at 34:31 (0.11 GiB gap). The operator moved a layer to ROCm0 to buy +headroom on ROCm1 — the card that carries the drafter, its KV, the output +layers and the heavier compute buffers — accepting a 3.15 GiB gap by +construction: + +| Card | Fit | Measured at load | Headroom | +|---|---:|---:|---:| +| ROCm0 (36 layers + mmproj) | 29.19 GiB | 28.40 GiB | 3.46 GiB | +| ROCm1 (29 layers + drafter) | 26.04 GiB | 26.66 GiB | 5.20 GiB | + +Both clear the 1.5 GiB floor with room; the fit lands within 0.8 GiB on both +cards, which is the accuracy it has always had (at the 35:30 point it +reproduced ROCm1 to 0.00 and ROCm0 to 0.86). The seam's slack is the 3.15 GiB +gap rounded up, with the trade written in its comment. + +**The rope caveat.** The GGUF carries no `context_length` and no rope-scaling +keys (freq_base 1e7, `dimension_sections [11,11,10,0]`); the checkpoint's +trained window is 262144. Positions beyond that — the top 65k of the pool — +extrapolate on the base rope. It runs, and 1.25x is a modest ask of a rope, +but it is untrained territory, and the seam's ctx cap encodes 327680 with +that note. + +**Vision: this build has no mmproj autoload.** The first 327680/36:29 deploy +came up healthy and *text-only*: image requests answered 500 "image input is +not supported — hint: … provide the mmproj" and the load log had no mmproj +line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the +model directory. The serve script now passes `--mmproj` explicitly (the +frank-small pattern), refuses to start without the file when vision is on, and +the deploy handler's preflight checks for it. Verified end-to-end after the +fix: a generated 64x32 test image was described from its pixels — with +thinking on the model reasoned about "the three horizontal bars" (its reading +of the crude glyph) and with thinking off answered the letter question in one +token. + +The e2e crawl spec flaked once on the first 327680 deploy (1 failed / 31 +passed) and passed 32/32 on the redeploy; the service itself was healthy +throughout (health gate green, `NRestarts=0`). + ## Verified serving behaviour (2026-09-16) - Load: `model loaded` ~25 s after start with a warm page cache, then @@ -458,13 +503,13 @@ produced wrong numbers on this fleet: - **It separates devices differently from `llama-server`.** `/` splits devices *within one configuration*, `,` splits *sweep values* — so this service's - `--tensor-split 35,30` is `-ts 35/30` to llama-bench, and the comma form + `--tensor-split 36,29` is `-ts 36/29` to llama-bench, and the comma form silently benches each card on its own instead. Likewise `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`. - **There is no `-c`.** The context is a consequence of the test shape: `n_ctx = n_prompt + n_gen + n_depth`. No bench can therefore prove that a configuration's activation buffers fit beside the *deployed* KV — that needs - the live unit at 262144 across two slots. + the live unit at 327680 across two slots. - **A test that cannot allocate calls `exit(1)`**, taking the whole process with it, so the harness runs one invocation per cell and a failure costs one TSV line instead of the rest of the sweep.