docs: frank-large Q6 uncensored cutover + frank-small 2x200704 f16-KV reslice
- new active runbook frank-large-rocmfpx-q6agent-uncensored-mtp.md: the Q6_0_ROCMFPX_AGENT file (ftype 114, 339 type-102 + 159 type-103 tensors), 27.6-28.1 t/s at 86.8% draft acceptance, the measured ngram-mod chain rejection (-15% prose: recurrent-state rollback disabled), the bench-window hardening laws - frank-large-rocmfpx-q8agent-mtp.md retired; rollback section corrected (gufo payloads removed from disk 2026-10-03 - that rollback is now a re-download; the aligned Q8 file is the direct one, still on disk) - frank-small-llama-mimo-q5km.md reshaped to 2 x 200704 with fully unquantized f16 KV: boot-pair table gains the measured row (20,530 used / 4,046 free), 104.84 tok/s, slot-window contract at 200704, rollback to the 3x180224 + q8_0-V shape documented - AGENTS.md: fleet table rows, both endpoint sections, the chain law extended with the recurrent-model measurement, the --parallel law with the reslice, runbook index (active + retired) - LLM/index.md + root index entries updated
This commit is contained in:
Frank committed
1 parent
47a2c1287b
commit
b80886e318
6 files changed
+438
-216
No files matched your search
@@ -25,8 +25,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 20,530 used / **4,046 free** of 24,576 MiB, 104.8 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V: the freed slot paid for exact values); rollback `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` or `models/MiMo-Q6K/` | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -291,8 +291,9 @@ of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
|
||||
autoloads projectors), and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
**180224**, the per-slot window, not the model's native 262144: the harness
|
||||
must compact before llama.cpp answers a longer prompt with a 400. Note also
|
||||
**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
|
||||
model's native 262144: the harness must compact before llama.cpp answers a
|
||||
longer prompt with a 400. Note also
|
||||
that
|
||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||
@@ -303,26 +304,26 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(the MiMo tower is vision-only).
|
||||
|
||||
### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
|
||||
### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
|
||||
|
||||
The **kingjones30/ROCmFPX llama.cpp fork** at commit
|
||||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
|
||||
`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
|
||||
as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||||
`Restart=always`) running a bare `llama-server` (no container) on port 7770
|
||||
from the Strix Halo iGPU. Boot to healthy: **16 s**.
|
||||
from the Strix Halo iGPU. Boot to healthy: **~15 s**.
|
||||
|
||||
**The runtime is not interchangeable — it is chosen by the FILE.**
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung,
|
||||
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`,
|
||||
ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103,
|
||||
`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader:
|
||||
stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The
|
||||
file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant
|
||||
keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file),
|
||||
which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality
|
||||
AND the better drafter base. Vision comes from the bundled
|
||||
`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B).
|
||||
`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF`'s protected-heads
|
||||
rung, `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` (24,856,607,008 B,
|
||||
sha256 `e966deb3…`, ftype 114 ≈ 7.44 bpw), stores **339 tensors of type 102,
|
||||
`GGML_TYPE_Q6_0_ROCMFPX`, plus 159 type-103 `Q8_0_ROCMFPX`** — quantizations
|
||||
that exist in no other loader: stock llama.cpp and gufo both refuse the file at
|
||||
load (`unknown tensor type`). The AGENT construction keeps 64 output-head
|
||||
tensors at plain precision (the protected-heads design of the aligned repo's
|
||||
Q8 file, one quant rung down), and the uncensored tuning is what Frank swapped
|
||||
to on 2026-10-03. Vision comes from the BF16 `mmproj-Qwen3.8-27B-BF16.gguf`
|
||||
(931,145,888 B — hardlinked from the aligned payload dir, byte-identical).
|
||||
|
||||
**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
|
||||
`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
|
||||
@@ -334,22 +335,26 @@ the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…
|
||||
`nextn_predict_layers = 1`), pinned to the same card:
|
||||
`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
|
||||
Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens,
|
||||
`cache_prompt: false`, production service stopped for each arm:
|
||||
**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance
|
||||
length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**;
|
||||
prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's
|
||||
0.95 (that number is the card's own text at a shallow prompt); the gain is
|
||||
real and n-max 4 is where it was measured, never swept. This replaces the
|
||||
gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more
|
||||
speed at once.
|
||||
On the ALIGNED Q8 file (2026-10-02, bench windows, 2,202-token prompt, temp 0):
|
||||
**MTP 17.00–17.14 t/s vs 7.60–7.66 with the draft stripped — +123 %**,
|
||||
acceptance 0.462. On the UNCENSORED Q6 file (2026-10-03, production service,
|
||||
prose at 4,813-token depth): **27.57–28.10 t/s decode, draft acceptance
|
||||
396/456 = 0.868** (mean acceptance length ≈ 4.4 — drafting at nearly the full
|
||||
n-max 4), schema-echo traffic 31.44 t/s at 639/639 = 1.000. The drafter is
|
||||
materially better on this file; treat the card's 0.95 as a file property, not a
|
||||
throughput promise. **`ngram-mod` chained onto MTP was measured and REJECTED
|
||||
(2026-10-03):** `--spec-type draft-mtp,ngram-mod` cost −15 % on prose
|
||||
(23.40 t/s) and −7 % on verbatim block-copy even with the matcher demonstrably
|
||||
firing — the boot log says why: `recurrent state rollback is not compatible
|
||||
with 'ngram-mod' - disabling rollback support`, and the hybrid's MTP
|
||||
verification needs that rollback. Keep `draft-mtp` alone.
|
||||
|
||||
**Shape and budget:** ONE session at the native 262,144 window, unquantized
|
||||
f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
|
||||
aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
|
||||
--cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model +
|
||||
draft + projector resident. Prompt cache verified: a repeat pass over the
|
||||
same 6k prompt replays in 0.5 s.
|
||||
--cache-ram 8192 --cache-idle-slots`. Measured GTT ~61 GiB under live load
|
||||
(host baseline 16.7 GiB). Prompt cache verified (Q8 era): a repeat pass over
|
||||
the same 6k prompt replays in 0.5 s.
|
||||
|
||||
**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
|
||||
(without it `chat_template_kwargs` are silently ignored), tools are native
|
||||
@@ -365,14 +370,18 @@ measuring), and the fork logs
|
||||
`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
|
||||
resets — self-recovering, not a fault.
|
||||
|
||||
Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS
|
||||
sha256 verified, `--verify-only`, no token on the command line).
|
||||
Fetch: `scripts/download-frank-large-kingjones-uncensored.sh` (hardlinks the
|
||||
byte-identical MTP head and projector from the aligned payload dir when
|
||||
present, otherwise fetches; `-f` + byte count + LFS sha256 verified,
|
||||
`--verify-only`, no token on the command line).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`.
|
||||
Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1
|
||||
(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B
|
||||
payload and the DFlash2 drafter all remain on the host; restore the tag's
|
||||
unit/env/serve trio and redeploy.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md`.
|
||||
Rollback: the ALIGNED Q8 file (the previous shape) remains on disk at
|
||||
`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
|
||||
2026-10-03 swap commit (unit/env/serve trio) and redeploy. The older gufo tag
|
||||
`frank-large-gufo-q4kxl-20261001` is now a **re-download** rollback: its
|
||||
Q4_K_XL payload and DFlash2 drafter were removed from disk 2026-10-03
|
||||
(`scripts/build-gufo.sh` stays for the rebuild).
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -383,12 +392,14 @@ unit/env/serve trio and redeploy.
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
|
||||
sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
|
||||
port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
|
||||
runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
|
||||
**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
|
||||
Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
|
||||
the projector file is the entire vision path (and ~1.4 GiB of the card).
|
||||
port 7771. The runtime arrived 2026-10-01 night for one reason:
|
||||
**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
|
||||
together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
|
||||
from three slots to two made room for **fully unquantized V**
|
||||
(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came
|
||||
2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector
|
||||
file is the entire vision path (and ~1.4 GiB of the card).
|
||||
|
||||
**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
|
||||
DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
@@ -396,36 +407,34 @@ DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
|
||||
drives that term.
|
||||
|
||||
**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
|
||||
180,224 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
|
||||
only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
|
||||
layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
|
||||
free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
|
||||
solving the pool against two measured boots: non-KV cost is 7,390 MiB without
|
||||
a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
|
||||
vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
|
||||
windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
|
||||
table. Full f16 V would add ~4 GiB more and bust the card — that is the
|
||||
justification for the asymmetric contract.
|
||||
**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
|
||||
conversations of 200,704 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
|
||||
**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, batch
|
||||
2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **20,530
|
||||
used / 4,046 free** of 24,576 MiB — above the 2–3 GiB floor Frank asked for,
|
||||
solved from the boot-pair table in the runbook (non-KV cost ~7,986 MiB at this
|
||||
shape; f16 K+V is 32 KiB/token across the 8 attention layers). The 2026-10-02
|
||||
3 × 180,224 + q8_0-V shape stays one env edit away (`CTX_SIZE=540672
|
||||
PARALLEL=3 CACHE_TYPE_V=q8_0`). Three native windows need 18.4 GiB of KV alone
|
||||
and were never on the table.
|
||||
|
||||
Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
|
||||
`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
|
||||
**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
|
||||
2 × 262,144 shape, because a shorter slot means a shorter attention scan;
|
||||
three concurrent requests answered in parallel and a fourth queued behind a
|
||||
released slot; native `tool_calls`; thinking in `reasoning_content`; a
|
||||
solid-red PNG answers "Red." through an `image_url` content part (the legacy
|
||||
top-level `images` field is ignored by `/v1/chat/completions` — the model then
|
||||
invents a color). A prompt over the slot is a **400
|
||||
`exceed_context_size_error`**, which is why `models.yml` sets
|
||||
`contextWindow: 180224`.
|
||||
Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
|
||||
and `loaded multimodal model`; decode **104.84 tok/s** at temp 0 — ~2 % under
|
||||
the 3-slot q8_0-V shape (106.8), the honest price of exact V; health 200 in
|
||||
~5 s; native `tool_calls`; thinking in `reasoning_content`; a solid-red PNG
|
||||
answers "Red." through an `image_url` content part (the legacy top-level
|
||||
`images` field is ignored by `/v1/chat/completions` — the model then invents a
|
||||
color). A prompt over the slot is a **400 `exceed_context_size_error`**, which
|
||||
is why `models.yml` sets `contextWindow: 200704`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
|
||||
Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
|
||||
(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was
|
||||
staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
|
||||
Rollback: the previous shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
|
||||
(3 × 180,224, measured 2,441 MiB free); before that
|
||||
[[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
`/home/dank/models/MiMo-Q6K/`) and [[frank-small-sglang-mimo-v26]] (venv on
|
||||
disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
|
||||
|
||||
@@ -747,19 +756,26 @@ linked runbooks.
|
||||
the MAIN model's weights (`shares_model`), so there is no second copy of a
|
||||
12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
|
||||
tensors at all, and pointing `--model-draft` at the file would load it twice.
|
||||
- **What can be chained, and what cannot.** `--spec-type a,b` builds every
|
||||
enabled implementation and chains drafting through them, in a fixed internal
|
||||
order (ngram types first, then the drafters — CLI order is irrelevant). The
|
||||
gate is `ctx_dft`: **draftless** types (the ngram family) are added
|
||||
unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and
|
||||
`draft-eagle3` all require a draft CONTEXT. So a chain may hold one
|
||||
context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod`
|
||||
is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-
|
||||
identical to it on prose at the fleet's 48/64 window) — but
|
||||
`draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model
|
||||
doesn't contain MTP layers"*, because MTP builds its context against whatever
|
||||
model the draft context holds. Two context-backed types cannot share one
|
||||
process, and MTP's head only loads when MTP **is** the draft model.
|
||||
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
|
||||
`--spec-type a,b` builds every enabled implementation and chains drafting
|
||||
through them, in a fixed internal order (ngram types first, then the
|
||||
drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
|
||||
types (the ngram family) are added unconditionally, while `draft-mtp`,
|
||||
`draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
|
||||
CONTEXT. So a chain may hold one context-backed drafter plus any number of
|
||||
ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
|
||||
drafter alone on prompt-echo traffic, byte-identical to it on prose at the
|
||||
fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
|
||||
type MTP requested but model doesn't contain MTP layers"*, because MTP
|
||||
builds its context against whatever model the draft context holds. Two
|
||||
context-backed types cannot share one process, and MTP's head only loads
|
||||
when MTP **is** the draft model. **A LEGAL chain can still lose on a
|
||||
recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
|
||||
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
|
||||
logs `recurrent state rollback is not compatible with 'ngram-mod' -
|
||||
disabling rollback support`, and MTP verification depends on that rollback,
|
||||
so acceptance drops on ALL traffic even where the matcher fires. Measure
|
||||
chains per architecture, never by analogy.
|
||||
- **A drafter's KV scales with the TARGET's context, and its window policy
|
||||
decides how much that costs.** The draft context is built with
|
||||
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
|
||||
@@ -820,7 +836,11 @@ linked runbooks.
|
||||
the harness would discover the ceiling as a 400 on a long conversation. Side
|
||||
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
|
||||
100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
|
||||
Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
|
||||
The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
|
||||
pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
|
||||
decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. Both
|
||||
moves measured; both directions cost or buy window, never pool. See
|
||||
[[frank-small-llama-mimo-q5km]].
|
||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
||||
costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
|
||||
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
||||
@@ -939,7 +959,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
@@ -949,7 +969,8 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
|
||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
|
||||
|
||||
Reference in new issue
Block a user