dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-large Q6 uncensored cutover + frank-small 2x200704 f16-KV reslice

Browse Source 
- new active runbook frank-large-rocmfpx-q6agent-uncensored-mtp.md: the
  Q6_0_ROCMFPX_AGENT file (ftype 114, 339 type-102 + 159 type-103 tensors),
  27.6-28.1 t/s at 86.8% draft acceptance, the measured ngram-mod chain
  rejection (-15% prose: recurrent-state rollback disabled), the
  bench-window hardening laws
- frank-large-rocmfpx-q8agent-mtp.md retired; rollback section corrected
  (gufo payloads removed from disk 2026-10-03 - that rollback is now a
  re-download; the aligned Q8 file is the direct one, still on disk)
- frank-small-llama-mimo-q5km.md reshaped to 2 x 200704 with fully
  unquantized f16 KV: boot-pair table gains the measured row (20,530 used /
  4,046 free), 104.84 tok/s, slot-window contract at 200704, rollback to
  the 3x180224 + q8_0-V shape documented
- AGENTS.md: fleet table rows, both endpoint sections, the chain law
  extended with the recurrent-model measurement, the --parallel law with
  the reslice, runbook index (active + retired)
- LLM/index.md + root index entries updated
This commit is contained in:
Frank committed 2026-10-03 08:01:44 -07:00
1 parent 47a2c1287b
commit b80886e318
6 files changed
+438 -216
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
+78 -78
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two conversations of 200,704 tokens, fully unquantized f16 KV, mmproj vision, the boot-pair arithmetic that sizes the pool, and what to check when it misbehaves."
tags:
- llm
- frank-small
@@ -11,23 +11,26 @@ tags:
- mimo
- gguf
- kv-cache
timestamp: 2026-10-02
timestamp: 2026-10-03
---
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
> **ACTIVE since 2026-10-02.** One quant step down from
> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
> contract, then **resized twice the same day** on Frank's orders: first to
> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
> card free. Frank's arithmetic request — "as much as can fit while keeping
> overhead open space to 2-3gb on the 3090" — is answered by the two
> measurements in *Shape and the KV contract* below; the answer is
> **3 × 180,224**, not 3 × 262,144.
> **ACTIVE since 2026-10-02, resliced 2026-10-03.** One quant step down from
> [[frank-small-llama-mimo-q6k]] on the same runtime, resized twice on
> 2026-10-02 (three contexts with vision, then the pool that kept 2–3 GiB of
> the card free), and on 2026-10-03 Frank ordered the third shape: **two
> conversations of 200,704 tokens each with the KV VALUES unquantized** —
> *"2 streams with 200700 tokens context or more… if we have extra space on
> the 3090, unquantize the kv."* The freed third slot pays for exact V, and
> the boot lands **20,530 MiB used / 4,046 MiB free** — above the 2–3 GiB
> floor, with full-precision KV.
> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
> of per-conversation memory versus the two-slot shape: 180,224 tokens each
> instead of the model's native 262,144. See *What three slots cost*.
> **The tradeoffs of the reslice:** one fewer simultaneous conversation, a
> per-slot window that grew from 180,224 to 200,704 (still below the native
> 262,144), and ~2 % slower decode (104.8 vs 106.8 tok/s) from the larger
> per-token KV traffic. The 3 × 180,224 shape is one env edit away — see
> *Rollback*.
## Identity
@@ -41,77 +44,72 @@ timestamp: 2026-10-02
## Shape and the KV contract
`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
`PARALLEL`; the bats seam enforces it.
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03:
`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST
stay divisible by `PARALLEL`; the bats seam enforces it.
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized
(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it
replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two
slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra
V traffic is the measured ~2 % decode tax. The serve script still passes both
flags explicitly — the asymmetric-era law that a quantized V requires
`--flash-attn on` stays pinned by the seam.
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
Without the flag the engine falls back silently and the contract becomes a
lie. The serve script pins the trio; the bats seam pins the script.
- The VRAM budget: attention KV lives in the 8 full-attention layers only
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
**Pool sizes on this stack come from boots, not formulas.** The compute
buffers, CUDA graphs and vision tower all bill separately, so every shape
change is priced from measured rows:
**The pool size came from two boots, not from a formula.** `free`-style
arithmetic on this stack is unreliable because the compute buffers, CUDA
graphs and now the vision tower all bill separately, so the fit was measured
at two shapes and solved between them:
|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)|
|---|---|---|---|---|
|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**|
|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**|
|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB|
|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**|
The two rows price the projector and its image buffers at **~1.4 GiB** — not
the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
activation and compute buffers ride along. Solving the second row's non-KV cost
against Frank's 2–3 GiB band: the card is 24,576 MiB, so
`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
**2,441 MiB free** (measured, 21,684 MiB used).
The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim ×
2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB
of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free**
(measured steady-state — the KV pool is allocated whole at boot, so this is
the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not
an invitation: another 32,768 tokens of pool (1 GiB) would trade the last
safety margin for window the harness compacts before reaching anyway.
**Why 3 × 262,144 was never possible.** Three native windows are 786,432
tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
6 GiB.
**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is
524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the
floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the
measured band.
## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)
|Check|Result|
|---|---|
|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
|Prefill|**3,814 tok/s** (same request)|
|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged|
|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)|
|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
|Vision|projector loads at boot (`loaded multimodal model`); the red-PNG probe stands from the 2026-10-02 verification — the vision path is unchanged by this reslice|
|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
scan at equal prompt depth, and one less slot's DeltaNet state resident.
## What the slot geometry costs
## What three slots cost
The per-slot window is the whole price, in both directions:
The per-slot window is the whole price. Two consequences the fleet depends on:
- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
sets `contextWindow: 180224` for this endpoint — the harness compacts before
- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
sets `contextWindow: 200704` for this endpoint — the harness compacts before
it reaches the ceiling instead of discovering it.
- **frank-primary's compaction prompt fits inside one slot again.** At
frank-primary's shape-D window (200,704) its compaction trigger is 140,492
tokens (70 %), so one compaction request lands here with 39,732 tokens of
room. At the previous 262,144 window the trigger was 183,501 — *above* this
slot — and the request had to be truncated or split. Nothing in the fleet
routes compaction here automatically; the point is that the 2026-10-02 window
trim restored the fit.
- **frank-large's tallest pre-compaction prompt now fits here.** Compaction
for frank-large sessions runs on this endpoint (`compactionModel` in
`models.yml`); frank-large compacts at 70 % of 262,144 = 183,501 tokens,
which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
tokens to spare. The 2026-10-02 window trim had restored the fit for
frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens
the fit to frank-large itself. Nothing routes compaction automatically —
the point is the ceiling no longer forbids it.
- Two simultaneous generations are the slot ceiling; a third request queues
behind a released slot (the mechanism measured on the 3-slot shape).
## Client contract
@@ -147,8 +145,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|Target|How|
|---|---|
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
@@ -158,13 +157,14 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests queue behind each other: three concurrent long generations are the
slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
for the three, 3.0 s for the fourth).
- `400 … exceeds the available context size (180224 tokens)`: the conversation
outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
and the per-slot window is the binding limit. Shorten the prompt, compact, or
roll back to the 262,144 shape above.
- Requests queue behind each other: two concurrent long generations are the
slot ceiling by contract (2026-10-03 shape); a third waits for a slot to
free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for
the fourth).
- `400 … exceeds the available context size (200704 tokens)`: the conversation
outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
and the per-slot window is the binding limit. Shorten the prompt, compact,
or roll back to the 262,144 shape above.
- Vision answers a color it never saw: the client sent `images` instead of an
`image_url` content part (see *Client contract*).
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out