0 Fork 0

Files

T

Frank 8f129180d8 docs+scripts: reconcile payload inventory after both swaps

Staged 59 GiB of retired LLM payloads on Dankclaw to
~dank/.AGENTTRASH/20261002-retired-llm-payloads/ (vLLM MiMo BF16, Qwen3.5-9B
MTP source, both DFlash dirs, Gemma4-12B-QAT, SGLang MiMo INT8) and corrected
every doc claim about what is on disk: gufo's rollback pair (Q4_K_XL weights +
the Q8_K_XL DFlash2 drafter) and frank-small's Q6_K stay in place, so the
rollback paths in the tags' env files still resolve. The vLLM-era frank-large
fetch trio moves to old/ with its runtime; runbook links updated.

2026-10-02 02:31:39 -07:00

6.1 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)

ACTIVE since 2026-10-02. One quant step down from frank-small-llama-mimo-q6k on the same runtime, the same shape and the same KV contract. Frank's order: bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on llama.cpp, two contexts, whatever context size fits the 3090. The answer to "whatever fits" is unchanged — 2 x 262,144, because 262,144 is the model's native window and two of them already fit.

Identity

Field	Value
Endpoint	`192.168.0.143:7771`, served alias `frank-small`
Runtime	llama.cpp master (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap
Payload	`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` Q5_K_M, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)
Unit	root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`
Repo pins	`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`

Shape and the KV contract

--ctx-size 524288 --parallel 2 — llama.cpp divides the pool across slots, so each session gets n_ctx_slot = 262144, the model's full native window (boot log confirms n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false').

The KV contract is Frank's: K unquantized (f16), V q8_0 — --cache-type-k f16 --cache-type-v q8_0. Two hard dependencies ride on it:

--flash-attn on is mandatory: quantized V exists only on the FA path. Without the flag the engine falls back silently and the contract becomes a lie. The serve script pins the trio; the bats seam pins the script.
The VRAM budget: attention KV lives in the 8 full-attention layers only (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈ 12.5 GiB. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) + compute buffers/CUDA graphs ⇒ 20,164 MiB measured against the card's 24,576 MiB.

Why the quant step did not buy a bigger window. The model cannot be run past its trained 262,144 without scaling, and two of those windows are already the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That arithmetic is a test (the KV budget fits the 3090…), so a future CTX_SIZE or quant change that breaks the fit fails the suite instead of the boot.

Measured on this file (2026-10-02)

Check	Result
Boot to healthy	~20 s, `n_slots = 2`, `n_ctx_slot = 262144`
VRAM	20,164 MiB / 24,576 MiB (`nvidia-smi`)
Decode, temp 0, no thinking	100.4–101.8 tok/s (256–512 token completions at ~900-token depth)
Two concurrent requests	both answered, 256 tokens each in 3.2 s wall (slots are real)
Tools	native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)
Thinking	`reasoning_content` populated on a reasoning-requiring prompt

Client contract

Thinking arrives in reasoning_content (--reasoning-format deepseek), the field models.yml reads for this endpoint.
Thinking is off unless the request asks: the MiMo GGUF template defaults enable_thinking false, so models.yml ships chat_template_kwargs: {enable_thinking: true, preserve_thinking: true} in extraBody. A raw client that omits it gets a direct answer — measured, and not a fault. Trivial questions produce no thinking even with the flag on.
Tools: --jinja + server-side parsing, verified 2026-10-02.
Text-only. This GGUF carries no vision tensors. The HF repo does ship a paired mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf, so vision is one --mmproj away if it is ever wanted here (frank-primary is the fleet's vision endpoint today).

No spec decode

The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00 ghost), and the cross-model DFlash draft (/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf) is an optimization candidate, not a default — acceptance has to be measured on this traffic first. The bats seam fails if a draft flag ships unannounced.

Rollback

Target	How
Q6_K, same runtime	`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default
SGLang + INT8	frank-small-sglang-mimo-v26 — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`

When it misbehaves

Port 7771 dark: journalctl -u frank-small -n 50; the serve script exits 1 naming the missing file if the binary or GGUF moved.
No CUDA GPUs are available: the 3090 vanished from the bus or the UVM node is missing — nvidia-modprobe -u -c=0 (the serve script runs it).
Requests queue behind each other: two concurrent long generations are the slot ceiling by contract.
Slow decode after a CTX or quant change: check nvidia-smi — pushing KV out of VRAM into unified memory halves the rate silently.

6.1 KiB Raw Blame History Unescape Escape