Frank
72bd4c967e
frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115 tensor-type lock-in that forced the fork, the pinned build (set_rows history), the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live client contract (reasoning_content, effort fold, vision restored) and rollback. frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired (rollback target). gufo runbook marked retired (rollback target). Both indexes, the fleet topology table, the vision/models.yml notes, the benchmarking law and three new Key Lessons updated.
5.2 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| reference | frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired) | The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload. |
|
2026-10-01 |
frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
RETIRED 2026-10-01 → 10-02 (one night): superseded by frank-small-llama-mimo-q5km, the same runtime and shape one quant rung down. The Q6_K file stays at
/home/dank/models/MiMo-Q6K/as the rollback target.Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF path for asymmetric KV — unquantized K + q8_0 V — which only llama.cpp exposes per-tensor. That contract is what the current deployment kept.
Identity
| Field | Value |
|---|---|
| Endpoint | 192.168.0.143:7771, served alias frank-small |
| Runtime | llama.cpp master (qwen35 model support), CUDA build sm_86, at /home/dank/GitPackages/llama-master/build-cuda/bin/llama-server |
| Payload | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K, 7,793,710,624 B, sha256 ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d, at /home/dank/models/MiMo-Q6K/ |
| Unit | root frank-small.service (repo scripts/frank-small.service, User=dank, relinked), env scripts/frank-small.env, serve scripts/serve-frank-small.sh |
| Repo pins | tests/frank_small_unit.bats (stub-argv seam), deploy_llm_small gates in scripts/deploy-remote.sh |
Shape and the KV contract
--ctx-size 524288 --parallel 2 — llama.cpp divides the pool across slots,
so each session gets n_ctx_slot = 262144, the model's full native window
(boot log confirms n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false').
Frank's KV contract: K unquantized (f16), V q8_0 —
--cache-type-k f16 --cache-type-v q8_0. Two hard dependencies ride on it:
--flash-attn onis mandatory: quantized V exists only on the FA path. Without the flag the engine falls back silently and the contract becomes a lie. The serve script pins the trio; the bats seam pins the script.- The VRAM budget: attention KV lives in the 8 full-attention layers only
(4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
≈ 12.6 GiB. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
20.9 GiB measured against 24.5 GiB — the "room for checkpoints" the
order asked for. If graph capture OOMs, halve
CTX_SIZE; never quantize K to fix it.
Client contract
- Thinking arrives in
reasoning_content(--reasoning-format deepseek) — the same field SGLang's qwen3 parser used, somodels.ymlreasoningContentFielddid NOT move. Live-verified 2026-10-01 22:57. - Thinking is off unless the request asks: the MiMo GGUF template
defaults
enable_thinkingfalse;models.ymlnow shipschat_template_kwargs: {enable_thinking: true, preserve_thinking: true}inextraBody. A raw client that omits it gets a no-thinking answer (measured — this is not a fault). - Tools:
--jinja+ server-side tool parsing, verifiedtool_callsround-trip 2026-10-01 23:0x. - Text-only now. The GGUF carries no vision projector; the
in-checkpoint tower existed only in the SGLang safetensors.
models.ymlinput: [text].
No spec decode (yet)
The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
cross-model DFlash draft (/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf,
Q8_0) is an optimization-loop candidate, not a default — --spec-type draft-dflash needs acceptance measured on this traffic first. The bats
seam fails if a draft flag ships unannounced.
Build
No ccache, CUDA 12.4 system nvcc with --gcc-install-dir-free gcc-13 host
compiler (nvcc 12.4 rejects gcc ≥ 14):
cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
-DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
cmake --build build-cuda -j 16 --target llama-server
The SGLang venv (/home/dank/GitPackages/sglang-cu128) and the INT8
AutoRound payload (/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound)
stay on disk as the rollback: repoint the unit's serve script and env from
git history (git show frank-large-gufo-q4kxl-20261001 era is BEFORE this;
use the parent of the 2026-10-01 night swap commit).
When it misbehaves
- Port 7771 dark:
journalctl -u frank-small -n 50; the serve script exits 1 naming the missing file if the binary or GGUF moved. No CUDA GPUs are available: the 3090 vanished from the bus or the UVM node is missing —nvidia-modprobe -u -c=0(the serve script runs it).- Requests hang at first token while
/healthis 200: both slots busy — two concurrent long generations are the ceiling by contract. - Slow decode after a CTX change: check
nvidia-smi— swapping KV out of VRAM into unified memory halves the rate silently.