serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)	The retired 2026-10-01 frank-zealot runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload.	llm frank-zealot dankclaw llama-cpp rtx3090 mimo gguf kv-cache	2026-10-01

frank-zealot — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)

RETIRED 2026-10-01 → 10-02 (one night): superseded by frank-zealot-llama-mimo-q5km, the same runtime and shape one quant rung down. The Q6_K file stays at /home/dank/models/MiMo-Q6K/ as the rollback target (on disk 2026-10-18, stat -c %s → 7793710624). :7771 today runs frank-zealot-llama-prism-bonsai9-mtp (llama.cpp PrismML fork build 10777, Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf, 2 × 262,144, exact f16 K + f16 V, MTP off by measurement).

Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF path for asymmetric KV — unquantized K + q8_0 V — which only llama.cpp exposes per-tensor. The per-tensor lever stayed; this pairing did not — the live endpoint bills exact f16 K + f16 V — 32,768 B/token = 16,384 MiB over the 524,288-position pool of two 262,144 windows (scripts/frank-zealot.env:57-58,80-81).

Identity

Field	Value
Endpoint	192.168.0.143:7771, served alias frank-small in this era (ALIAS reached frank-secondary 2026-10-06 and frank-zealot 2026-10-09, commit 7c77326)
Runtime	llama.cpp master (qwen35 model support), CUDA build sm_86, at /home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
Payload	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K, 7,793,710,624 B, sha256 ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d, at /home/dank/models/MiMo-Q6K/
Unit	root frank-zealot.service (repo scripts/frank-zealot.service, User=dank, relinked), env scripts/frank-zealot.env, serve scripts/serve-frank-zealot.sh
Repo pins	tests/frank_zealot_unit.bats (stub-argv seam), deploy_llm_zealot gates in scripts/deploy-remote.sh

Shape and the KV contract

--ctx-size 524288 --parallel 2 — llama.cpp divides the pool across slots, so each session gets n_ctx_slot = 262144, the model's full native window (boot log confirms n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false').

Frank's KV contract: K unquantized (f16), V q8_0 — --cache-type-k f16 --cache-type-v q8_0. Two hard dependencies ride on it:

• --flash-attn on is mandatory: quantized V exists only on the FA path. Without the flag the engine falls back silently and the contract becomes a lie. The serve script pins the trio; the bats seam pins the script.
• The VRAM budget: attention KV lives in the 8 full-attention layers only (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈ 12.6 GiB. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈ 20.9 GiB measured against 24.5 GiB — the "room for checkpoints" the order asked for. If graph capture OOMs, halve CTX_SIZE; never quantize K to fix it.

Client contract

• Thinking arrives in reasoning_content (--reasoning-format deepseek) — the same field SGLang's qwen3 parser used, so models.yml reasoningContentField did NOT move. Live-verified 2026-10-01 22:57.
• Thinking is off unless the request asks: the MiMo GGUF template defaults enable_thinking false; models.yml now ships chat_template_kwargs: {enable_thinking: true, preserve_thinking: true} in extraBody. A raw client that omits it gets a no-thinking answer (measured — this is not a fault).
• Tools: --jinja + server-side tool parsing, verified tool_calls round-trip 2026-10-01 23:0x.
• Text-only in this era. The GGUF carries no vision projector; the in-checkpoint tower existed only in the SGLang safetensors, so models.yml carried input: [text]. Vision returned with the next payloads: the endpoint is input: [text, image] today (~/.omp/agent/models.yml:214).

No spec decode (yet)

The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the cross-model DFlash draft (/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf, Q8_0 — that path is no longer on disk; the 2026-10-18 census of /home/dank/models/ has no DFlash-Qwen35/) is an optimization-loop candidate, not a default — --spec-type draft-dflash needs acceptance measured on this traffic first. The bats seam fails if a draft flag ships unannounced.

Build

No ccache, CUDA 12.4 system nvcc with --gcc-install-dir-free gcc-13 host compiler (nvcc 12.4 rejects gcc ≥ 14):

cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
  -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
cmake --build build-cuda -j 16 --target llama-server

The SGLang venv (/home/dank/GitPackages/sglang-cu128) is still on disk; the INT8 AutoRound payload (/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound) is not — it was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ on 2026-10-02 and that stage dir is gone as of 2026-10-18, so this rollback re-fetches with old/download-frank-zealot-sglang-int8.sh: repoint the unit's serve script and env from git history (git show frank-dragoon-gufo-q4kxl-20261001 era is BEFORE this; use the parent of the 2026-10-01 night swap commit).

When it misbehaves

• Port 7771 dark: journalctl -u frank-zealot -n 50; the serve script exits 1 naming the missing file if the binary or GGUF moved.
• No CUDA GPUs are available: the 3090 vanished from the bus or the UVM node is missing — nvidia-modprobe -u -c=0 (the serve script runs it).
• Requests hang at first token while /health is 200: both slots busy — two concurrent long generations are the ceiling by contract.
• Slow decode after a CTX change: check nvidia-smi — swapping KV out of VRAM into unified memory halves the rate silently.