serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md

---

type: reference title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S description: The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves. tags:

• llm
• frank-small
• dankclaw
• llama.cpp
• rtx3090
• qwen35
• mtp
• vision timestamp: 2026-09-16

---

frank-small — llama.cpp on the RTX 3090

RETIRED 2026-09-17 — replaced by frank-small-gemma4-12b. Kept as the rollback target and the record of the deployment it succeeded. Originally ACTIVE from 2026-09-16, created when Dankclaw's inference was re-split: halogen-flash-server took the gfx1151 iGPU back for frank-large, which freed the RTX 3090 for this endpoint.

Identity

what	value
runtime	mainline llama.cpp master 37b53fd (2026-09-16), CUDA build for sm_86
binary	/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
unit	root systemd unit frank-small.service (repo scripts/frank-small.service, relink_united into /etc/systemd/system/)
flags	scripts/frank-small.env; entrypoint scripts/serve-frank-small.sh
endpoint	192.168.0.143:7771, alias frank-small
model	ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, the IQ3_S-mtp build + BF16 mmproj

It is a root unit rather than a Quadlet because the runtime is a native CUDA binary, not a container: there is no image to install and no generated unit to avoid. Boot persistence is the unit's [Install] WantedBy=multi-user.target; Restart=on-failure / RestartSec=10 make it self-healing; LimitMEMLOCK=infinity covers CUDA's pinned staging buffers.

Build recipe (the deployed binary):

cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build-cuda --target llama-server -j

GGML_CUDA_FA_ALL_QUANTS is deliberately off: it is only needed for quantized KV with flash attention (see the CUDA lesson in AGENTS.md — without it, quantized-KV prefill collapses ~12x), and this service runs f16 KV.

The model, and why the -mtp build specifically

GSQ-RCO is a per-tensor mixed-precision allocation produced by a gradient-based search under a total size budget, rather than one quantization type applied throughout. IQ3_S at 3.50 bpw is the release's task-lossless operating point: AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly, GPQA-Diamond is 0.51 behind — at 11.8 GB against 53.8 GB.

Two files are required, and the second suffix matters:

file	size	why
Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf	12,120,016,960 B	the -mtp build is the MTP build: it carries the head as blk.64.nextn.* (qwen35.nextn_predict_layers = 1)
mmproj-Qwen3.8-27B-BF16.gguf	931,146,528 B	the vision encoder + projector (clip.projector_type = qwen3vl_merger, 27 blocks)

Both live in /home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/, fetched with huggingface_hub (which verifies the LFS sha256 and resumes).

MTP is a property of the file, not of the flags. --spec-type draft-mtp sets load_mtp, and the draft context runs on the main model's weights (common/common.cpp passes shares_model when no --model-draft is given; common/speculative.cpp gives it LLAMA_CONTEXT_TYPE_MTP). So there is no second copy of a 12 GiB model in VRAM, and --model-draft would be wrong here — it would load the whole file again.

Architecture and the KV arithmetic

GGUF arch qwen35: 64 layers laid out as 16 x (3 x (Gated DeltaNet -> FFN) + 1 x (Gated Attention -> FFN)). Only the 16 full-attention layers take KV, so a token costs

2 (K,V) x 4 heads x 256 head_dim x 2 B = 64 KiB at f16   (34 KiB q8_0, 18 KiB q4_0)

and the 48 DeltaNet layers add ~0.16 GiB per sequence of constant-size state in f32, at any depth. The MTP draft context is n_ctx long over one nextn layer, so it costs 4 KiB/token of its own.

Sizing (RTX 3090, 24576 MiB)

term	GiB
weights	11.29
mmproj	0.87
MTP draft KV (131,072 x 4 KiB)	0.50
DeltaNet state (one sequence)	0.16
CUDA context + prefill/decode buffers	~1.5
KV, one slot at 131,072 x 64 KiB	8.00
modelled total	~22.3 of 24

The binding number is the KV: about 9.5 GiB is left after the fixed terms, which is ~155k tokens at 64 KiB/token. Alternatives inside the same budget: 2 x 65536 at f16, or 4 x 65536 / 1 x 262144 at q8_0 — the quantized variants need a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON. --parallel N divides --ctx-size between the slots; it does not multiply.

Measured at deploy: 22,362 MiB in use of 24,576, n_slots = 1, n_ctx_slot = 131072.

Deployed flags

flag	value	why
CTX_SIZE / PARALLEL	131072 / 1	one conversation per slot, full f16 precision, no KV quantization risk
CACHE_TYPE_K/V	f16	quantizing would need the FA_ALL_QUANTS build for no benefit here
NGL	999	every layer on the card
FLASH_ATTN	on
SPEC_TYPE	draft-mtp	the model's own head; draft depth stays llama.cpp's default of 3
BATCH_SIZE / UBATCH_SIZE	2048 / 512	the defaults; ubatch 512 keeps the DeltaNet chunk buffers small
NUM_THREADS / THREAD_BATCH	8 / 8	the work is on the GPU; the host CPU belongs to halogen's engine
TEMPERATURE	0.0	greedy default (fleet contract) — clients that send their own values get them
REASONING_FORMAT / REASONING_BUDGET	deepseek / -1	thinking into reasoning_content, unlimited budget (the model decides)
--jinja	pinned	the GGUF's own template supplies reasoning_effort, preserve_thinking and the tool-call grammar. This build defaults it on; pinning it means a future default change cannot silently take tools away
--metrics	on	Prometheus scrapes /metrics here (frank-small job)

Client contract

• One slot. A second concurrent request queues; it is not served in parallel, and agent fan-out through this endpoint serializes. Widen with PARALLEL (which divides the context) if that becomes the wrong trade.
• Thinking counts against the budget. A request whose budget runs out mid-thought returns finish_reason: "length" with an empty content and the partial reasoning in reasoning_content — observed live on a vision request at max_tokens: 64. Give reasoning models a real budget.
• Tools work (tool_calls returned; verified with a function-call round trip) because --jinja serves the model's own template. This is also what fixed subagents fleet-wide: the ik runtime that served frank-large until 2026-09-16 rejected every request carrying tools with 500 tools param requires --jinja flag.
• Images are accepted as OpenAI image_url content parts with a data: URL or bare base64. This is the fleet's only image path: halogen's frank-large runs with no tower and refuses images with a named reason.
• reasoning_effort accepts only xhigh, medium, low. The template raises on anything else — a 500, not a clamp — so the harness folds its own levels onto that set in compat.reasoningEffortMap (high: xhigh, max: xhigh). An unmapped level fails on the first request.
• /v1/models reports frank-small; /health answers 200 once the model is loaded and the projector is up.

Verified at deploy (2026-09-16)

• systemctl is-enabled/is-active frank-small.service → enabled / active.
• /v1/models → frank-small; a chat completion returns content and reasoning_content.
• A tool call returns tool_calls with the expected arguments.
• Slot timing lines show draft acceptance = 0.66-0.94, mean len = 2.95-3.83 — MTP is engaged, not merely configured.
• An image request returns the right colour answer from the projector.
• Harness routing: role smol lands here, role slow on frank-large, and a real subagent (the task role) completes a tool-using turn.

Operate

systemctl status frank-small.service          # root unit
journalctl -u frank-small.service -n 60       # load + per-request timing
curl -s http://127.0.0.1:7771/health
curl -s http://127.0.0.1:7771/v1/models
curl -s http://127.0.0.1:7771/metrics | grep -E 'spec_decode|kv_cache_usage'
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

/metrics carries llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total, which is the cheapest way to confirm MTP is running, and llamacpp:kv_cache_usage_ratio for how full the slot is.

Troubleshooting

symptom	cause
unit restarts immediately, frank-small: missing <path> in the journal	the model, the projector or the binary is absent — the serve script refuses to start rather than half-load; finish the download
500 … Jinja Exception: Unexpected reasoning effort <level>	a client sent an effort the template does not accept; map it (see the client contract)
500 tools param requires --jinja flag	impossible here (--jinja is pinned) — you are looking at a different endpoint, most likely the retired ik runtime's logs
finish_reason: "length" with empty content	the budget ran out inside the thinking block; raise max_tokens
a request that waits, then answers normally	the single slot was busy (or the KV is nearly full); raise PARALLEL or expect serialization
CUDA OOM at load	another process holds the 3090, or the KV was widened past ~9.5 GiB; check nvidia-smi and re-derive from the sizing table

Rollback

frank-small is a new endpoint, so rollback is retirement: systemctl disable --now frank-small.service, move /etc/systemd/system/frank-small.service into ~/.AGENTTRASH/, and remove the harness provider + the smol/tiny/task/vision role entries. The old frank-small (FreeToken on Dankserver, retired 2026-09-15) is a different service with a different card — see freetoken-frank-small.

Citations

• ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF — the quantized checkpoints, the -mtp builds and the projector
• Qwen/Qwen3.8-27B — base model: 64 layers, Gated DeltaNet/Gated Attention layout, 262,144 native context
• ggml-org/llama.cpp — --spec-type draft-mtp, the MTP context type, and the Qwen3-VL projector path