Frank
87caca4894
AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no longer why), and the runbook index. New frank-small-gemma4-12b runbook: payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and the fit-probe error that looks like a fault at boot. The Qwen runbook and both indexes carry the retirement; work log updated.
206 lines
11 KiB
Markdown
206 lines
11 KiB
Markdown
---
|
|
type: reference
|
|
title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S
|
|
description: The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves.
|
|
tags:
|
|
- llm
|
|
- frank-small
|
|
- dankclaw
|
|
- llama.cpp
|
|
- rtx3090
|
|
- qwen35
|
|
- mtp
|
|
- vision
|
|
timestamp: 2026-09-16
|
|
---
|
|
|
|
# frank-small — llama.cpp on the RTX 3090
|
|
|
|
> **RETIRED 2026-09-17** — replaced by [[frank-small-gemma4-12b]]. Kept as
|
|
> the rollback target and the record of the deployment it succeeded.
|
|
> Originally ACTIVE from 2026-09-16, created when Dankclaw's inference was
|
|
> re-split: [[halogen-flash-server]] took the gfx1151 iGPU back for
|
|
> `frank-large`, which freed the RTX 3090 for this endpoint.
|
|
|
|
## Identity
|
|
|
|
| what | value |
|
|
|---|---|
|
|
| runtime | mainline llama.cpp master `37b53fd` (2026-09-16), CUDA build for sm_86 |
|
|
| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
|
|
| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
|
|
| flags | `scripts/frank-small.env`; entrypoint `scripts/serve-frank-small.sh` |
|
|
| endpoint | `192.168.0.143:7771`, alias `frank-small` |
|
|
| model | `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`, the `IQ3_S-mtp` build + BF16 mmproj |
|
|
|
|
It is a root unit rather than a Quadlet because the runtime is a native CUDA
|
|
binary, not a container: there is no image to install and no generated unit to
|
|
avoid. Boot persistence is the unit's `[Install] WantedBy=multi-user.target`;
|
|
`Restart=on-failure` / `RestartSec=10` make it self-healing; `LimitMEMLOCK=infinity`
|
|
covers CUDA's pinned staging buffers.
|
|
|
|
Build recipe (the deployed binary):
|
|
|
|
```bash
|
|
cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
|
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
|
|
cmake --build build-cuda --target llama-server -j
|
|
```
|
|
|
|
`GGML_CUDA_FA_ALL_QUANTS` is deliberately **off**: it is only needed for
|
|
*quantized* KV with flash attention (see the CUDA lesson in AGENTS.md — without
|
|
it, quantized-KV prefill collapses ~12x), and this service runs f16 KV.
|
|
|
|
## The model, and why the `-mtp` build specifically
|
|
|
|
GSQ-RCO is a **per-tensor mixed-precision** allocation produced by a
|
|
gradient-based search under a total size budget, rather than one quantization
|
|
type applied throughout. IQ3_S at 3.50 bpw is the release's task-lossless
|
|
operating point: AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
|
|
GPQA-Diamond is 0.51 behind — at 11.8 GB against 53.8 GB.
|
|
|
|
Two files are required, and the second suffix matters:
|
|
|
|
| file | size | why |
|
|
|---|---|---|
|
|
| `Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` | 12,120,016,960 B | the `-mtp` build **is** the MTP build: it carries the head as `blk.64.nextn.*` (`qwen35.nextn_predict_layers = 1`) |
|
|
| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,146,528 B | the vision encoder + projector (`clip.projector_type = qwen3vl_merger`, 27 blocks) |
|
|
|
|
Both live in `/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/`, fetched with
|
|
`huggingface_hub` (which verifies the LFS sha256 and resumes).
|
|
|
|
**MTP is a property of the file, not of the flags.** `--spec-type draft-mtp`
|
|
sets `load_mtp`, and the draft context runs on the *main model's* weights
|
|
(`common/common.cpp` passes `shares_model` when no `--model-draft` is given;
|
|
`common/speculative.cpp` gives it `LLAMA_CONTEXT_TYPE_MTP`). So there is no
|
|
second copy of a 12 GiB model in VRAM, and `--model-draft` would be wrong here —
|
|
it would load the whole file again.
|
|
|
|
## Architecture and the KV arithmetic
|
|
|
|
GGUF arch `qwen35`: 64 layers laid out as `16 x (3 x (Gated DeltaNet -> FFN) +
|
|
1 x (Gated Attention -> FFN))`. Only the 16 full-attention layers take KV, so a
|
|
token costs
|
|
|
|
```
|
|
2 (K,V) x 4 heads x 256 head_dim x 2 B = 64 KiB at f16 (34 KiB q8_0, 18 KiB q4_0)
|
|
```
|
|
|
|
and the 48 DeltaNet layers add ~0.16 GiB per sequence of constant-size state in
|
|
f32, at any depth. The MTP draft context is `n_ctx` long over one nextn layer,
|
|
so it costs 4 KiB/token of its own.
|
|
|
|
## Sizing (RTX 3090, 24576 MiB)
|
|
|
|
| term | GiB |
|
|
|---|---|
|
|
| weights | 11.29 |
|
|
| mmproj | 0.87 |
|
|
| MTP draft KV (131,072 x 4 KiB) | 0.50 |
|
|
| DeltaNet state (one sequence) | 0.16 |
|
|
| CUDA context + prefill/decode buffers | ~1.5 |
|
|
| **KV, one slot at 131,072 x 64 KiB** | **8.00** |
|
|
| **modelled total** | **~22.3 of 24** |
|
|
|
|
The binding number is the KV: about 9.5 GiB is left after the fixed terms, which
|
|
is ~155k tokens at 64 KiB/token. Alternatives inside the same budget: `2 x
|
|
65536` at f16, or `4 x 65536` / `1 x 262144` at q8_0 — the quantized variants
|
|
need a rebuild with `-DGGML_CUDA_FA_ALL_QUANTS=ON`. **`--parallel N` divides
|
|
`--ctx-size` between the slots**; it does not multiply.
|
|
|
|
Measured at deploy: **22,362 MiB** in use of 24,576, `n_slots = 1`,
|
|
`n_ctx_slot = 131072`.
|
|
|
|
## Deployed flags
|
|
|
|
| flag | value | why |
|
|
|---|---|---|
|
|
| `CTX_SIZE` / `PARALLEL` | `131072` / `1` | one conversation per slot, full f16 precision, no KV quantization risk |
|
|
| `CACHE_TYPE_K/V` | `f16` | quantizing would need the FA_ALL_QUANTS build for no benefit here |
|
|
| `NGL` | `999` | every layer on the card |
|
|
| `FLASH_ATTN` | `on` | |
|
|
| `SPEC_TYPE` | `draft-mtp` | the model's own head; draft depth stays llama.cpp's default of 3 |
|
|
| `BATCH_SIZE` / `UBATCH_SIZE` | `2048` / `512` | the defaults; ubatch 512 keeps the DeltaNet chunk buffers small |
|
|
| `NUM_THREADS` / `THREAD_BATCH` | `8` / `8` | the work is on the GPU; the host CPU belongs to halogen's engine |
|
|
| `TEMPERATURE` | `0.0` | greedy default (fleet contract) — clients that send their own values get them |
|
|
| `REASONING_FORMAT` / `REASONING_BUDGET` | `deepseek` / `-1` | thinking into `reasoning_content`, unlimited budget (the model decides) |
|
|
| `--jinja` | pinned | the GGUF's own template supplies `reasoning_effort`, `preserve_thinking` and the tool-call grammar. This build defaults it on; pinning it means a future default change cannot silently take tools away |
|
|
| `--metrics` | on | Prometheus scrapes `/metrics` here (`frank-small` job) |
|
|
|
|
## Client contract
|
|
|
|
- **One slot.** A second concurrent request queues; it is not served in
|
|
parallel, and agent fan-out through this endpoint serializes. Widen with
|
|
`PARALLEL` (which divides the context) if that becomes the wrong trade.
|
|
- **Thinking counts against the budget.** A request whose budget runs out
|
|
mid-thought returns `finish_reason: "length"` with an empty `content` and the
|
|
partial reasoning in `reasoning_content` — observed live on a vision request
|
|
at `max_tokens: 64`. Give reasoning models a real budget.
|
|
- **Tools work** (`tool_calls` returned; verified with a function-call round
|
|
trip) because `--jinja` serves the model's own template. This is also what
|
|
fixed subagents fleet-wide: the ik runtime that served `frank-large` until
|
|
2026-09-16 rejected every request carrying `tools` with
|
|
`500 tools param requires --jinja flag`.
|
|
- **Images are accepted** as OpenAI `image_url` content parts with a `data:`
|
|
URL or bare base64. This is the fleet's only image path: halogen's
|
|
`frank-large` runs with no tower and refuses images with a named reason.
|
|
- **`reasoning_effort` accepts only `xhigh`, `medium`, `low`.** The template
|
|
raises on anything else — a 500, not a clamp — so the harness folds its own
|
|
levels onto that set in `compat.reasoningEffortMap` (`high: xhigh`,
|
|
`max: xhigh`). An unmapped level fails on the first request.
|
|
- `/v1/models` reports `frank-small`; `/health` answers 200 once the model is
|
|
loaded and the projector is up.
|
|
|
|
## Verified at deploy (2026-09-16)
|
|
|
|
- `systemctl is-enabled/is-active frank-small.service` → `enabled` / `active`.
|
|
- `/v1/models` → `frank-small`; a chat completion returns `content` **and**
|
|
`reasoning_content`.
|
|
- A tool call returns `tool_calls` with the expected arguments.
|
|
- Slot timing lines show `draft acceptance = 0.66-0.94, mean len = 2.95-3.83`
|
|
— MTP is engaged, not merely configured.
|
|
- An image request returns the right colour answer from the projector.
|
|
- Harness routing: role `smol` lands here, role `slow` on `frank-large`, and a
|
|
real subagent (the `task` role) completes a tool-using turn.
|
|
|
|
## Operate
|
|
|
|
```bash
|
|
systemctl status frank-small.service # root unit
|
|
journalctl -u frank-small.service -n 60 # load + per-request timing
|
|
curl -s http://127.0.0.1:7771/health
|
|
curl -s http://127.0.0.1:7771/v1/models
|
|
curl -s http://127.0.0.1:7771/metrics | grep -E 'spec_decode|kv_cache_usage'
|
|
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
|
```
|
|
|
|
`/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`,
|
|
which is the cheapest way to confirm MTP is running, and
|
|
`llamacpp:kv_cache_usage_ratio` for how full the slot is.
|
|
|
|
## Troubleshooting
|
|
|
|
| symptom | cause |
|
|
|---|---|
|
|
| unit restarts immediately, `frank-small: missing <path>` in the journal | the model, the projector or the binary is absent — the serve script refuses to start rather than half-load; finish the download |
|
|
| `500 … Jinja Exception: Unexpected reasoning effort <level>` | a client sent an effort the template does not accept; map it (see the client contract) |
|
|
| `500 tools param requires --jinja flag` | impossible here (`--jinja` is pinned) — you are looking at a *different* endpoint, most likely the retired ik runtime's logs |
|
|
| `finish_reason: "length"` with empty content | the budget ran out inside the thinking block; raise `max_tokens` |
|
|
| a request that waits, then answers normally | the single slot was busy (or the KV is nearly full); raise `PARALLEL` or expect serialization |
|
|
| CUDA OOM at load | another process holds the 3090, or the KV was widened past ~9.5 GiB; check `nvidia-smi` and re-derive from the sizing table |
|
|
|
|
## Rollback
|
|
|
|
`frank-small` is a new endpoint, so rollback is retirement:
|
|
`systemctl disable --now frank-small.service`, move
|
|
`/etc/systemd/system/frank-small.service` into `~/.AGENTTRASH/`, and remove the
|
|
harness provider + the `smol`/`tiny`/`task`/`vision` role entries. The old
|
|
`frank-small` (FreeToken on Dankserver, retired 2026-09-15) is a different
|
|
service with a different card — see [[freetoken-frank-small]].
|
|
|
|
## Citations
|
|
|
|
- [ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF) — the quantized checkpoints, the `-mtp` builds and the projector
|
|
- [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — base model: 64 layers, Gated DeltaNet/Gated Attention layout, 262,144 native context
|
|
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `--spec-type draft-mtp`, the MTP context type, and the Qwen3-VL projector path
|