dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9902267f85182b4c3364b996ad224dc0d0ce4dfb
serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md
T
Frank 49e8dfff66 docs: the vault and AGENTS.md describe two Dankclaw endpoints 
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.

AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.

Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
2026-09-16 14:52:51 -07:00

205 lines
10 KiB
Markdown
Raw Blame History

---
type: reference
title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S
description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves.
tags:
- llm
- frank-small
- dankclaw
- llama.cpp
- rtx3090
- qwen35
- mtp
- vision
timestamp: 2026-09-16
---
# frank-small — llama.cpp on the RTX 3090
> **ACTIVE since 2026-09-16.** Created when Dankclaw's inference was re-split:
> [[halogen-flash-server]] took the gfx1151 iGPU back for `frank-large`, which
> freed the RTX 3090 for this endpoint. Two endpoints, one host: `frank-large`
> on `:7770`, `frank-small` on `:7771`.
## Identity
| what | value |
|---|---|
| runtime | mainline llama.cpp master `37b53fd` (2026-09-16), CUDA build for sm_86 |
| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
| flags | `scripts/frank-small.env`; entrypoint `scripts/serve-frank-small.sh` |
| endpoint | `192.168.0.143:7771`, alias `frank-small` |
| model | `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`, the `IQ3_S-mtp` build + BF16 mmproj |
It is a root unit rather than a Quadlet because the runtime is a native CUDA
binary, not a container: there is no image to install and no generated unit to
avoid. Boot persistence is the unit's `[Install] WantedBy=multi-user.target`;
`Restart=on-failure` / `RestartSec=10` make it self-healing; `LimitMEMLOCK=infinity`
covers CUDA's pinned staging buffers.
Build recipe (the deployed binary):
```bash
cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build-cuda --target llama-server -j
```
`GGML_CUDA_FA_ALL_QUANTS` is deliberately **off**: it is only needed for
*quantized* KV with flash attention (see the CUDA lesson in AGENTS.md — without
it, quantized-KV prefill collapses ~12x), and this service runs f16 KV.
## The model, and why the `-mtp` build specifically
GSQ-RCO is a **per-tensor mixed-precision** allocation produced by a
gradient-based search under a total size budget, rather than one quantization
type applied throughout. IQ3_S at 3.50 bpw is the release's task-lossless
operating point: AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
GPQA-Diamond is 0.51 behind — at 11.8 GB against 53.8 GB.
Two files are required, and the second suffix matters:
| file | size | why |
|---|---|---|
| `Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` | 12,120,016,960 B | the `-mtp` build **is** the MTP build: it carries the head as `blk.64.nextn.*` (`qwen35.nextn_predict_layers = 1`) |
| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,146,528 B | the vision encoder + projector (`clip.projector_type = qwen3vl_merger`, 27 blocks) |
Both live in `/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/`, fetched with
`huggingface_hub` (which verifies the LFS sha256 and resumes).
**MTP is a property of the file, not of the flags.** `--spec-type draft-mtp`
sets `load_mtp`, and the draft context runs on the *main model's* weights
(`common/common.cpp` passes `shares_model` when no `--model-draft` is given;
`common/speculative.cpp` gives it `LLAMA_CONTEXT_TYPE_MTP`). So there is no
second copy of a 12 GiB model in VRAM, and `--model-draft` would be wrong here —
it would load the whole file again.
## Architecture and the KV arithmetic
GGUF arch `qwen35`: 64 layers laid out as `16 x (3 x (Gated DeltaNet -> FFN) +
1 x (Gated Attention -> FFN))`. Only the 16 full-attention layers take KV, so a
token costs
```
2 (K,V) x 4 heads x 256 head_dim x 2 B = 64 KiB at f16 (34 KiB q8_0, 18 KiB q4_0)
```
and the 48 DeltaNet layers add ~0.16 GiB per sequence of constant-size state in
f32, at any depth. The MTP draft context is `n_ctx` long over one nextn layer,
so it costs 4 KiB/token of its own.
## Sizing (RTX 3090, 24576 MiB)
| term | GiB |
|---|---|
| weights | 11.29 |
| mmproj | 0.87 |
| MTP draft KV (131,072 x 4 KiB) | 0.50 |
| DeltaNet state (one sequence) | 0.16 |
| CUDA context + prefill/decode buffers | ~1.5 |
| **KV, one slot at 131,072 x 64 KiB** | **8.00** |
| **modelled total** | **~22.3 of 24** |
The binding number is the KV: about 9.5 GiB is left after the fixed terms, which
is ~155k tokens at 64 KiB/token. Alternatives inside the same budget: `2 x
65536` at f16, or `4 x 65536` / `1 x 262144` at q8_0 — the quantized variants
need a rebuild with `-DGGML_CUDA_FA_ALL_QUANTS=ON`. **`--parallel N` divides
`--ctx-size` between the slots**; it does not multiply.
Measured at deploy: **22,362 MiB** in use of 24,576, `n_slots = 1`,
`n_ctx_slot = 131072`.
## Deployed flags
| flag | value | why |
|---|---|---|
| `CTX_SIZE` / `PARALLEL` | `131072` / `1` | one conversation per slot, full f16 precision, no KV quantization risk |
| `CACHE_TYPE_K/V` | `f16` | quantizing would need the FA_ALL_QUANTS build for no benefit here |
| `NGL` | `999` | every layer on the card |
| `FLASH_ATTN` | `on` | |
| `SPEC_TYPE` | `draft-mtp` | the model's own head; draft depth stays llama.cpp's default of 3 |
| `BATCH_SIZE` / `UBATCH_SIZE` | `2048` / `512` | the defaults; ubatch 512 keeps the DeltaNet chunk buffers small |
| `NUM_THREADS` / `THREAD_BATCH` | `8` / `8` | the work is on the GPU; the host CPU belongs to halogen's engine |
| `TEMPERATURE` | `0.0` | greedy default (fleet contract) — clients that send their own values get them |
| `REASONING_FORMAT` / `REASONING_BUDGET` | `deepseek` / `-1` | thinking into `reasoning_content`, unlimited budget (the model decides) |
| `--jinja` | pinned | the GGUF's own template supplies `reasoning_effort`, `preserve_thinking` and the tool-call grammar. This build defaults it on; pinning it means a future default change cannot silently take tools away |
| `--metrics` | on | Prometheus scrapes `/metrics` here (`frank-small` job) |
## Client contract
- **One slot.** A second concurrent request queues; it is not served in
parallel, and agent fan-out through this endpoint serializes. Widen with
`PARALLEL` (which divides the context) if that becomes the wrong trade.
- **Thinking counts against the budget.** A request whose budget runs out
mid-thought returns `finish_reason: "length"` with an empty `content` and the
partial reasoning in `reasoning_content` — observed live on a vision request
at `max_tokens: 64`. Give reasoning models a real budget.
- **Tools work** (`tool_calls` returned; verified with a function-call round
trip) because `--jinja` serves the model's own template. This is also what
fixed subagents fleet-wide: the ik runtime that served `frank-large` until
2026-09-16 rejected every request carrying `tools` with
`500 tools param requires --jinja flag`.
- **Images are accepted** as OpenAI `image_url` content parts with a `data:`
URL or bare base64. This is the fleet's only image path: halogen's
`frank-large` runs with no tower and refuses images with a named reason.
- **`reasoning_effort` accepts only `xhigh`, `medium`, `low`.** The template
raises on anything else — a 500, not a clamp — so the harness folds its own
levels onto that set in `compat.reasoningEffortMap` (`high: xhigh`,
`max: xhigh`). An unmapped level fails on the first request.
- `/v1/models` reports `frank-small`; `/health` answers 200 once the model is
loaded and the projector is up.
## Verified at deploy (2026-09-16)
- `systemctl is-enabled/is-active frank-small.service` → `enabled` / `active`.
- `/v1/models` → `frank-small`; a chat completion returns `content` **and**
`reasoning_content`.
- A tool call returns `tool_calls` with the expected arguments.
- Slot timing lines show `draft acceptance = 0.66-0.94, mean len = 2.95-3.83`
— MTP is engaged, not merely configured.
- An image request returns the right colour answer from the projector.
- Harness routing: role `smol` lands here, role `slow` on `frank-large`, and a
real subagent (the `task` role) completes a tool-using turn.
## Operate
```bash
systemctl status frank-small.service # root unit
journalctl -u frank-small.service -n 60 # load + per-request timing
curl -s http://127.0.0.1:7771/health
curl -s http://127.0.0.1:7771/v1/models
curl -s http://127.0.0.1:7771/metrics | grep -E 'spec_decode|kv_cache_usage'
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
```
`/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`,
which is the cheapest way to confirm MTP is running, and
`llamacpp:kv_cache_usage_ratio` for how full the slot is.
## Troubleshooting
| symptom | cause |
|---|---|
| unit restarts immediately, `frank-small: missing <path>` in the journal | the model, the projector or the binary is absent — the serve script refuses to start rather than half-load; finish the download |
| `500 … Jinja Exception: Unexpected reasoning effort <level>` | a client sent an effort the template does not accept; map it (see the client contract) |
| `500 tools param requires --jinja flag` | impossible here (`--jinja` is pinned) — you are looking at a *different* endpoint, most likely the retired ik runtime's logs |
| `finish_reason: "length"` with empty content | the budget ran out inside the thinking block; raise `max_tokens` |
| a request that waits, then answers normally | the single slot was busy (or the KV is nearly full); raise `PARALLEL` or expect serialization |
| CUDA OOM at load | another process holds the 3090, or the KV was widened past ~9.5 GiB; check `nvidia-smi` and re-derive from the sizing table |
## Rollback
`frank-small` is a new endpoint, so rollback is retirement:
`systemctl disable --now frank-small.service`, move
`/etc/systemd/system/frank-small.service` into `~/.AGENTTRASH/`, and remove the
harness provider + the `smol`/`tiny`/`task`/`vision` role entries. The old
`frank-small` (FreeToken on Dankserver, retired 2026-09-15) is a different
service with a different card — see [[freetoken-frank-small]].
## Citations
- [ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF) — the quantized checkpoints, the `-mtp` builds and the projector
- [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — base model: 64 layers, Gated DeltaNet/Gated Attention layout, 262,144 native context
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `--spec-type draft-mtp`, the MTP context type, and the Qwen3-VL projector path
Reference in new issue View Git Blame Copy Permalink