frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
This commit is contained in:
Frank
@@ -0,0 +1,211 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||||
description: The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant.
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- vllm
|
||||
- rtx3090
|
||||
- gemma
|
||||
- mtp
|
||||
- vision
|
||||
- audio
|
||||
timestamp: 2026-09-19
|
||||
---
|
||||
|
||||
# frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||||
|
||||
> **ACTIVE since 2026-09-19.** Replaces [[frank-small-gemma4-12b]]
|
||||
> (llama.cpp, now the rollback target). Same port (7771), same alias
|
||||
> (`frank-small`), same client contract surface — the runtime, the model
|
||||
> payload, the drafter and the service shape (root systemd unit → rootless
|
||||
> Quadlet) all changed.
|
||||
|
||||
## Identity
|
||||
|
||||
| what | value |
|
||||
|---|---|
|
||||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d7…`) |
|
||||
| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (root systemd unit retired 2026-09-19) |
|
||||
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
|
||||
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
|
||||
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
|
||||
| port | 7771 → 8000 (alias `frank-small`; `frank-large` keeps 7770) |
|
||||
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
|
||||
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
|
||||
|
||||
## Why vLLM
|
||||
|
||||
- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
|
||||
kernel is its native int4 GEMM path. llama.cpp could not load it at all.
|
||||
- Native audio encoder in the checkpoint — the fleet's only audio-capable
|
||||
endpoint moves here (llama.cpp's path needed the mmproj audio tower).
|
||||
- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
|
||||
this host had no graph support.
|
||||
|
||||
## The drafter-Marlin bug (why the assistant is bf16)
|
||||
|
||||
The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
|
||||
**cannot draft in vLLM**: the drafter's linear layers are loaded through a
|
||||
path that does not apply the Marlin kernel, so the packed int4 weights are
|
||||
misread as bf16 and every draft is garbage — measured acceptance **1.00**
|
||||
(pure pass-through). There is no "Using MarlinLinearKernel" log line after
|
||||
the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
|
||||
has no quantization to miss, and measures **acceptance 2.80–3.56**
|
||||
(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
|
||||
(634 → 845 MB) is absorbed by pool rounding: both assistants yield the
|
||||
identical 289,647-token KV pool.
|
||||
|
||||
## KV arithmetic — why two full-context sessions
|
||||
|
||||
Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
|
||||
16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
|
||||
per-token KV is dominated by the full-attention layers:
|
||||
6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
|
||||
289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
|
||||
sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
|
||||
shares the 1024-window ring across all concurrent requests, so the
|
||||
marginal cost of a *second* 213k session is ~6 full-attention GiB, not
|
||||
a second copy of everything.
|
||||
|
||||
Measured (salted, 2026-09-19):
|
||||
|
||||
| load | KV usage | result |
|
||||
|---|---|---|
|
||||
| 1 × 213k | ~35 % | runs |
|
||||
| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
|
||||
| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |
|
||||
|
||||
**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
|
||||
deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
|
||||
bytes, and vLLM's pool arithmetic is coarser. This is the accepted
|
||||
trade for the native w4a16 format + audio + CUDA graphs.)
|
||||
|
||||
## Measured throughput (salted, 2026-09-19)
|
||||
|
||||
| probe | tokens | wall | rate |
|
||||
|---|---|---|---|
|
||||
| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
|
||||
| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
|
||||
| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
|
||||
| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
|
||||
| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
|
||||
| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |
|
||||
|
||||
- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
|
||||
layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
|
||||
as llama.cpp's flash-attn path here.
|
||||
- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
|
||||
token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
|
||||
right at it. At normal depths decode is far faster (the 67k-depth probe
|
||||
is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
|
||||
numbers.
|
||||
- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
|
||||
scale linearly in tokens — BPE compresses the repetition harder as the
|
||||
span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
|
||||
predicts). Size fillers with the model's own tokenizer
|
||||
(`AutoTokenizer.encode` on the exact prefix), not char counts.
|
||||
- `POST /flush_cache` is **404** in this build — salt every probe
|
||||
(uuid prefix) instead.
|
||||
|
||||
## The image
|
||||
|
||||
`dankclaw/frank-small/Dockerfile` (repo) — 16 comment lines + `FROM
|
||||
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
|
||||
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
|
||||
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
|
||||
`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
|
||||
after the driver swap, so no arch override is needed.
|
||||
|
||||
## No nvidia-container-toolkit — manual CUDA mounts
|
||||
|
||||
Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
|
||||
by hand:
|
||||
|
||||
- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
|
||||
`/dev/nvidia-uvm-tools`
|
||||
- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
|
||||
into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
|
||||
- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`
|
||||
|
||||
If the driver is upgraded on the host, these mount sources move with it
|
||||
(they are versionless symlinks) — no unit change needed.
|
||||
|
||||
## Host driver note
|
||||
|
||||
Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
|
||||
Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
|
||||
595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
|
||||
the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
|
||||
not a hardware mystery.
|
||||
|
||||
## Deployed config
|
||||
|
||||
Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`
|
||||
(mounted into the container at `/scripts/`, which is the entrypoint) +
|
||||
`dankclaw/frank-small/quadlet/frank-small.container`. Key values:
|
||||
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
|
||||
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
|
||||
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
|
||||
`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
|
||||
is **removed**. The serve script refuses to start if the weights, the
|
||||
assistant, or the vllm binary are missing.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
|
||||
parser), not `reasoning_content` — the harness maps it via
|
||||
`compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
|
||||
The field is empty when the model emits no thinking tag (behavior, not
|
||||
a parser bug).
|
||||
- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
|
||||
accepted (200, not 400) — vLLM ignores the unknown key.
|
||||
- Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
|
||||
- Tools → `tool_calls` (verified `get_weather`).
|
||||
- Image → 200, described correctly (native vision tower).
|
||||
- Audio → 200, a 440 Hz WAV tone described correctly (native audio
|
||||
encoder; verified in the derived image).
|
||||
- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
|
||||
`input: [text, image, audio]`.
|
||||
|
||||
## Boot ledger
|
||||
|
||||
t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
|
||||
bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
|
||||
KV pool 289,647 tokens). Health is 503 until ready — do not kill the
|
||||
container while it loads.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
|
||||
(drafter-Marlin bug). Confirm the assistant mount points at
|
||||
`gemma4-12b-assistant-bf16`.
|
||||
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
|
||||
must be absent. 8.6 is correct.
|
||||
- **Audio 400/500** — the container is running the *base* image, not the
|
||||
derived `:frank-small` (missing `av`). Check the deployed image digest
|
||||
is `8dad46d7…`.
|
||||
- **CUDA not visible in container** — a `Device=` line or a lib mount
|
||||
dropped out of the unit after an edit; the unit is the source of truth,
|
||||
the host-side generated unit is not.
|
||||
- Logs: `journalctl --user -u frank-small` (rootless Quadlet; `podman
|
||||
logs` covers only the current `--replace --rm` instance).
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||||
mmproj and MTP sidecar are still on the host under
|
||||
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
|
||||
`scripts/.AGENTTRASH/frank-small.service.llama.cpp`. Restoring it means
|
||||
reinstalling that unit and stopping the Quadlet (one owner per port).
|
||||
|
||||
## References
|
||||
|
||||
- `scripts/serve-frank-small.sh` — the actual exec line
|
||||
- `dankclaw/frank-small/Dockerfile` — the audio-derived image
|
||||
- `dankclaw/frank-small/quadlet/frank-small.container` — the unit
|
||||
- [[frank-small-gemma4-12b]] — RETIRED predecessor (rollback)
|
||||
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
|
||||
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
|
||||
@@ -5,9 +5,9 @@ Documentation for the LLM inference fleet on the two hosts (2026-09-16):
|
||||
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
|
||||
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
|
||||
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
|
||||
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
|
||||
7771).
|
||||
w4b on the gfx1151 iGPU, port 7770) and vLLM (frank-small,
|
||||
google/gemma-4-12B-it-qat-w4a16-ct with the bf16 MTP assistant on the RTX
|
||||
3090, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -35,10 +35,10 @@ gfx1151 iGPU 64 GiB unified memory]
|
||||
frank-large, port 7770]
|
||||
F[peonist-ai Qwen3.8-Flash-Next w4b
|
||||
2 x 200192 ctx over one 400384-position pool]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
G[vLLM gemma-x86_64-cu129
|
||||
frank-small, port 7771]
|
||||
H[Qwen3.8-27B GSQ-RCO IQ3_S
|
||||
131072 ctx / 1 slot, MTP + vision]
|
||||
H[gemma-4-12B-it-qat-w4a16-ct
|
||||
262144 ctx / 2 full-context sessions, MTP + vision + audio]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -87,11 +87,15 @@ frank-small, port 7771]
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB
|
||||
- [[frank-small-vllm-gemma4]] — ACTIVE frank-small runbook since 2026-09-19
|
||||
(rootless Quadlet, port 7771): vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-token
|
||||
pool, 2 concurrent full-context sessions, native text+image+audio
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target): the
|
||||
prior frank-small runbook (2026-09-17 → 09-19), mainline llama.cpp serving
|
||||
HauhauCS Gemma4-12B-QAT-Uncensored Q4_K_M with the MTP sidecar, q8_0 KV,
|
||||
4 slots of 262k
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
|
||||
@@ -166,11 +166,17 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
|
||||
2026-09-17: mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
|
||||
- [[frank-small-vllm-gemma4]] — the ACTIVE frank-small runbook since
|
||||
2026-09-19: vLLM serving
|
||||
google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16, Marlin)
|
||||
with Google's bf16 MTP assistant (bf16 KV, 289,647-token pool, 2
|
||||
full-context sessions, native text+image+audio) on
|
||||
Dankclaw's RTX 3090, port 7771
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-19 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
(MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262k) on
|
||||
Dankclaw's RTX 3090
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
|
||||
the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
|
||||
|
||||
Reference in New Issue
Block a user