serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-vllm-gemma4.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)	The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant.	llm frank-small dankclaw vllm rtx3090 gemma mtp vision audio	2026-09-19

frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)

ACTIVE since 2026-09-19. Replaces frank-small-gemma4-12b (llama.cpp, now the rollback target). Same port (7771), same alias (frank-small), same client contract surface — the runtime, the model payload, the drafter and the service shape (root systemd unit → rootless Quadlet) all changed.

Identity

what	value
runtime	vLLM vllm/vllm-openai dev build gemma-x86_64-cu129 (base sha256:e24d3d8e…001e), audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (8dad46d7…)
unit	rootless Quadlet dankclaw/frank-small/quadlet/frank-small.container (root systemd unit retired 2026-09-19)
model	google/gemma-4-12B-it-qat-w4a16-ct — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B)
model dir	/home/dank/models/gemma4-12b-qat-w4a16-ct/
drafter	google/gemma-4-12B-it-assistant bf16 (845,719,296 B) at /home/dank/models/gemma4-12b-assistant-bf16/
port	7771 → 8000 (alias frank-small; frank-large keeps 7770)
window	262,144; KV pool 289,647 tokens bf16; 2 concurrent full-context sessions
multimodal	native: text + image + audio (derived image carries av/scipy/soundfile/mistral_common[audio])

Why vLLM

• The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin kernel is its native int4 GEMM path. llama.cpp could not load it at all.
• Native audio encoder in the checkpoint — the fleet's only audio-capable endpoint moves here (llama.cpp's path needed the mmproj audio tower).
• CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on this host had no graph support.

The drafter-Marlin bug (why the assistant is bf16)

The w4a16-ct assistant (kunhunjon/gemma4-12b-qat-assistant-w4a16-ct) cannot draft in vLLM: the drafter's linear layers are loaded through a path that does not apply the Marlin kernel, so the packed int4 weights are misread as bf16 and every draft is garbage — measured acceptance 1.00 (pure pass-through). There is no "Using MarlinLinearKernel" log line after the drafter loads. Google's bf16 assistant (google/gemma-4-12B-it-assistant) has no quantization to miss, and measures acceptance 2.80–3.56 (per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta (634 → 845 MB) is absorbed by pool rounding: both assistants yield the identical 289,647-token KV pool.

KV arithmetic — why two full-context sessions

Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA 16 heads × 512 dim. The SWA layers cache only the 1024-token window, so per-token KV is dominated by the full-attention layers: 6 × 16 × 512 × 2 (K,V) × 2 B = 196.6 KiB/token (bf16). 289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not sound like it fits a 24 GiB card — it fits because vLLM's SWA pool shares the 1024-window ring across all concurrent requests, so the marginal cost of a second 213k session is ~6 full-attention GiB, not a second copy of everything.

Measured (salted, 2026-09-19):

load	KV usage	result
1 × 213k	~35 %	runs
2 × 213k	69.3 %	both run in parallel (17.5 % slowdown each)
4 × 213k	—	Running: 2, Waiting: 2 — the 3rd queues

Ceiling: 2 concurrent full-context sessions. (The llama.cpp deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the bytes, and vLLM's pool arithmetic is coarser. This is the accepted trade for the native w4a16 format + audio + CUDA graphs.)

Measured throughput (salted, 2026-09-19)

probe	tokens	wall	rate
prefill @ 12.7k	12,706	27.2 s	466 t/s (degraded — concurrent traffic)
prefill @ 67k	67,252	52.0 s	1,293 t/s
prefill @ 213k	213,044	350.3 s	608 t/s
2 × 213k prefill	213,044 each	374 s batch	608 → 570 t/s per stream
decode @ 213k depth	300 out	372 s total	~13.6 t/s (bandwidth-bound)
decode @ 85k depth	300 out	82.3 s total	~17.6 t/s

• Prefill degrades with depth (1,293 → 608 t/s): the 6 full-attention layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient as llama.cpp's flash-attn path here.
• Decode at 213k is memory-bandwidth-bound: ~8.4 GB of KV read per token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is right at it. At normal depths decode is far faster (the 67k-depth probe is in the bench log). MTP acceptance ~3.2–3.6 is folded into these numbers.
• Bench gotcha (cost an hour): a repeated-paragraph filler does NOT scale linearly in tokens — BPE compresses the repetition harder as the span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit predicts). Size fillers with the model's own tokenizer (AutoTokenizer.encode on the exact prefix), not char counts.
• POST /flush_cache is 404 in this build — salt every probe (uuid prefix) instead.

The image

dankclaw/frank-small/Dockerfile (repo) — 16 comment lines + FROM localhost/vllm/vllm-openai:gemma4-unified + RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]". No COPY lines — the earlier triton-sm89-sitecustomize.py hack is gone: sm_86 (Ampere GA102) is the correct compute capability, confirmed after the driver swap, so no arch override is needed.

No nvidia-container-toolkit — manual CUDA mounts

Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA by hand:

• Device= for /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools
• host libcuda.so.1, libcuda.so, libnvidia-ml.so.1 mounted read-only into both /usr/lib/x86_64-linux-gnu/ and /usr/local/nvidia/lib64/
• NVIDIA_VISIBLE_DEVICES=all, --ulimit memlock=-1:-1

If the driver is upgraded on the host, these mount sources move with it (they are versionless symlinks) — no unit change needed.

Host driver note

Dankclaw is Ubuntu 26.04 LTS (not CachyOS — that is the dev box). Driver nvidia-driver-610 610.57.04 (installed 2026-09-19, replaces 595.91.07). nvidia-smi reports compute capability 8.6 — correct for the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact, not a hardware mystery.

Deployed config

Repo truth: scripts/frank-small.env + scripts/serve-frank-small.sh (mounted into the container at /scripts/, which is the entrypoint) + dankclaw/frank-small/quadlet/frank-small.container. Key values: KV_CACHE_DTYPE=auto (resolves to bf16 — fp8_e5m2 is refused by compressed-tensors checkpoints and fp8 is hardware-dead on sm_86), ENFORCE_EAGER=0 (CUDA graphs on), max_model_len=262144, MTP via --speculative-config pointing at the bf16 assistant. TRITON_OVERRIDE_ARCH is removed. The serve script refuses to start if the weights, the assistant, or the vllm binary are missing.

Client contract

• Reasoning lives in message.reasoning (vLLM's gemma4 reasoning parser), not reasoning_content — the harness maps it via compat.reasoningContentField: "reasoning" in ~/.omp/agent/models.yml. The field is empty when the model emits no thinking tag (behavior, not a parser bug).
• preserve_thinking: true, temperature, top_p in extraBody are accepted (200, not 400) — vLLM ignores the unknown key.
• Every reasoning_effort level accepted (all seven probed 2026-09-17).
• Tools → tool_calls (verified get_weather).
• Image → 200, described correctly (native vision tower).
• Audio → 200, a 440 Hz WAV tone described correctly (native audio encoder; verified in the derived image).
• Harness: contextWindow: 262144, maxTokens: 32768, input: [text, image, audio].

Boot ledger

t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11; KV pool 289,647 tokens). Health is 503 until ready — do not kill the container while it loads.

Troubleshooting

• Acceptance 1.00 — the drafter is the w4a16-ct assistant again (drafter-Marlin bug). Confirm the assistant mount points at gemma4-12b-assistant-bf16.
• sm_89 in an error — stale arch override; TRITON_OVERRIDE_ARCH must be absent. 8.6 is correct.
• Audio 400/500 — the container is running the base image, not the derived :frank-small (missing av). Check the deployed image digest is 8dad46d7….
• CUDA not visible in container — a Device= line or a lib mount dropped out of the unit after an edit; the unit is the source of truth, the host-side generated unit is not.
• Logs: journalctl --user -u frank-small (rootless Quadlet; podman logs covers only the current --replace --rm instance).

Rollback

frank-small-gemma4-12b — the llama.cpp deployment. Its weights, mmproj and MTP sidecar are still on the host under /home/dank/models/Gemma4-12B-QAT/, and the retired root unit is at scripts/.AGENTTRASH/frank-small.service.llama.cpp. Restoring it means reinstalling that unit and stopping the Quadlet (one owner per port).

References

• scripts/serve-frank-small.sh — the actual exec line
• dankclaw/frank-small/Dockerfile — the audio-derived image
• dankclaw/frank-small/quadlet/frank-small.container — the unit
• frank-small-gemma4-12b — RETIRED predecessor (rollback)
• halogen-flash-server — the sibling endpoint on the iGPU
• VLLM Overview / VLLM Index — runtime background