--- type: reference title: frank-secondary on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors) description: "The RETIRED frank-secondary runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-secondary-gemma4-12b]] on the same day." tags: - llm - frank-secondary - dankclaw - vllm - rtx3090 - gemma - mtp - vision - audio timestamp: 2026-09-19 --- # frank-secondary — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16) > **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut > back to [[frank-secondary-gemma4-12b]] (llama.cpp, now the ACTIVE runbook) > the same day it landed — this page is the rollback target. Same port > (7771), same alias (`frank-secondary`), same client contract surface — the > runtime, the model payload, the drafter and the service shape (root > systemd unit → rootless Quadlet) all changed. ## Identity | what | value | |---|---| | runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-secondary` (`8dad46d7…`) | | unit | **rootless Quadlet** `dankclaw/frank-secondary/quadlet/frank-secondary.container` (root systemd unit retired 2026-09-19) | | model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) | | model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` | | drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` | | port | 7771 → 8000 (alias `frank-secondary`; `frank-large` keeps 7770) | | window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** | | multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) | ## Why vLLM - The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin kernel is its native int4 GEMM path. llama.cpp could not load it at all. - Native audio encoder in the checkpoint — the fleet's only audio-capable endpoint moves here (llama.cpp's path needed the mmproj audio tower). - CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on this host had no graph support. ## The drafter-Marlin bug (why the assistant is bf16) The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`) **cannot draft in vLLM**: the drafter's linear layers are loaded through a path that does not apply the Marlin kernel, so the packed int4 weights are misread as bf16 and every draft is garbage — measured acceptance **1.00** (pure pass-through). There is no "Using MarlinLinearKernel" log line after the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`) has no quantization to miss, and measures **acceptance 2.80–3.56** (per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta (634 → 845 MB) is absorbed by pool rounding: both assistants yield the identical 289,647-token KV pool. ## KV arithmetic — why two full-context sessions Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA 16 heads × 512 dim. The SWA layers cache only the 1024-token window, so per-token KV is dominated by the full-attention layers: 6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16). 289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not sound like it fits a 24 GiB card — it fits because vLLM's SWA pool shares the 1024-window ring across all concurrent requests, so the marginal cost of a *second* 213k session is ~6 full-attention GiB, not a second copy of everything. Measured (salted, 2026-09-19): | load | KV usage | result | |---|---|---| | 1 × 213k | ~35 % | runs | | 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) | | 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues | **Ceiling: 2 concurrent full-context sessions.** (The llama.cpp deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the bytes, and vLLM's pool arithmetic is coarser. This is the accepted trade for the native w4a16 format + audio + CUDA graphs.) ## Measured throughput (salted, 2026-09-19) | probe | tokens | wall | rate | |---|---|---|---| | prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) | | prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** | | prefill @ 213k | 213,044 | 350.3 s | **608 t/s** | | 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream | | decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) | | decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** | - **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient as llama.cpp's flash-attn path here. - **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is right at it. At normal depths decode is far faster (the 67k-depth probe is in the bench log). MTP acceptance ~3.2–3.6 is folded into these numbers. - **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT scale linearly in tokens — BPE compresses the repetition harder as the span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit predicts). Size fillers with the model's own tokenizer (`AutoTokenizer.encode` on the exact prefix), not char counts. - `POST /flush_cache` is **404** in this build — salt every probe (uuid prefix) instead. ## The image `dankclaw/frank-secondary/Dockerfile` (repo) — 16 comment lines + `FROM localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install --no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone: `sm_86` (Ampere GA102) is the *correct* compute capability, confirmed after the driver swap, so no arch override is needed. ## No nvidia-container-toolkit — manual CUDA mounts Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA by hand: - `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, `/dev/nvidia-uvm-tools` - host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/` - `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1` If the driver is upgraded on the host, these mount sources move with it (they are versionless symlinks) — no unit change needed. ## Host driver note Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box). Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces 595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact, not a hardware mystery. ## Deployed config Repo truth: `scripts/frank-secondary.env` + `scripts/serve-frank-secondary.sh` (mounted into the container at `/scripts/`, which is the entrypoint) + `dankclaw/frank-secondary/quadlet/frank-secondary.container`. Key values: `KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by compressed-tensors checkpoints and fp8 is hardware-dead on sm_86), `ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via `--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH` is **removed**. The serve script refuses to start if the weights, the assistant, or the vllm binary are missing. ## Client contract - **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning parser), not `reasoning_content` — the harness maps it via `compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`. The field is empty when the model emits no thinking tag (behavior, not a parser bug). - `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are accepted (200, not 400) — vLLM ignores the unknown key. - Every `reasoning_effort` level accepted (all seven probed 2026-09-17). - Tools → `tool_calls` (verified `get_weather`). - Image → 200, described correctly (native vision tower). - Audio → 200, a 440 Hz WAV tone described correctly (native audio encoder; verified in the derived image). - Harness: `contextWindow: 262144`, `maxTokens: 32768`, `input: [text, image, audio]`. ## Boot ledger t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11; KV pool 289,647 tokens). Health is 503 until ready — do not kill the container while it loads. ## Troubleshooting - **Acceptance 1.00** — the drafter is the w4a16-ct assistant again (drafter-Marlin bug). Confirm the assistant mount points at `gemma4-12b-assistant-bf16`. - **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH` must be absent. 8.6 is correct. - **Audio 400/500** — the container is running the *base* image, not the derived `:frank-secondary` (missing `av`). Check the deployed image digest is `8dad46d7…`. - **CUDA not visible in container** — a `Device=` line or a lib mount dropped out of the unit after an edit; the unit is the source of truth, the host-side generated unit is not. - Logs: `journalctl --user -u frank-secondary` (rootless Quadlet; `podman logs` covers only the current `--replace --rm` instance). ## Rollback [[frank-secondary-gemma4-12b]] — the llama.cpp deployment. Its weights, mmproj and MTP sidecar are still on the host under `/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at `scripts/.AGENTTRASH/frank-secondary.service.llama.cpp`. Restoring it means reinstalling that unit and stopping the Quadlet (one owner per port). ## References - `scripts/serve-frank-secondary.sh` — the actual exec line - `dankclaw/frank-secondary/Dockerfile` — the audio-derived image - `dankclaw/frank-secondary/quadlet/frank-secondary.container` — the unit - [[frank-secondary-gemma4-12b]] — RETIRED predecessor (rollback) - [[halogen-flash-server]] — the sibling endpoint on the iGPU - [[VLLM Overview]] / [[VLLM Index]] — runtime background