serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-vllm-gemma4.md

---
type: reference
title: frank-zealot on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
description: "The RETIRED frank-zealot runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-zealot-gemma4-12b]] on the same day."
tags:
  - llm
  - frank-zealot
  - dankclaw
  - vllm
  - rtx3090
  - gemma
  - mtp
  - vision
  - audio
timestamp: 2026-09-19
---

# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)

> **RETIRED 2026-09-19 (same-day revert) — rollback target only in git.**
> This vLLM deployment was cut
> back to [[frank-zealot-gemma4-12b]] (llama.cpp — itself retired
> 2026-10-01; :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]])
> the same day it landed. Same port
> (7771), same alias (`frank-small` — the name `frank-zealot` dates from
> 2026-10-06/10-09), same client contract surface — the
> runtime, the model payload, the drafter and the service shape (root
> systemd unit → rootless Quadlet) all changed.

## Identity

| what | value |
|---|---|
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46` — still in rootless storage on the 2026-10-18 `podman images` census; the tag carries the era name) |
| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (era path; deleted by the revert commit `374041c` — root systemd unit retired 2026-09-19) |
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` (absent from the 2026-10-18 census of `/home/dank/models/` — re-fetch to use this rollback) |
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` (absent from the same census) |
| port | 7771 → 8000 (alias `frank-small` in this era; `frank-dragoon` keeps 7770) |
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |

## Why vLLM

- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
  kernel is its native int4 GEMM path. llama.cpp could not load it at all.
- Native audio encoder in the checkpoint — the fleet's only audio-capable
  endpoint moves here (llama.cpp's path needed the mmproj audio tower).
- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
  this host had no graph support.

## The drafter-Marlin bug (why the assistant is bf16)

The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
**cannot draft in vLLM**: the drafter's linear layers are loaded through a
path that does not apply the Marlin kernel, so the packed int4 weights are
misread as bf16 and every draft is garbage — measured acceptance **1.00**
(pure pass-through). There is no "Using MarlinLinearKernel" log line after
the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
has no quantization to miss, and measures **acceptance 2.80–3.56**
(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
(634 → 845 MB) is absorbed by pool rounding: both assistants yield the
identical 289,647-token KV pool.

## KV arithmetic — why two full-context sessions

Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
per-token KV is dominated by the full-attention layers:
6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
shares the 1024-window ring across all concurrent requests, so the
marginal cost of a *second* 213k session is ~6 full-attention GiB, not
a second copy of everything.

Measured (salted, 2026-09-19):

| load | KV usage | result |
|---|---|---|
| 1 × 213k | ~35 % | runs |
| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |

**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
bytes, and vLLM's pool arithmetic is coarser. This is the accepted
trade for the native w4a16 format + audio + CUDA graphs.)

## Measured throughput (salted, 2026-09-19)

| probe | tokens | wall | rate |
|---|---|---|---|
| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |

- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
  layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
  as llama.cpp's flash-attn path here.
- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
  token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
  right at it. At normal depths decode is far faster (the 67k-depth probe
  is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
  numbers.
- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
  scale linearly in tokens — BPE compresses the repetition harder as the
  span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
  predicts). Size fillers with the model's own tokenizer
  (`AutoTokenizer.encode` on the exact prefix), not char counts.
- `POST /flush_cache` is **404** in this build — salt every probe
  (uuid prefix) instead.

## The image

`dankclaw/frank-small/Dockerfile` (era path, deleted by the revert commit
`374041c`; recover with `git show 374041c^:dankclaw/frank-small/Dockerfile`) — 16 comment lines + `FROM
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
after the driver swap, so no arch override is needed.

## No nvidia-container-toolkit — manual CUDA mounts

Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
by hand:

- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
  `/dev/nvidia-uvm-tools`
- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
  into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`

If the driver is upgraded on the host, these mount sources move with it
(they are versionless symlinks) — no unit change needed.

## Host driver note

Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
not a hardware mystery.

## Deployed config

Repo truth **for this era** (the files carry llama.cpp knobs today): `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
(mounted into the container at `/scripts/`, which is the entrypoint) +
`dankclaw/frank-small/quadlet/frank-small.container` (era path, deleted at `374041c`). Key values:
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
is **removed**. The serve script refuses to start if the weights, the
assistant, or the vllm binary are missing.

## Client contract

- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
  parser), not `reasoning_content` — the harness maps it via
  `compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
  The field is empty when the model emits no thinking tag (behavior, not
  a parser bug).
- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
  accepted (200, not 400) — vLLM ignores the unknown key.
- Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
- Tools → `tool_calls` (verified `get_weather`).
- Image → 200, described correctly (native vision tower).
- Audio → 200, a 440 Hz WAV tone described correctly (native audio
  encoder; verified in the derived image).
- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
  `input: [text, image, audio]` — era values; the endpoint reads 200,704 /
  16,384 / `[text, image]` today (`~/.omp/agent/models.yml:206,212,214`), and
  audio left the fleet with this payload on 2026-10-01.

## Boot ledger

t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
KV pool 289,647 tokens). Health is 503 until ready — do not kill the
container while it loads.

## Troubleshooting

- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
  (drafter-Marlin bug). Confirm the assistant mount points at
  `gemma4-12b-assistant-bf16`.
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
  must be absent. 8.6 is correct.
- **Audio 400/500** — the container is running the *base* image, not the
  derived `:frank-small` (missing `av`). In this era, check the deployed image
  id is `8dad46d7…` (`podman images`, 2026-10-18: it still exists, unused —
  nothing on :7771 runs in podman today).
- **CUDA not visible in container** — a `Device=` line or a lib mount
  dropped out of the unit after an edit; the unit is the source of truth,
  the host-side generated unit is not.
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet in this era;
  `podman logs` covers only the current `--replace --rm` instance). The unit is
  a ROOT unit today: `journalctl -u frank-zealot`.

## Rollback

[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
mmproj and MTP sidecar are **no longer on the host** under
`/home/dank/models/Gemma4-12B-QAT/` (2026-10-18 census: the directory is
gone — it was staged to `.AGENTTRASH/20261002-retired-llm-payloads/` and that
stage dir is gone too), so this rollback re-fetches
`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` and restores the era
trio from git (`git show 374041c:scripts/frank-small.service`,
`…:scripts/frank-small.env`, `…:scripts/serve-frank-small.sh`); the era quadlet
lived at `dankclaw/frank-small/quadlet/frank-small.container`. One owner per
port: the llama.cpp unit is root, so stop the container unit before starting it.

## References

- `scripts/serve-frank-zealot.sh` — the actual exec line (today: llama.cpp; the
  vLLM exec line is in `git show 374041c^:scripts/serve-frank-small.sh`)
- `dankclaw/frank-small/Dockerfile` — the audio-derived image (era path, deleted at `374041c`)
- `dankclaw/frank-small/quadlet/frank-small.container` — the unit (era path, deleted at `374041c`)
- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[VLLM Overview]] / [[VLLM Index]] — runtime background