Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
213 lines
10 KiB
Markdown
213 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
title: frank-zealot on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
|
||
description: "The RETIRED frank-zealot runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-zealot-gemma4-12b]] on the same day."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- vllm
|
||
- rtx3090
|
||
- gemma
|
||
- mtp
|
||
- vision
|
||
- audio
|
||
timestamp: 2026-09-19
|
||
---
|
||
|
||
# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
|
||
|
||
> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
|
||
> back to [[frank-zealot-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
|
||
> the same day it landed — this page is the rollback target. Same port
|
||
> (7771), same alias (`frank-zealot`), same client contract surface — the
|
||
> runtime, the model payload, the drafter and the service shape (root
|
||
> systemd unit → rootless Quadlet) all changed.
|
||
|
||
## Identity
|
||
|
||
| what | value |
|
||
|---|---|
|
||
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-zealot` (`8dad46d7…`) |
|
||
| unit | **rootless Quadlet** `dankclaw/frank-zealot/quadlet/frank-zealot.container` (root systemd unit retired 2026-09-19) |
|
||
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
|
||
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
|
||
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
|
||
| port | 7771 → 8000 (alias `frank-zealot`; `frank-dragoon` keeps 7770) |
|
||
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
|
||
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
|
||
|
||
## Why vLLM
|
||
|
||
- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
|
||
kernel is its native int4 GEMM path. llama.cpp could not load it at all.
|
||
- Native audio encoder in the checkpoint — the fleet's only audio-capable
|
||
endpoint moves here (llama.cpp's path needed the mmproj audio tower).
|
||
- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
|
||
this host had no graph support.
|
||
|
||
## The drafter-Marlin bug (why the assistant is bf16)
|
||
|
||
The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
|
||
**cannot draft in vLLM**: the drafter's linear layers are loaded through a
|
||
path that does not apply the Marlin kernel, so the packed int4 weights are
|
||
misread as bf16 and every draft is garbage — measured acceptance **1.00**
|
||
(pure pass-through). There is no "Using MarlinLinearKernel" log line after
|
||
the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
|
||
has no quantization to miss, and measures **acceptance 2.80–3.56**
|
||
(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
|
||
(634 → 845 MB) is absorbed by pool rounding: both assistants yield the
|
||
identical 289,647-token KV pool.
|
||
|
||
## KV arithmetic — why two full-context sessions
|
||
|
||
Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
|
||
16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
|
||
per-token KV is dominated by the full-attention layers:
|
||
6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
|
||
289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
|
||
sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
|
||
shares the 1024-window ring across all concurrent requests, so the
|
||
marginal cost of a *second* 213k session is ~6 full-attention GiB, not
|
||
a second copy of everything.
|
||
|
||
Measured (salted, 2026-09-19):
|
||
|
||
| load | KV usage | result |
|
||
|---|---|---|
|
||
| 1 × 213k | ~35 % | runs |
|
||
| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
|
||
| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |
|
||
|
||
**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
|
||
deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
|
||
bytes, and vLLM's pool arithmetic is coarser. This is the accepted
|
||
trade for the native w4a16 format + audio + CUDA graphs.)
|
||
|
||
## Measured throughput (salted, 2026-09-19)
|
||
|
||
| probe | tokens | wall | rate |
|
||
|---|---|---|---|
|
||
| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
|
||
| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
|
||
| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
|
||
| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
|
||
| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
|
||
| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |
|
||
|
||
- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
|
||
layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
|
||
as llama.cpp's flash-attn path here.
|
||
- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
|
||
token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
|
||
right at it. At normal depths decode is far faster (the 67k-depth probe
|
||
is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
|
||
numbers.
|
||
- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
|
||
scale linearly in tokens — BPE compresses the repetition harder as the
|
||
span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
|
||
predicts). Size fillers with the model's own tokenizer
|
||
(`AutoTokenizer.encode` on the exact prefix), not char counts.
|
||
- `POST /flush_cache` is **404** in this build — salt every probe
|
||
(uuid prefix) instead.
|
||
|
||
## The image
|
||
|
||
`dankclaw/frank-zealot/Dockerfile` (repo) — 16 comment lines + `FROM
|
||
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
|
||
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
|
||
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
|
||
`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
|
||
after the driver swap, so no arch override is needed.
|
||
|
||
## No nvidia-container-toolkit — manual CUDA mounts
|
||
|
||
Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
|
||
by hand:
|
||
|
||
- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
|
||
`/dev/nvidia-uvm-tools`
|
||
- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
|
||
into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
|
||
- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`
|
||
|
||
If the driver is upgraded on the host, these mount sources move with it
|
||
(they are versionless symlinks) — no unit change needed.
|
||
|
||
## Host driver note
|
||
|
||
Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
|
||
Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
|
||
595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
|
||
the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
|
||
not a hardware mystery.
|
||
|
||
## Deployed config
|
||
|
||
Repo truth: `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
|
||
(mounted into the container at `/scripts/`, which is the entrypoint) +
|
||
`dankclaw/frank-zealot/quadlet/frank-zealot.container`. Key values:
|
||
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
|
||
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
|
||
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
|
||
`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
|
||
is **removed**. The serve script refuses to start if the weights, the
|
||
assistant, or the vllm binary are missing.
|
||
|
||
## Client contract
|
||
|
||
- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
|
||
parser), not `reasoning_content` — the harness maps it via
|
||
`compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
|
||
The field is empty when the model emits no thinking tag (behavior, not
|
||
a parser bug).
|
||
- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
|
||
accepted (200, not 400) — vLLM ignores the unknown key.
|
||
- Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
|
||
- Tools → `tool_calls` (verified `get_weather`).
|
||
- Image → 200, described correctly (native vision tower).
|
||
- Audio → 200, a 440 Hz WAV tone described correctly (native audio
|
||
encoder; verified in the derived image).
|
||
- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
|
||
`input: [text, image, audio]`.
|
||
|
||
## Boot ledger
|
||
|
||
t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
|
||
bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
|
||
KV pool 289,647 tokens). Health is 503 until ready — do not kill the
|
||
container while it loads.
|
||
|
||
## Troubleshooting
|
||
|
||
- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
|
||
(drafter-Marlin bug). Confirm the assistant mount points at
|
||
`gemma4-12b-assistant-bf16`.
|
||
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
|
||
must be absent. 8.6 is correct.
|
||
- **Audio 400/500** — the container is running the *base* image, not the
|
||
derived `:frank-zealot` (missing `av`). Check the deployed image digest
|
||
is `8dad46d7…`.
|
||
- **CUDA not visible in container** — a `Device=` line or a lib mount
|
||
dropped out of the unit after an edit; the unit is the source of truth,
|
||
the host-side generated unit is not.
|
||
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet; `podman
|
||
logs` covers only the current `--replace --rm` instance).
|
||
|
||
## Rollback
|
||
|
||
[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
|
||
mmproj and MTP sidecar are still on the host under
|
||
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
|
||
`scripts/.AGENTTRASH/frank-zealot.service.llama.cpp`. Restoring it means
|
||
reinstalling that unit and stopping the Quadlet (one owner per port).
|
||
|
||
## References
|
||
|
||
- `scripts/serve-frank-zealot.sh` — the actual exec line
|
||
- `dankclaw/frank-zealot/Dockerfile` — the audio-derived image
|
||
- `dankclaw/frank-zealot/quadlet/frank-zealot.container` — the unit
|
||
- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
|
||
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
|
||
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
|