# Tunables for frank-small (since 2026-09-18): vLLM serving
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
# Google publishes "for native, optimized inference with vLLM") with the
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
#
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
# Dockerfile) so the model's native audio path works. That image carries
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
# card. The older gemma4-unified build (June 3) predates this checkpoint
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
# import (torchvision ABI); this one is the image that boots.
#
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
# embeddings, vision and audio. The fleet's image path lives here: the model
# ingests image (and audio) content natively, so no projector file exists
# any more and nothing external has to load.
#
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
# is that a QAT target must draft with a QAT drafter at the same precision.
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
# MTP path, not a generic draft model.
#
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
# for as many concurrent sessions as the card holds.
#   MAX_MODEL_LEN=262144      the model's max_position_embeddings
#   GPU_MEM_UTIL=0.95         the card is dedicated to this service
#   KV_CACHE_DTYPE=turboquant_k8v4   TurboQuant KV: K at 8-bit, V at 4-bit,
#                             per-16-element block scales packed INSIDE the
#                             tensor (Hadamard-rotated, bf16 compute,
#                             dedicated TURBOQUANT attention backend with
#                             an explicit Ampere path). 2.67x smaller KV
#                             than bf16 (289,647 -> ~772k-token pool, two
#                             -> five-or-six full-context sessions).
#                             The cheaper int8/fp8 per-token-head dtypes
#                             CANNOT boot this model: they keep a separate
#                             fp32 scale tensor per (token, head), and
#                             Gemma4's hybrid head dims (SWA 256 / full
#                             512) make the per-token page bytes 4160 vs
#                             8256 — not an integer ratio, so vLLM's
#                             unify_kv_cache_spec_page_size raises
#                             NotImplementedError (measured 2026-09-19,
#                             nine crash-loop restarts). fp8 KV is
#                             hardware-dead on this card anyway (sm_86,
#                             no cvt.e4m3x2).
#   MAX_NUM_SEQS=16           the batch ceiling; the KV pool, not this,
#                             caps full-context concurrency
#   MAX_NUM_BATCHED_TOKENS=16384  the prefill chunk; larger chunks prefill
#                             faster but grow the activation arena that
#                             competes with the KV pool for the card
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
# 1024-token window per request, so the true full-context session count is
# higher. Verify it behaviourally (concurrent long requests) on every
# restart, not from these comments.
#
# Sampling: greedy by default (fleet contract) via
# --override-generation-config; a client that sends its own temperature gets
# its own, and the harness applies per-model overrides from models.yml.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.

# Container paths, bind-mounted by the quadlet. The host paths are declared
# in the quadlet and the deploy handler, not here: this file is read inside
# the container.
MODEL_DIR=/models/main
DRAFT_DIR=/models/assistant

# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
# host 7771, the endpoint Hermes, the harness and the homepage probe have
# always used. The alias is the id at /v1/models and what clients ask for.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=8000

# Full window, dedicated card, turboquant k8v4 KV (see above), roomy batch ceiling,
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
MAX_MODEL_LEN=262144
GPU_MEM_UTIL=0.95
KV_CACHE_DTYPE=turboquant_k8v4
# ATTENTION_BACKEND: Gemma4's config hook force-selects TRITON_ATTN for the
# mixed 256/512 head dims whenever no backend is selected, and TRITON_ATTN
# rejects the turboquant KV dtype (measured: ValueError "kv_cache_dtype not
# supported", crash loop). Pin the TURBOQUANT backend explicitly via the
# --attention-backend flag (this vLLM build removed the VLLM_ATTENTION_
# BACKEND env var — it warns "Unknown vLLM environment variable" and
# ignores it). The TURBOQUANT backend accepts any head size and the
# turboquant_* dtypes, and the Gemma4 MTP speculator carries the target's
# selected backend through to the draft layers, so the drafter follows.
ATTENTION_BACKEND=TURBOQUANT
MAX_NUM_SEQS=16
MAX_NUM_BATCHED_TOKENS=16384
SPEC_NUM_TOKENS=4