# Tunables for frank-small (since 2026-09-18): vLLM serving
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
# Google publishes "for native, optimized inference with vLLM") with the
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
#
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
# Dockerfile) so the model's native audio path works. That image carries
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
# card. The older gemma4-unified build (June 3) predates this checkpoint
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
# import (torchvision ABI); this one is the image that boots.
#
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
# embeddings, vision and audio. The fleet's image path lives here: the model
# ingests image (and audio) content natively, so no projector file exists
# any more and nothing external has to load.
#
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
# is that a QAT target must draft with a QAT drafter at the same precision.
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
# MTP path, not a generic draft model.
#
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
# for as many concurrent sessions as the card holds.
#   MAX_MODEL_LEN=262144      the model's max_position_embeddings
#   GPU_MEM_UTIL=0.95         the card is dedicated to this service
#   KV_CACHE_DTYPE=auto       bf16 KV. fp8_e4m3 would halve the KV charge
#                             (289,756 -> 623,666-token pool, one -> two
#                             full-context sessions), but the 3090 is
#                             sm_86 (Ampere) and has no native fp8: the
#                             TRITON_ATTN backend Gemma4 forces emits
#                             cvt.e4m3x2 instructions that exist only from
#                             sm_89 (Ada), so fp8 KV cannot boot on this
#                             card (measured 2026-09-19; the "sm_89
#                             patch" experiment is in .AGENTTRASH).
#   MAX_NUM_SEQS=16           the batch ceiling; the KV pool, not this,
#                             caps full-context concurrency
#   MAX_NUM_BATCHED_TOKENS=16384  the prefill chunk; larger chunks prefill
#                             faster but grow the activation arena that
#                             competes with the KV pool for the card
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
# 1024-token window per request, so the true full-context session count is
# higher. Verify it behaviourally (concurrent long requests) on every
# restart, not from these comments.
#
# Sampling: greedy by default (fleet contract) via
# --override-generation-config; a client that sends its own temperature gets
# its own, and the harness applies per-model overrides from models.yml.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.

# Container paths, bind-mounted by the quadlet. The host paths are declared
# in the quadlet and the deploy handler, not here: this file is read inside
# the container.
MODEL_DIR=/models/main
DRAFT_DIR=/models/assistant

# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
# host 7771, the endpoint Hermes, the harness and the homepage probe have
# always used. The alias is the id at /v1/models and what clients ask for.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=8000

# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling,
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
MAX_MODEL_LEN=262144
GPU_MEM_UTIL=0.95
KV_CACHE_DTYPE=auto
MAX_NUM_SEQS=16
MAX_NUM_BATCHED_TOKENS=16384
SPEC_NUM_TOKENS=4