Frank
5e581666f2
int8_per_token_head keeps a separate fp32 scale tensor per (token, head); Gemma4's hybrid head dims (SWA 256 / full 512) make the per-token page bytes 4160 vs 8256, not an integer ratio, so unify_kv_cache_spec_page_size raises NotImplementedError — nine crash-loop restarts measured. TurboQuant k8v4 packs its block scales inside the tensor, keeps the 2:1 page ratio, and has a dedicated backend with an explicit Ampere path. 289,647 -> ~772k-token pool.
91 lines
5.0 KiB
Bash
91 lines
5.0 KiB
Bash
# Tunables for frank-small (since 2026-09-18): vLLM serving
|
|
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
|
|
# Google publishes "for native, optimized inference with vLLM") with the
|
|
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
|
|
#
|
|
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
|
|
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
|
|
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
|
|
# Dockerfile) so the model's native audio path works. That image carries
|
|
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
|
|
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
|
|
# card. The older gemma4-unified build (June 3) predates this checkpoint
|
|
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
|
|
# import (torchvision ABI); this one is the image that boots.
|
|
#
|
|
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
|
|
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
|
|
# embeddings, vision and audio. The fleet's image path lives here: the model
|
|
# ingests image (and audio) content natively, so no projector file exists
|
|
# any more and nothing external has to load.
|
|
#
|
|
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
|
|
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
|
|
# is that a QAT target must draft with a QAT drafter at the same precision.
|
|
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
|
|
# MTP path, not a generic draft model.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
|
|
# for as many concurrent sessions as the card holds.
|
|
# MAX_MODEL_LEN=262144 the model's max_position_embeddings
|
|
# GPU_MEM_UTIL=0.95 the card is dedicated to this service
|
|
# KV_CACHE_DTYPE=turboquant_k8v4 TurboQuant KV: K at 8-bit, V at 4-bit,
|
|
# per-16-element block scales packed INSIDE the
|
|
# tensor (Hadamard-rotated, bf16 compute,
|
|
# dedicated TURBOQUANT attention backend with
|
|
# an explicit Ampere path). 2.67x smaller KV
|
|
# than bf16 (289,647 -> ~772k-token pool, two
|
|
# -> five-or-six full-context sessions).
|
|
# The cheaper int8/fp8 per-token-head dtypes
|
|
# CANNOT boot this model: they keep a separate
|
|
# fp32 scale tensor per (token, head), and
|
|
# Gemma4's hybrid head dims (SWA 256 / full
|
|
# 512) make the per-token page bytes 4160 vs
|
|
# 8256 — not an integer ratio, so vLLM's
|
|
# unify_kv_cache_spec_page_size raises
|
|
# NotImplementedError (measured 2026-09-19,
|
|
# nine crash-loop restarts). fp8 KV is
|
|
# hardware-dead on this card anyway (sm_86,
|
|
# no cvt.e4m3x2).
|
|
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
|
|
# caps full-context concurrency
|
|
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
|
|
# faster but grow the activation arena that
|
|
# competes with the KV pool for the card
|
|
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
|
|
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
|
|
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
|
|
# 1024-token window per request, so the true full-context session count is
|
|
# higher. Verify it behaviourally (concurrent long requests) on every
|
|
# restart, not from these comments.
|
|
#
|
|
# Sampling: greedy by default (fleet contract) via
|
|
# --override-generation-config; a client that sends its own temperature gets
|
|
# its own, and the harness applies per-model overrides from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
|
|
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# Container paths, bind-mounted by the quadlet. The host paths are declared
|
|
# in the quadlet and the deploy handler, not here: this file is read inside
|
|
# the container.
|
|
MODEL_DIR=/models/main
|
|
DRAFT_DIR=/models/assistant
|
|
|
|
# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
|
|
# host 7771, the endpoint Hermes, the harness and the homepage probe have
|
|
# always used. The alias is the id at /v1/models and what clients ask for.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=8000
|
|
|
|
# Full window, dedicated card, turboquant k8v4 KV (see above), roomy batch ceiling,
|
|
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
|
|
MAX_MODEL_LEN=262144
|
|
GPU_MEM_UTIL=0.95
|
|
KV_CACHE_DTYPE=turboquant_k8v4
|
|
MAX_NUM_SEQS=16
|
|
MAX_NUM_BATCHED_TOKENS=16384
|
|
SPEC_NUM_TOKENS=4
|