Frank
8d88b3a0a9
fp8 KV is hardware-dead on the 3090 (sm_86, no cvt.e4m3x2), so int8 is the cheapest quant the TRITON_ATTN backend can run: pure ALU dequant, same backend, no fp8 tensor cores needed. 289,647 -> ~579k-token pool, two -> four full-context sessions. Quality verified against the bf16 baseline (needle-in-haystack at 213k + short battery) before the swap.
88 lines
4.8 KiB
Bash
88 lines
4.8 KiB
Bash
# Tunables for frank-small (since 2026-09-18): vLLM serving
|
|
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
|
|
# Google publishes "for native, optimized inference with vLLM") with the
|
|
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
|
|
#
|
|
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
|
|
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
|
|
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
|
|
# Dockerfile) so the model's native audio path works. That image carries
|
|
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
|
|
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
|
|
# card. The older gemma4-unified build (June 3) predates this checkpoint
|
|
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
|
|
# import (torchvision ABI); this one is the image that boots.
|
|
#
|
|
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
|
|
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
|
|
# embeddings, vision and audio. The fleet's image path lives here: the model
|
|
# ingests image (and audio) content natively, so no projector file exists
|
|
# any more and nothing external has to load.
|
|
#
|
|
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
|
|
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
|
|
# is that a QAT target must draft with a QAT drafter at the same precision.
|
|
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
|
|
# MTP path, not a generic draft model.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
|
|
# for as many concurrent sessions as the card holds.
|
|
# MAX_MODEL_LEN=262144 the model's max_position_embeddings
|
|
# GPU_MEM_UTIL=0.95 the card is dedicated to this service
|
|
# KV_CACHE_DTYPE=int8_per_token_head int8 KV (per-token, per-head scales)
|
|
# in the same TRITON_ATTN backend — pure ALU
|
|
# dequant, no fp8 hardware needed, so it runs
|
|
# on sm_86. fp8_e4m3 is the cheaper option but
|
|
# the 3090 is sm_86 (Ampere) and has no native
|
|
# fp8: the TRITON_ATTN backend Gemma4 forces
|
|
# emits cvt.e4m3x2 instructions that exist only
|
|
# from sm_89 (Ada), so fp8 KV cannot boot on
|
|
# this card (measured 2026-09-19; the "sm_89
|
|
# patch" experiment is in .AGENTTRASH). int8
|
|
# halves the KV charge (289,647 -> ~579k-token
|
|
# pool, two -> four full-context sessions) at
|
|
# the cost of a dequant in the attention path;
|
|
# quality verified against the bf16 baseline
|
|
# (needle + short battery) on 2026-09-19.
|
|
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
|
|
# caps full-context concurrency
|
|
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
|
|
# faster but grow the activation arena that
|
|
# competes with the KV pool for the card
|
|
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
|
|
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
|
|
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
|
|
# 1024-token window per request, so the true full-context session count is
|
|
# higher. Verify it behaviourally (concurrent long requests) on every
|
|
# restart, not from these comments.
|
|
#
|
|
# Sampling: greedy by default (fleet contract) via
|
|
# --override-generation-config; a client that sends its own temperature gets
|
|
# its own, and the harness applies per-model overrides from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
|
|
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# Container paths, bind-mounted by the quadlet. The host paths are declared
|
|
# in the quadlet and the deploy handler, not here: this file is read inside
|
|
# the container.
|
|
MODEL_DIR=/models/main
|
|
DRAFT_DIR=/models/assistant
|
|
|
|
# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
|
|
# host 7771, the endpoint Hermes, the harness and the homepage probe have
|
|
# always used. The alias is the id at /v1/models and what clients ask for.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=8000
|
|
|
|
# Full window, dedicated card, int8 KV (see above), roomy batch ceiling,
|
|
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
|
|
MAX_MODEL_LEN=262144
|
|
GPU_MEM_UTIL=0.95
|
|
KV_CACHE_DTYPE=int8_per_token_head
|
|
MAX_NUM_SEQS=16
|
|
MAX_NUM_BATCHED_TOKENS=16384
|
|
SPEC_NUM_TOKENS=4
|