Frank
a2b39b179a
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
82 lines
4.3 KiB
Bash
82 lines
4.3 KiB
Bash
# Tunables for frank-small (since 2026-09-18): vLLM serving
|
|
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
|
|
# Google publishes "for native, optimized inference with vLLM") with the
|
|
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
|
|
#
|
|
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
|
|
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
|
|
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
|
|
# Dockerfile) so the model's native audio path works. That image carries
|
|
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
|
|
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
|
|
# card. The older gemma4-unified build (June 3) predates this checkpoint
|
|
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
|
|
# import (torchvision ABI); this one is the image that boots.
|
|
#
|
|
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
|
|
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
|
|
# embeddings, vision and audio. The fleet's image path lives here: the model
|
|
# ingests image (and audio) content natively, so no projector file exists
|
|
# any more and nothing external has to load.
|
|
#
|
|
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
|
|
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
|
|
# is that a QAT target must draft with a QAT drafter at the same precision.
|
|
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
|
|
# MTP path, not a generic draft model.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
|
|
# for as many concurrent sessions as the card holds.
|
|
# MAX_MODEL_LEN=262144 the model's max_position_embeddings
|
|
# GPU_MEM_UTIL=0.95 the card is dedicated to this service
|
|
# KV_CACHE_DTYPE=auto bf16 KV. fp8_e4m3 would halve the KV charge
|
|
# (289,756 -> 623,666-token pool, one -> two
|
|
# full-context sessions), but the 3090 is
|
|
# sm_86 (Ampere) and has no native fp8: the
|
|
# TRITON_ATTN backend Gemma4 forces emits
|
|
# cvt.e4m3x2 instructions that exist only from
|
|
# sm_89 (Ada), so fp8 KV cannot boot on this
|
|
# card (measured 2026-09-19; the "sm_89
|
|
# patch" experiment is in .AGENTTRASH).
|
|
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
|
|
# caps full-context concurrency
|
|
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
|
|
# faster but grow the activation arena that
|
|
# competes with the KV pool for the card
|
|
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
|
|
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
|
|
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
|
|
# 1024-token window per request, so the true full-context session count is
|
|
# higher. Verify it behaviourally (concurrent long requests) on every
|
|
# restart, not from these comments.
|
|
#
|
|
# Sampling: greedy by default (fleet contract) via
|
|
# --override-generation-config; a client that sends its own temperature gets
|
|
# its own, and the harness applies per-model overrides from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
|
|
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# Container paths, bind-mounted by the quadlet. The host paths are declared
|
|
# in the quadlet and the deploy handler, not here: this file is read inside
|
|
# the container.
|
|
MODEL_DIR=/models/main
|
|
DRAFT_DIR=/models/assistant
|
|
|
|
# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
|
|
# host 7771, the endpoint Hermes, the harness and the homepage probe have
|
|
# always used. The alias is the id at /v1/models and what clients ask for.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=8000
|
|
|
|
# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling,
|
|
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
|
|
MAX_MODEL_LEN=262144
|
|
GPU_MEM_UTIL=0.95
|
|
KV_CACHE_DTYPE=auto
|
|
MAX_NUM_SEQS=16
|
|
MAX_NUM_BATCHED_TOKENS=16384
|
|
SPEC_NUM_TOKENS=4
|