frank-small: llama.cpp -> vLLM (gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin)
Runtime swap on the RTX 3090 (port 7771): - vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines) - google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56 - root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/); manual CUDA device+lib mounts (no nvidia-container-toolkit on host) - bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent full-context sessions, CUDA graphs on - native text+image+audio: the fleet's multimodal endpoint - measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s, tg@85k ~17.6 t/s - host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override - planner/scoping: frank-small now selected by dankclaw/frank-small/ + env/serve/download trio; old unit retired to scripts/.AGENTTRASH/ - docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index, top docs index updated; old runbook moved to retired
This commit is contained in:
Frank committed
1 parent
cf06c4eab8
commit
a2b39b179a
15 files changed
+821
-377
No files matched your search
+69
-100
@@ -1,112 +1,81 @@
|
||||
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
||||
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
||||
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
||||
# Tunables for frank-small (since 2026-09-18): vLLM serving
|
||||
# google/gemma-4-12B-it-qat-w4a16-ct (compressed-tensors w4a16 — the format
|
||||
# Google publishes "for native, optimized inference with vLLM") with the
|
||||
# google bf16 MTP assistant, on Dankclaw's RTX 3090.
|
||||
#
|
||||
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
||||
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
||||
# CUDA (see the CUDA lesson in AGENTS.md).
|
||||
# Runtime: the vllm/vllm-openai gemma-x86_64-cu129 build (vLLM
|
||||
# 0.22.1rc1.dev357+g74b5964f0; base digest in scripts/deploy-remote.sh),
|
||||
# with the vllm[audio] extra installed on top (dankclaw/frank-small/
|
||||
# Dockerfile) so the model's native audio path works. That image carries
|
||||
# the native gemma4_unified implementation, the Gemma4MTPModel drafter,
|
||||
# the gemma4 reasoning and tool-call parsers, and CUDA 12.9 for the Ada
|
||||
# card. The older gemma4-unified build (June 3) predates this checkpoint
|
||||
# and fails at weight load; the cu129-nightly build (Sep 15) fails at
|
||||
# import (torchvision ABI); this one is the image that boots.
|
||||
#
|
||||
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
||||
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
||||
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
||||
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
||||
# only a 1024-position window, which is why the KV pool is a fraction of
|
||||
# the full-ctx x all-layers number.
|
||||
# Model: 11.95B "unified" (encoder-free) — 48 layers of 5 sliding-window
|
||||
# (1024-token) + 1 full-attention, int4 group-32 weights with bf16
|
||||
# embeddings, vision and audio. The fleet's image path lives here: the model
|
||||
# ingests image (and audio) content natively, so no projector file exists
|
||||
# any more and nothing external has to load.
|
||||
#
|
||||
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
||||
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
||||
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
||||
# context loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
||||
# is the only legal shape that carries both.
|
||||
# MTP: the draft model is a SEPARATE 4-layer assistant checkpoint in the
|
||||
# same int4 w4a16 compressed-tensors format — the card's compatibility rule
|
||||
# is that a QAT target must draft with a QAT drafter at the same precision.
|
||||
# It is passed in --speculative-config with method "mtp", vLLM's Gemma 4
|
||||
# MTP path, not a generic draft model.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
||||
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
||||
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
||||
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
||||
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
||||
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
||||
# which would shrink each slot below the full 262144 window once
|
||||
# parallel > 1.
|
||||
# SIZING (RTX 3090, 24576 MiB): the goal is the model's FULL 262144 window
|
||||
# for as many concurrent sessions as the card holds.
|
||||
# MAX_MODEL_LEN=262144 the model's max_position_embeddings
|
||||
# GPU_MEM_UTIL=0.95 the card is dedicated to this service
|
||||
# KV_CACHE_DTYPE=auto bf16 KV. fp8_e4m3 would halve the KV charge
|
||||
# (289,756 -> 623,666-token pool, one -> two
|
||||
# full-context sessions), but the 3090 is
|
||||
# sm_86 (Ampere) and has no native fp8: the
|
||||
# TRITON_ATTN backend Gemma4 forces emits
|
||||
# cvt.e4m3x2 instructions that exist only from
|
||||
# sm_89 (Ada), so fp8 KV cannot boot on this
|
||||
# card (measured 2026-09-19; the "sm_89
|
||||
# patch" experiment is in .AGENTTRASH).
|
||||
# MAX_NUM_SEQS=16 the batch ceiling; the KV pool, not this,
|
||||
# caps full-context concurrency
|
||||
# MAX_NUM_BATCHED_TOKENS=16384 the prefill chunk; larger chunks prefill
|
||||
# faster but grow the activation arena that
|
||||
# competes with the KV pool for the card
|
||||
# The boot log prints the KV pool size and "Maximum concurrency for N tokens
|
||||
# per request" — for a hybrid SWA model that line is the NAIVE ratio (pool /
|
||||
# max_model_len) and undercounts: the 42 sliding-window layers only retain a
|
||||
# 1024-token window per request, so the true full-context session count is
|
||||
# higher. Verify it behaviourally (concurrent long requests) on every
|
||||
# restart, not from these comments.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
||||
# params; it carries an audio encoder too). The fleet's image path lives
|
||||
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
||||
# WORKS: input_audio content parts transcribe verbatim (verified
|
||||
# 2026-09-17 with a 16k mono WAV).
|
||||
# Sampling: greedy by default (fleet contract) via
|
||||
# --override-generation-config; a client that sends its own temperature gets
|
||||
# its own, and the harness applies per-model overrides from models.yml.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
# temperature gets its own, and the harness applies per-model overrides
|
||||
# from models.yml.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
||||
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (the
|
||||
# ROCm hosts read those bare vars — see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
||||
# FA_ALL_QUANTS=ON.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
# Container paths, bind-mounted by the quadlet. The host paths are declared
|
||||
# in the quadlet and the deploy handler, not here: this file is read inside
|
||||
# the container.
|
||||
MODEL_DIR=/models/main
|
||||
DRAFT_DIR=/models/assistant
|
||||
|
||||
# Model package. The MTP sidecar file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
||||
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
||||
# what clients ask for -- the harness entry in models.yml names it.
|
||||
# Fleet identity. 8000 is the in-container port; the quadlet publishes it as
|
||||
# host 7771, the endpoint Hermes, the harness and the homepage probe have
|
||||
# always used. The alias is the id at /v1/models and what clients ask for.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
LISTEN_PORT=8000
|
||||
|
||||
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
||||
# pool; --parallel divides it). See the SIZING note above for the fit
|
||||
# arithmetic; a fifth slot does not fit.
|
||||
CTX_SIZE=1048576
|
||||
PARALLEL=4
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
|
||||
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
|
||||
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
|
||||
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
|
||||
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
|
||||
# tried. Idle VRAM is 23,300 MiB at this geometry.
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
||||
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention (required for q8 KV).
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
||||
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
||||
# explicit here (both were the implicit build defaults: 3 and 50).
|
||||
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
|
||||
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
|
||||
# costs more attention re-read at long context than the extra accepted
|
||||
# token earns. Keep 3/50.
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
POLL_LEVEL=50
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
# Full window, dedicated card, bf16 KV (see above), roomy batch ceiling,
|
||||
# 16k prefill chunk, 4-token MTP draft (the assistant carries 4 layers).
|
||||
MAX_MODEL_LEN=262144
|
||||
GPU_MEM_UTIL=0.95
|
||||
KV_CACHE_DTYPE=auto
|
||||
MAX_NUM_SEQS=16
|
||||
MAX_NUM_BATCHED_TOKENS=16384
|
||||
SPEC_NUM_TOKENS=4
|
||||
Reference in new issue
Block a user