frank-primary: KV pool snapped to the qsa_sparse page-64 multiple (299968)

Measured first boot: the auto attention backend is qsa_sparse, which
overrides page size to 64 and refuses --num-tokens 300000 ('nearest valid
values: 299968 or 300032'). 299,968 x 24 KiB = 7.0 GiB still clears the
177,356-token harness worst case and one native 262,144 conversation.
This commit is contained in:
Frank
2026-10-05 19:19:24 -07:00
parent bd5c28c4e9
commit 0ff016a6b2
+10 -12
View File
@@ -37,15 +37,8 @@
# (shrinking needs a restart); `ft ctl stats` reports pool occupancy.
# - KV MATH: 12 full-attention layers x 2 KV heads x 256 head_dim x 2 (K+V)
# x 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state,
# not per-token KV). FRANK_PRIMARY_FT_NUM_TOKENS=300000 -> ~7.2 GiB.
# Harness worst case = 140,492-token prompt + 36,864 max_tokens = 177,356
# < 300,000; one NATIVE 262,144-token conversation + full budget also fits.
# - PLE: the ~47.7 GiB ngram-embedding table pins in HOST RAM by default
# (--ple-backend disk is the alternative); the unit's LimitMEMLOCK=infinity
# makes the pin legal. TENS of GiB of pinned pages are why this stays a
# ROOT unit — see frank-primary.service.
# --- Identity (deploy + serve contract) --------------------------------
# not per-token KV). See FRANK_PRIMARY_FT_NUM_TOKENS below for the pool
# size and its measured page-size law.
FRANK_PRIMARY_IMAGE=localhost/frank-primary-freetoken:pr587-77a7201
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
FRANK_PRIMARY_MODEL_SHARDS=131
@@ -62,9 +55,14 @@ FRANK_PRIMARY_PUBLISH=7770:7770
# the in-container index 0 IS the right card; --gpu 0 selects it.
FRANK_PRIMARY_FT_GPU=0
# KV pool in TOKENS (not bytes — vLLM-era FRANK_PRIMARY_KV_BYTES is gone).
# 300,000 tokens x 24 KiB = 7.2 GiB; clears the harness worst case 177,356.
# Grows live with `ft ctl cache --kv <n>`; shrink needs a restart.
FRANK_PRIMARY_FT_NUM_TOKENS=300000
# 299,968 tokens x 24 KiB = 7.0 GiB; clears the harness worst case
# (177,356 tokens) and one NATIVE 262,144-token conversation + full budget.
# MEASURED FIRST BOOT (2026-10-05): the auto attention backend is
# qsa_sparse, which OVERRIDES page size to 64, so --num-tokens must be a
# MULTIPLE OF 64 — 300,000 was refused ("nearest valid values: 299968 or
# 300032"). Grows live with `ft ctl cache --kv <n>`; shrink needs a
# restart.
FRANK_PRIMARY_FT_NUM_TOKENS=299968
# Concurrent decode slots. 2 is the first-boot conservative value; the
# radix pool admits ~2 full-length 200,704-token conversations. Raise with
# --max-running-requests only after a real-traffic look at slot occupancy.