frank-primary: KV pool snapped to the qsa_sparse page-64 multiple (299968)
Measured first boot: the auto attention backend is qsa_sparse, which
overrides page size to 64 and refuses --num-tokens 300000 ('nearest valid
values: 299968 or 300032'). 299,968 x 24 KiB = 7.0 GiB still clears the
177,356-token harness worst case and one native 262,144 conversation.
This commit is contained in:
+10
-12
@@ -37,15 +37,8 @@
|
||||
# (shrinking needs a restart); `ft ctl stats` reports pool occupancy.
|
||||
# - KV MATH: 12 full-attention layers x 2 KV heads x 256 head_dim x 2 (K+V)
|
||||
# x 2 B = 24 KiB/token (f16 KV; the 36 GDN layers use fixed per-seq state,
|
||||
# not per-token KV). FRANK_PRIMARY_FT_NUM_TOKENS=300000 -> ~7.2 GiB.
|
||||
# Harness worst case = 140,492-token prompt + 36,864 max_tokens = 177,356
|
||||
# < 300,000; one NATIVE 262,144-token conversation + full budget also fits.
|
||||
# - PLE: the ~47.7 GiB ngram-embedding table pins in HOST RAM by default
|
||||
# (--ple-backend disk is the alternative); the unit's LimitMEMLOCK=infinity
|
||||
# makes the pin legal. TENS of GiB of pinned pages are why this stays a
|
||||
# ROOT unit — see frank-primary.service.
|
||||
|
||||
# --- Identity (deploy + serve contract) --------------------------------
|
||||
# not per-token KV). See FRANK_PRIMARY_FT_NUM_TOKENS below for the pool
|
||||
# size and its measured page-size law.
|
||||
FRANK_PRIMARY_IMAGE=localhost/frank-primary-freetoken:pr587-77a7201
|
||||
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
||||
FRANK_PRIMARY_MODEL_SHARDS=131
|
||||
@@ -62,9 +55,14 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# the in-container index 0 IS the right card; --gpu 0 selects it.
|
||||
FRANK_PRIMARY_FT_GPU=0
|
||||
# KV pool in TOKENS (not bytes — vLLM-era FRANK_PRIMARY_KV_BYTES is gone).
|
||||
# 300,000 tokens x 24 KiB = 7.2 GiB; clears the harness worst case 177,356.
|
||||
# Grows live with `ft ctl cache --kv <n>`; shrink needs a restart.
|
||||
FRANK_PRIMARY_FT_NUM_TOKENS=300000
|
||||
# 299,968 tokens x 24 KiB = 7.0 GiB; clears the harness worst case
|
||||
# (177,356 tokens) and one NATIVE 262,144-token conversation + full budget.
|
||||
# MEASURED FIRST BOOT (2026-10-05): the auto attention backend is
|
||||
# qsa_sparse, which OVERRIDES page size to 64, so --num-tokens must be a
|
||||
# MULTIPLE OF 64 — 300,000 was refused ("nearest valid values: 299968 or
|
||||
# 300032"). Grows live with `ft ctl cache --kv <n>`; shrink needs a
|
||||
# restart.
|
||||
FRANK_PRIMARY_FT_NUM_TOKENS=299968
|
||||
# Concurrent decode slots. 2 is the first-boot conservative value; the
|
||||
# radix pool admits ~2 full-length 200,704-token conversations. Raise with
|
||||
# --max-running-requests only after a real-traffic look at slot occupancy.
|
||||
|
||||
Reference in New Issue
Block a user