dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary: 5 x full 262144 contexts at fp8 (final shape)

Browse Source 
Frank's call after the night ladder: five contexts at the model's full native
262,144 window beat six shortened ones. Six full-length contexts do NOT fit at
any KV dtype (6 x 241664 fp8 asked 10.70 GiB against 9.18 GiB free — the
6-seq graph/activation footprint leaves too little for the pool); 5 x 262144
at fp8 PASSED on the first rung: KV_BYTES 10,388,766,720/rank, measured
'GPU KV cache size: 1,350,182 tokens, Maximum concurrency for 262,144 tokens
per request: 5.15x' (39,462 tokens of margin).

Also documented: the SEQS=3 bf16 max (241,664, KV 732,359 / 3.03x) measured
earlier the same night, and the root-cause read that the 7.2 GiB KV ceiling
law was the MTP4 workspace, not VRAM. Seam and harness (contextWindow 262144)
re-pinned.
This commit is contained in:
Frank
2026-10-01 03:52:10 -07:00
parent 7cab15ac39
commit bd311bf591
6 changed files with 83 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+13 -8
View File
@@ -60,14 +60,19 @@ FRANK_PRIMARY_PUBLISH=7770:7770
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
# --- qualified engine shape (2026-10-01 night ladder, Frank's method) ---------
# MAXLEN ladder per Frank: 262144 OOMed in graph capture (76 MiB short),
# 245760 OOMed (0 bytes free at graph capture), 237568 PASSED (718,606
# tokens / 3.0x), then the +4096 bonus attempt 241664 PASSED and is
# PRODUCTION (KV 732,359 tokens / 3.03x at 241,664-token requests).
FRANK_PRIMARY_MAXLEN=241664
FRANK_PRIMARY_SEQS=3
FRANK_PRIMARY_KV_BYTES=10184000000
FRANK_PRIMARY_KV_DTYPE=auto
# SEQS 5 x MAXLEN 262144 at fp8 KV is the DEPLOYED shape (Frank's call: five
# contexts of the model's FULL native window beats six shortened ones).
# Measured: "GPU KV cache size: 1,350,182 tokens, Maximum concurrency for
# 262,144 tokens per request: 5.15x" — 39,462 tokens of margin over the
# 1,310,720-token contract.
# The night's ladder, both contracts: at SEQS=3 bf16, 262144/245760 OOMed at
# graph capture and 241664 was max possible (KV 732,359 / 3.03x). At SEQS=6
# fp8, 6 x 241664 OOMed (tried 10.70 GiB with 9.18 GiB free) — six
# full-length contexts do NOT fit on this box; five do.
FRANK_PRIMARY_MAXLEN=262144
FRANK_PRIMARY_SEQS=5
FRANK_PRIMARY_KV_BYTES=10388766720
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=1024
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64