frank-large: raise ctx to 262144 to close the harness 400-zone
The harness clamps max_tokens to min(45056, contextWindow - est - 4096), its chars/4 est under-counts the rendered prompt by ~3 % (measured: 152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at 85 % -- so with window == engine ceiling 200192, every request in the est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) + KV pool 524288; harness window 217088 = ceiling - 45k absorbs the estimate error (worst-case request ~219k <= 262144). models.yml updated out-of-repo: contextWindow 217088, maxTokens 45056.
This commit is contained in:
+24
-14
@@ -22,15 +22,25 @@
|
||||
# tool calls verified, and the native 262,144 context.
|
||||
#
|
||||
# SHAPE — the knobs that are NOT free, with the arithmetic:
|
||||
# HALOGEN_CTX=200192 is the PER-SLOT window and the per-conversation hard
|
||||
# HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard
|
||||
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
||||
# truncation. Harness compaction fires at 70% = 140,134, and 140,134 +
|
||||
# 57,344 (harness maxTokens) = 197,478 <= 200,192 keeps prompt + budget
|
||||
# inside the slot. models.yml contextWindow must quote 200192 or the
|
||||
# harness discovers the wall as a 400.
|
||||
# HALOGEN_KV_POOL_POSITIONS=400384 = KV_SLOTS(2) x CTX — the explicit
|
||||
# truncation. RAISED from 200192 on 2026-10-07: the harness clamps
|
||||
# max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts
|
||||
# the rendered prompt by ~3 % on code/agent traffic, and compaction only
|
||||
# fires at 85 % — so with window == engine ceiling, every request over
|
||||
# ~146k est (65-85 % of the window) was a guaranteed 400. Measured
|
||||
# 2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804
|
||||
# — over by 354, margin 4,096 < the 4,450 under-count.
|
||||
# Harness contract (models.yml): contextWindow 217088, maxTokens 45056.
|
||||
# Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k
|
||||
# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
|
||||
# est-error ≈ 219k <= 262144 — the 45k gap between harness window and
|
||||
# engine ceiling absorbs the estimate error with ~7x margin (robust to a
|
||||
# 10 % under-count). Compaction rides on frank-secondary: input ≈
|
||||
# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
|
||||
# HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
|
||||
# form of the engine's default, so the seam test can pin the product.
|
||||
# KV costs ~28 KiB/position (FLAGS.md) => 400,384 x 28 KiB = 10.7 GiB.
|
||||
# KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
|
||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||
# drafter is NOT slot-locked on this engine (batching and speculation
|
||||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||||
@@ -39,11 +49,11 @@
|
||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
||||
# card's own quickstart arithmetic does not bill it):
|
||||
# weights ~62 GiB resident + KV 10.7 + arena (HALOGEN_MAX_TOK=8192 =>
|
||||
# weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
|
||||
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
|
||||
# = ~82.7 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
||||
# = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
||||
# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
|
||||
# ~10-15. ~18 GiB margin. The boot ledger is the referee — read it
|
||||
# ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
|
||||
# after deploy and re-size if the engine prints a clamp.
|
||||
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
||||
# turns the MTP head off for whole requests after low acceptance and
|
||||
@@ -57,8 +67,8 @@
|
||||
# an absent temperature is no longer greedy. models.yml extraBody
|
||||
# carries the same values — consistent, not load-bearing.
|
||||
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
|
||||
# max_tokens; the harness always sends 57344. Requests over
|
||||
# HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
|
||||
# max_tokens; the harness sends at most its maxTokens 45056. Requests
|
||||
# over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
|
||||
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
|
||||
# worst case needs > 6550 s; 7200 honors it. Do not move one alone.
|
||||
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget
|
||||
@@ -121,8 +131,8 @@ FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config
|
||||
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
|
||||
# stays at the image default, which is what was qualified) ------------------
|
||||
HALOGEN_MODEL_ID=frank-large
|
||||
HALOGEN_CTX=200192
|
||||
HALOGEN_KV_POOL_POSITIONS=400384
|
||||
HALOGEN_CTX=262144
|
||||
HALOGEN_KV_POOL_POSITIONS=524288
|
||||
HALOGEN_KV_SLOTS=2
|
||||
HALOGEN_HOST_RESERVE_GIB=20
|
||||
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
|
||||
|
||||
Reference in New Issue
Block a user