dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: raise ctx to 262144 to close the harness 400-zone

Browse Source 
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
This commit is contained in:
Frank
2026-10-08 00:40:12 -07:00
parent f9330fa201
commit 76aaa2543b
4 changed files with 82 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+18 -11
View File
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port | | Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------| |--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 | | Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 | | Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -367,15 +367,22 @@ endpoint).
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400 for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
naming the limit, never truncation); `models.yml` `contextWindow: 200192` naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits. under the 400-zone law: the harness clamps max_tokens to
`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
request 43,158 vs room 42,804), so a harness window == engine ceiling
made every request over ~146k est a guaranteed 400. `models.yml`
`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter" `HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
law does NOT carry over; batching and speculation coexist (the card's law does NOT carry over; batching and speculation coexist (the card's
quickstart ships 2 slots with the head), so 2 is concurrent capacity. quickstart ships 2 slots with the head), so 2 is concurrent capacity.
`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒ `HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the 14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` + harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table). `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` = **A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no 1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
@@ -391,8 +398,8 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content +
"video". "video".
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): **Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB ~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
page cache beside the checkpoint — the card's quickstart arithmetic does page cache beside the checkpoint — the card's quickstart arithmetic does
not bill it as resident, but READ the boot ledger and re-size if the not bill it as resident, but READ the boot ledger and re-size if the
@@ -469,7 +476,7 @@ OpenAI `image_url` content part reaches the model.
**Compaction hub**: `models.yml` routes `compactionModel` for **Compaction hub**: `models.yml` routes `compactionModel` for
`frank-primary`, `frank-large` and `frank-secondary` here — 200,704 is the `frank-primary`, `frank-large` and `frank-secondary` here — 200,704 is the
only window in the fleet that fits a worst-case compaction input (~133k only window in the fleet that fits a worst-case compaction input (~170k
tokens). With `--parallel 2` a compaction request no longer serializes behind tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law). a live generation (the Gemma4 era's `--parallel 1` law).
@@ -1002,7 +1009,7 @@ Active:
| Service | Runbook | | Service | Runbook |
|---------|---------| |---------|---------|
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` | | frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` | | frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
scripts/frank-large.env
+24 -14
View File
@@ -22,15 +22,25 @@
# tool calls verified, and the native 262,144 context. # tool calls verified, and the native 262,144 context.
# #
# SHAPE — the knobs that are NOT free, with the arithmetic: # SHAPE — the knobs that are NOT free, with the arithmetic:
# HALOGEN_CTX=200192 is the PER-SLOT window and the per-conversation hard # HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard
# ceiling — a prompt over it is a 400 naming the limit, never a silent # ceiling — a prompt over it is a 400 naming the limit, never a silent
# truncation. Harness compaction fires at 70% = 140,134, and 140,134 + # truncation. RAISED from 200192 on 2026-10-07: the harness clamps
# 57,344 (harness maxTokens) = 197,478 <= 200,192 keeps prompt + budget # max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts
# inside the slot. models.yml contextWindow must quote 200192 or the # the rendered prompt by ~3 % on code/agent traffic, and compaction only
# harness discovers the wall as a 400. # fires at 85 % — so with window == engine ceiling, every request over
# HALOGEN_KV_POOL_POSITIONS=400384 = KV_SLOTS(2) x CTX — the explicit # ~146k est (65-85 % of the window) was a guaranteed 400. Measured
# 2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804
# — over by 354, margin 4,096 < the 4,450 under-count.
# Harness contract (models.yml): contextWindow 217088, maxTokens 45056.
# Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k
# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
# est-error ≈ 219k <= 262144 — the 45k gap between harness window and
# engine ceiling absorbs the estimate error with ~7x margin (robust to a
# 10 % under-count). Compaction rides on frank-secondary: input ≈
# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
# HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
# form of the engine's default, so the seam test can pin the product. # form of the engine's default, so the seam test can pin the product.
# KV costs ~28 KiB/position (FLAGS.md) => 400,384 x 28 KiB = 10.7 GiB. # KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP # HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
# drafter is NOT slot-locked on this engine (batching and speculation # drafter is NOT slot-locked on this engine (batching and speculation
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own # coexist — DFlash-era slots>1 law does NOT carry over; the card's own
@@ -39,11 +49,11 @@
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is # Memory ledger against the box's 121.2 GiB unified (the n-gram table is
# mmap'd page cache beside the checkpoint, not a pinned charge — the # mmap'd page cache beside the checkpoint, not a pinned charge — the
# card's own quickstart arithmetic does not bill it): # card's own quickstart arithmetic does not bill it):
# weights ~62 GiB resident + KV 10.7 + arena (HALOGEN_MAX_TOK=8192 => # weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2 # ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
# = ~82.7 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host # = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ) # honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
# ~10-15. ~18 GiB margin. The boot ledger is the referee — read it # ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
# after deploy and re-size if the engine prints a clamp. # after deploy and re-size if the engine prints a clamp.
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy # HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
# turns the MTP head off for whole requests after low acceptance and # turns the MTP head off for whole requests after low acceptance and
@@ -57,8 +67,8 @@
# an absent temperature is no longer greedy. models.yml extraBody # an absent temperature is no longer greedy. models.yml extraBody
# carries the same values — consistent, not load-bearing. # carries the same values — consistent, not load-bearing.
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit # * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
# max_tokens; the harness always sends 57344. Requests over # max_tokens; the harness sends at most its maxTokens 45056. Requests
# HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and # over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial # queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
# worst case needs > 6550 s; 7200 honors it. Do not move one alone. # worst case needs > 6550 s; 7200 honors it. Do not move one alone.
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget # * max_tokens INCLUDES thinking; a reasoning reply over a small budget
@@ -121,8 +131,8 @@ FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset # --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
# stays at the image default, which is what was qualified) ------------------ # stays at the image default, which is what was qualified) ------------------
HALOGEN_MODEL_ID=frank-large HALOGEN_MODEL_ID=frank-large
HALOGEN_CTX=200192 HALOGEN_CTX=262144
HALOGEN_KV_POOL_POSITIONS=400384 HALOGEN_KV_POOL_POSITIONS=524288
HALOGEN_KV_SLOTS=2 HALOGEN_KV_SLOTS=2
HALOGEN_HOST_RESERVE_GIB=20 HALOGEN_HOST_RESERVE_GIB=20
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD); # Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md
+12 -9
View File
@@ -80,21 +80,21 @@ resume law throughout; no token — the repos are public).
| Env | Value | Why | | Env | Value | Why |
|---|---|---| |---|---|---|
| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 | | `HALOGEN_CTX` | 262144 | per-slot HARD ceiling (the checkpoint's native window; a longer prompt is a 400 naming the limit, never truncation). **RAISED from 200192 on 2026-10-07 — the 400-zone law:** the harness clamps `max_tokens = min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured twice: est 152,938 → server 157,388, request over room by 354; est 158,906 → server 165,301, over by 2,300), and compaction only fires at 85 % of the harness window — so with window == engine ceiling, every request in the est ~146k–184k zone was a guaranteed 400. Harness window 217,088 = ceiling − 45k absorbs the error: worst-case request ≈ 219k ≤ 262,144 (≈10 % under-count headroom) |
| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity | | `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV | | `HALOGEN_KV_POOL_POSITIONS` | 524288 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ 14.0 GiB KV |
| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) | | `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade | | `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) | | `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) | | `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) | | `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published | | `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published |
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose | | MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger): **Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`) weights ~62 GiB resident + KV 14.0 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
+ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with + vision ~2 ≈ **86 GiB engine-side**, + 20 host reserve = ~106 with
~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram ~15 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
table rides as mmap'd page cache beside the checkpoint — the card's own table rides as mmap'd page cache beside the checkpoint — the card's own
quickstart arithmetic does not bill it as resident, but WATCH the ledger quickstart arithmetic does not bill it as resident, but WATCH the ledger
at boot and re-size if the engine prints a clamp. Prompt cache is ON by at boot and re-size if the engine prints a clamp. Prompt cache is ON by
@@ -177,11 +177,14 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
## Client contract (harness) ## Client contract (harness)
`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @ `~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
`http://192.168.0.143:7770/v1`: `contextWindow: 200192`, `http://192.168.0.143:7770/v1`: `contextWindow: 217088` — deliberately
`maxTokens: 57344`, `input: [text, image]`, BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
~3 % chars/4 est under-count (see the `HALOGEN_CTX` row above) —
`maxTokens: 45056`, `input: [text, image]`,
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort `reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
frank-secondary (the `compactionModel` steering is unchanged). frank-secondary (the `compactionModel` steering is unchanged; the input
≈ 170k fits its 200,704 slot).
## Rollback ## Rollback
tests/frank_large_unit.bats
+28 -15
View File
@@ -2,7 +2,7 @@
# Seam: frank-large serves the Quat3rnion ABLITERATED Swift1.5 finetune of # Seam: frank-large serves the Quat3rnion ABLITERATED Swift1.5 finetune of
# peonist's Qwen3.8-Flash-Next v2 checkpoint (a .hgn flash-server payload) # peonist's Qwen3.8-Flash-Next v2 checkpoint (a .hgn flash-server payload)
# on the Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2, # on the Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2,
# digest-pinned), driven by the root unit's serve script — two 200,192-token # digest-pinned), driven by the root unit's serve script — two 262,144-token
# slots with the MTP head always on (HALOGEN_SPEC_ADAPT=0) and the vision # slots with the MTP head always on (HALOGEN_SPEC_ADAPT=0) and the vision
# tower mounted. 2026-10-06 cutover from the halogen 27B-dense shape. # tower mounted. 2026-10-06 cutover from the halogen 27B-dense shape.
# #
@@ -80,8 +80,8 @@ good_env() {
"FRANK_LARGE_VISION_BYTES=0" "FRANK_LARGE_VISION_BYTES=0"
"FRANK_LARGE_TOKENIZER_FILES=tok.json" "FRANK_LARGE_TOKENIZER_FILES=tok.json"
"HALOGEN_MODEL_ID=frank-large" "HALOGEN_MODEL_ID=frank-large"
"HALOGEN_CTX=200192" "HALOGEN_CTX=262144"
"HALOGEN_KV_POOL_POSITIONS=400384" "HALOGEN_KV_POOL_POSITIONS=524288"
"HALOGEN_KV_SLOTS=2" "HALOGEN_KV_SLOTS=2"
"HALOGEN_SPEC_ADAPT=0" "HALOGEN_SPEC_ADAPT=0"
"HALOGEN_MAX_TOK=8192" "HALOGEN_MAX_TOK=8192"
@@ -263,8 +263,8 @@ $(cat "$TEST_TMP/serve.log")"
[[ "$output" == *"--name frank-large-halogen"* ]] [[ "$output" == *"--name frank-large-halogen"* ]]
[[ "$output" == *"--publish 7770:8731"* ]] [[ "$output" == *"--publish 7770:8731"* ]]
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-large"* ]] [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-large"* ]]
[[ "$output" == *"-e HALOGEN_CTX=200192"* ]] [[ "$output" == *"-e HALOGEN_CTX=262144"* ]]
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=400384"* ]] [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=524288"* ]]
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]] [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]] [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
[[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]] [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
@@ -280,16 +280,29 @@ $(cat "$TEST_TMP/serve.log")"
! grep -q 'PODMAN wait' "$TEST_TMP/serve.log" ! grep -q 'PODMAN wait' "$TEST_TMP/serve.log"
} }
@test "the shipped env pins the fleet's window and shape" { @test "the shipped env keeps the harness fit law inside the ceiling" {
# 200192 is the per-slot hard ceiling (a longer prompt is a 400 # 262144 (the checkpoint's native window) is the per-slot hard ceiling
# naming the limit, never a silent truncation), and the harness # (a longer prompt is a 400 naming the limit, never a silent
# compaction math must fit prompt + budget under it: OMP compacts at # truncation). The harness clamps max_tokens = min(45056, 217088 − est
# 70% = 140134, and 140134 + 57344 = 197478 <= 200192. models.yml # − 4096), compacts at 85 % of its declared 217088 window, and its
# contextWindow must quote the same number or the harness discovers # chars/4 est UNDER-counts the rendered prompt by ~3 % on code/agent
# the wall as a 400. # traffic (measured 4,450 tokens at est 152,938 on 2026-10-07 — with
grep -qE '^HALOGEN_CTX=200192$' "$ENV_FILE" # window == ceiling, every request over ~146k est was a guaranteed
ctx=200192 # 400 before compaction could fire at 85 %). The 45k gap between the
[ $(( ctx * 70 / 100 + 57344 )) -le "$ctx" ] # harness window and the engine ceiling must absorb that error at
# both monotone maxima of est + budget (est ≤ 217088 − 4096):
# full budget at the compaction threshold: 1.03*184524 + 45056
# clamped budget at the clamp's zero point: 1.03*212992 + 0
# and the compaction request that rides on frank-secondary (its
# 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k).
grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE"
ctx=262144
hw=217088; mt=45056
thr=$(( hw * 85 / 100 ))
[ $(( thr * 103 / 100 + mt )) -le "$ctx" ]
zero=$(( hw - 4096 ))
[ $(( zero * 103 / 100 )) -le "$ctx" ]
[ $(( (thr - 20000) * 103 / 100 )) -le 200704 ]
} }
@test "no HIP-sensitive bare names leak through the env file" { @test "no HIP-sensitive bare names leak through the env file" {