dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: raise ctx to 262144 to close the harness 400-zone

Browse Source 
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
This commit is contained in:
Frank
2026-10-08 00:40:12 -07:00
parent f9330fa201
commit 76aaa2543b
4 changed files with 82 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+18 -11
View File
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -367,15 +367,22 @@ endpoint).
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400
naming the limit, never truncation); `models.yml` `contextWindow: 200192`
must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits.
for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
under the 400-zone law: the harness clamps max_tokens to
`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
request 43,158 vs room 42,804), so a harness window == engine ceiling
made every request over ~146k est a guaranteed 400. `models.yml`
`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
law does NOT carry over; batching and speculation coexist (the card's
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒
~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
@@ -391,8 +398,8 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content +
"video".
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB
weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
page cache beside the checkpoint — the card's quickstart arithmetic does
not bill it as resident, but READ the boot ledger and re-size if the
@@ -469,7 +476,7 @@ OpenAI `image_url` content part reaches the model.
**Compaction hub**: `models.yml` routes `compactionModel` for
`frank-primary`, `frank-large` and `frank-secondary` here — 200,704 is the
only window in the fleet that fits a worst-case compaction input (~133k
only window in the fleet that fits a worst-case compaction input (~170k
tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law).
@@ -1002,7 +1009,7 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |