dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: raise ctx to 262144 to close the harness 400-zone

Browse Source 
The harness clamps max_tokens to min(45056, contextWindow - est - 4096),
its chars/4 est under-counts the rendered prompt by ~3 % (measured:
152,938 -> 157,388 and 158,906 -> 165,301), and compaction fires only at
85 % -- so with window == engine ceiling 200192, every request in the
est ~146k-184k zone was a guaranteed 400. HALOGEN_CTX=262144 (native) +
KV pool 524288; harness window 217088 = ceiling - 45k absorbs the
estimate error (worst-case request ~219k <= 262144). models.yml updated
out-of-repo: contextWindow 217088, maxTokens 45056.
This commit is contained in:
Frank
2026-10-08 00:40:12 -07:00
parent f9330fa201
commit 76aaa2543b
4 changed files with 82 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md
+12 -9
View File
@@ -80,21 +80,21 @@ resume law throughout; no token — the repos are public).
| Env | Value | Why |
|---|---|---|
| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 |
| `HALOGEN_CTX` | 262144 | per-slot HARD ceiling (the checkpoint's native window; a longer prompt is a 400 naming the limit, never truncation). **RAISED from 200192 on 2026-10-07 — the 400-zone law:** the harness clamps `max_tokens = min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured twice: est 152,938 → server 157,388, request over room by 354; est 158,906 → server 165,301, over by 2,300), and compaction only fires at 85 % of the harness window — so with window == engine ceiling, every request in the est ~146k–184k zone was a guaranteed 400. Harness window 217,088 = ceiling − 45k absorbs the error: worst-case request ≈ 219k ≤ 262,144 (≈10 % under-count headroom) |
| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV |
| `HALOGEN_KV_POOL_POSITIONS` | 524288 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ 14.0 GiB KV |
| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published |
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
+ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with
~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
weights ~62 GiB resident + KV 14.0 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
+ vision ~2 ≈ **86 GiB engine-side**, + 20 host reserve = ~106 with
~15 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
table rides as mmap'd page cache beside the checkpoint — the card's own
quickstart arithmetic does not bill it as resident, but WATCH the ledger
at boot and re-size if the engine prints a clamp. Prompt cache is ON by
@@ -177,11 +177,14 @@ the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
## Client contract (harness)
`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
`http://192.168.0.143:7770/v1`: `contextWindow: 200192`,
`maxTokens: 57344`, `input: [text, image]`,
`http://192.168.0.143:7770/v1`: `contextWindow: 217088` — deliberately
BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
~3 % chars/4 est under-count (see the `HALOGEN_CTX` row above) —
`maxTokens: 45056`, `input: [text, image]`,
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
frank-secondary (the `compactionModel` steering is unchanged).
frank-secondary (the `compactionModel` steering is unchanged; the input
≈ 170k fits its 200,704 slot).
## Rollback