frank-large: halogen container cutover, rootless rlimit law, and the DFlash2 drafter shape

This commit is contained in:
Frank committed 2026-10-05 17:35:05 -07:00
1 parent b0b840b149
commit be840b4881
4 files changed
+73 -9

No files matched your search

@@ -48,7 +48,7 @@ bearing numbers:
- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
pair (FLAGS.md table): a request at the cap must not outlive the timeout.
65536 > the harness's 57,344 `maxTokens` so an eager request never meets
the engine's 413.
the engine's 400.
- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
@@ -56,17 +56,23 @@ bearing numbers:
that ledger, not a formula.
- **No server-side sampler:** an absent temperature is GREEDY. The card's
thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
— on the wire only.
— on the wire only. The template defaults `reasoning_effort` to **xhigh**
when a request sends none, and **thinking tokens consume the
`max_tokens` budget**: a budget exhausted mid-thought returns
`finish_reason: "length"` with EMPTY `content` and the partial reasoning
in `reasoning_content` — clients must read both fields.
## Client contract deltas vs llama.cpp
|Behavior|halogen|
|---|---|
|Prompt over slot|400 naming the limit (no truncation)|
|`max_tokens` over cap|413, max_tokens over queue timeout|503s for the queue|
|`max_tokens` over cap|400 naming the limit (never silent truncation)|
|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
|`response_format`|NOT supported (no constrained decoding)|
|thinking field|`reasoning_content` (same as before)|
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays|
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|image input|ignored/error — route to frank-small|