frank-large: halogen container cutover, rootless rlimit law, and the DFlash2 drafter shape
This commit is contained in:
1 parent
b0b840b149
commit
be840b4881
4 files changed
+73
-9
No files matched your search
@@ -48,7 +48,7 @@ bearing numbers:
|
||||
- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
|
||||
pair (FLAGS.md table): a request at the cap must not outlive the timeout.
|
||||
65536 > the harness's 57,344 `maxTokens` so an eager request never meets
|
||||
the engine's 413.
|
||||
the engine's 400.
|
||||
- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
|
||||
pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
|
||||
set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
|
||||
@@ -56,17 +56,23 @@ bearing numbers:
|
||||
that ledger, not a formula.
|
||||
- **No server-side sampler:** an absent temperature is GREEDY. The card's
|
||||
thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
|
||||
— on the wire only.
|
||||
— on the wire only. The template defaults `reasoning_effort` to **xhigh**
|
||||
when a request sends none, and **thinking tokens consume the
|
||||
`max_tokens` budget**: a budget exhausted mid-thought returns
|
||||
`finish_reason: "length"` with EMPTY `content` and the partial reasoning
|
||||
in `reasoning_content` — clients must read both fields.
|
||||
|
||||
## Client contract deltas vs llama.cpp
|
||||
|
||||
|Behavior|halogen|
|
||||
|---|---|
|
||||
|Prompt over slot|400 naming the limit (no truncation)|
|
||||
|`max_tokens` over cap|413, max_tokens over queue timeout|503s for the queue|
|
||||
|`max_tokens` over cap|400 naming the limit (never silent truncation)|
|
||||
|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
|
||||
|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
|
||||
|`response_format`|NOT supported (no constrained decoding)|
|
||||
|thinking field|`reasoning_content` (same as before)|
|
||||
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays|
|
||||
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|
||||
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|
||||
|image input|ignored/error — route to frank-small|
|
||||
|
||||
|
||||
Reference in new issue
Block a user