docs(fleet): halogen owns no timing instruments; warm-cache cost measured after the pool raise
Post-deploy probe on frank-dragoon: one 101,987-token prompt through /v1/chat/completions took 81.5 s cold (1,251 t/s of prefill) and 0.3 s on two repeats — HALOGEN_CACHE_INPLACE=1 keeps the KV rows in the pool, so doubling the pool evicted nothing the prompt cache needed. The RAM price is still real: MemAvailable 26 -> 22 GiB. Census of the same engine's surface corrected the Benchmarking Policy, which quoted llama.cpp instruments at a halogen endpoint: frank-dragoon answers /health, /metrics, /v1/models and POST /v1/chat/completions and 404s /ping, /props, /completion and /tokenize, and no response carries Server-Timing. The /completion `timings` object and the ttft;dur= header are frank-zealot's; on frank-dragoon the instrument is wall clock plus usage.prompt_tokens.
This commit is contained in:
1 parent
c0759492e6
commit
e7841bf9d6
3 files changed
+43
-13
No files matched your search
@@ -152,9 +152,15 @@ of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`. Prompt cache ON (12
|
||||
entries × 111 MiB at ctx 200704 = the 1.3 GiB in the KV-budget line), n-gram
|
||||
47.68 GiB still disk-backed (`Pin::None`), no clamp line. Host `MemAvailable`
|
||||
fell 26 → 22 GiB across the raise, which is the FLAGS.md collision being paid
|
||||
in page cache. **Not re-measured since the raise:** the warm prompt-cache ratio
|
||||
(136.7× on the pre-raise boot) — that is the number to watch if a long warm
|
||||
turn starts costing full prefill again.
|
||||
in page cache. **Warm reuse survived it — measured after the deploy.** One
|
||||
**101,987**-token wikitext prompt through `/v1/chat/completions` took **81.5 s**
|
||||
cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats:
|
||||
`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn prefills
|
||||
nothing and the doubling evicted nothing the cache needed. (The pre-raise
|
||||
136.7× figure was a `ttft` reading; this engine sends **no `Server-Timing`**
|
||||
header and has **no `/completion` path** — `/ping`, `/props`, `/tokenize` are
|
||||
404 too — so wall-clock around the POST plus `usage.prompt_tokens` is the only
|
||||
instrument available here.)
|
||||
|
||||
## Measured A/B — same battery, both boots (2026-10-17)
|
||||
|
||||
@@ -235,7 +241,12 @@ is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px
|
||||
probes; `response_format` unsupported; the chat template rejects
|
||||
`reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays
|
||||
mandatory); unknown sampling params are a 400; the prompt cache is on by
|
||||
default, so a warm 100k follow-up costs seconds. See
|
||||
default, so a warm 100k follow-up costs seconds (measured after the pool raise:
|
||||
**81.5 s cold vs 0.3 s warm** at 101,987 prompt tokens). The HTTP surface is
|
||||
thin — `/health`, `/metrics`, `/v1/models` and POST `/v1/chat/completions`
|
||||
answer, while `/ping`, `/props`, `/completion` and `/tokenize` are 404 and no
|
||||
response carries `Server-Timing`: llama.cpp's timing instruments do not exist
|
||||
on this engine, so measure wall-clock plus `usage.prompt_tokens`. See
|
||||
[[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that
|
||||
carries verbatim, and [[deployment]] for the deploy/health gate.
|
||||
|
||||
|
||||
Reference in new issue
Block a user