docs(fleet): halogen owns no timing instruments; warm-cache cost measured after the pool raise
Post-deploy probe on frank-dragoon: one 101,987-token prompt through /v1/chat/completions took 81.5 s cold (1,251 t/s of prefill) and 0.3 s on two repeats — HALOGEN_CACHE_INPLACE=1 keeps the KV rows in the pool, so doubling the pool evicted nothing the prompt cache needed. The RAM price is still real: MemAvailable 26 -> 22 GiB. Census of the same engine's surface corrected the Benchmarking Policy, which quoted llama.cpp instruments at a halogen endpoint: frank-dragoon answers /health, /metrics, /v1/models and POST /v1/chat/completions and 404s /ping, /props, /completion and /tokenize, and no response carries Server-Timing. The /completion `timings` object and the ttft;dur= header are frank-zealot's; on frank-dragoon the instrument is wall clock plus usage.prompt_tokens.
This commit is contained in:
@@ -2200,10 +2200,13 @@ every restart since the cutover.
|
||||
to 200704`, and `62.9 GiB of weights locked … 11.0 GiB of KV pool, 5.7 GiB of
|
||||
working memory, 79.6 GiB in all`; prompt cache ON (12 entries × 111 MiB at
|
||||
ctx 200704 = 1.3 GiB), n-gram 47.68 GiB still `Pin::None`, no clamp line.
|
||||
- The price is exactly the FLAGS.md collision: host `MemAvailable` fell
|
||||
**26 → 22 GiB**, because every pool position is RAM the n-gram table's page
|
||||
cache loses. The warm prompt-cache ratio (136.7× measured pre-raise) was NOT
|
||||
re-measured — recorded in the runbook as the number to watch.
|
||||
- Warm reuse survived the raise, measured right after the deploy: one
|
||||
**101,987**-token wikitext prompt through `/v1/chat/completions` took
|
||||
**81.5 s** cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats —
|
||||
`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn
|
||||
prefills nothing and the doubling evicted nothing the cache needed. The RAM
|
||||
price is still real: host `MemAvailable` fell **26 → 22 GiB**, because every
|
||||
pool position is RAM the n-gram table's page cache loses.
|
||||
- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the
|
||||
engine reserves **11.0 GiB** and its preflight bills **11.3**. Re-keyed in
|
||||
AGENTS.md (topology row, KV-pool law, memory ledger), the `frank-dragoon.env`
|
||||
@@ -2226,3 +2229,11 @@ every restart since the cutover.
|
||||
200,704 → **262,144** with the 9B description; dragoon 262,144 → **200,704**,
|
||||
which had kept the pre-lowering window while its own description said
|
||||
200,704.
|
||||
- Census of the halogen HTTP surface while doing it, because the fleet bench
|
||||
law quoted the wrong engine: frank-dragoon answers **`/health`, `/metrics`,
|
||||
`/v1/models`, `/v1/chat/completions`(POST)** and 404s `/ping`, `/props`,
|
||||
`/completion`, `/tokenize`; the chat response carries **no `Server-Timing`**
|
||||
header (`server: uvicorn`, only `date`/`content-*`). So the `/completion`
|
||||
`timings` object and the `ttft;dur=` header are **llama.cpp** instruments —
|
||||
frank-zealot's, not dragoon's. AGENTS.md's Benchmarking Policy re-keyed to
|
||||
say which engine owns which instrument.
|
||||
|
||||
Reference in New Issue
Block a user