docs(fleet): halogen owns no timing instruments; warm-cache cost measured after the pool raise

Post-deploy probe on frank-dragoon: one 101,987-token prompt through
/v1/chat/completions took 81.5 s cold (1,251 t/s of prefill) and 0.3 s on two
repeats — HALOGEN_CACHE_INPLACE=1 keeps the KV rows in the pool, so doubling the
pool evicted nothing the prompt cache needed. The RAM price is still real:
MemAvailable 26 -> 22 GiB.

Census of the same engine's surface corrected the Benchmarking Policy, which
quoted llama.cpp instruments at a halogen endpoint: frank-dragoon answers
/health, /metrics, /v1/models and POST /v1/chat/completions and 404s /ping,
/props, /completion and /tokenize, and no response carries Server-Timing. The
/completion `timings` object and the ttft;dur= header are frank-zealot's; on
frank-dragoon the instrument is wall clock plus usage.prompt_tokens.
This commit is contained in:
Frank
2026-10-18 13:36:35 -07:00
parent c0759492e6
commit e7841bf9d6
3 changed files with 43 additions and 13 deletions
@@ -152,9 +152,15 @@ of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`. Prompt cache ON (12
entries × 111 MiB at ctx 200704 = the 1.3 GiB in the KV-budget line), n-gram
47.68 GiB still disk-backed (`Pin::None`), no clamp line. Host `MemAvailable`
fell 26 → 22 GiB across the raise, which is the FLAGS.md collision being paid
in page cache. **Not re-measured since the raise:** the warm prompt-cache ratio
(136.7× on the pre-raise boot) — that is the number to watch if a long warm
turn starts costing full prefill again.
in page cache. **Warm reuse survived it — measured after the deploy.** One
**101,987**-token wikitext prompt through `/v1/chat/completions` took **81.5 s**
cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats:
`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn prefills
nothing and the doubling evicted nothing the cache needed. (The pre-raise
136.7× figure was a `ttft` reading; this engine sends **no `Server-Timing`**
header and has **no `/completion` path** — `/ping`, `/props`, `/tokenize` are
404 too — so wall-clock around the POST plus `usage.prompt_tokens` is the only
instrument available here.)
## Measured A/B — same battery, both boots (2026-10-17)
@@ -235,7 +241,12 @@ is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px
probes; `response_format` unsupported; the chat template rejects
`reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays
mandatory); unknown sampling params are a 400; the prompt cache is on by
default, so a warm 100k follow-up costs seconds. See
default, so a warm 100k follow-up costs seconds (measured after the pool raise:
**81.5 s cold vs 0.3 s warm** at 101,987 prompt tokens). The HTTP surface is
thin — `/health`, `/metrics`, `/v1/models` and POST `/v1/chat/completions`
answer, while `/ping`, `/props`, `/completion` and `/tokenize` are 404 and no
response carries `Server-Timing`: llama.cpp's timing instruments do not exist
on this engine, so measure wall-clock plus `usage.prompt_tokens`. See
[[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that
carries verbatim, and [[deployment]] for the deploy/health gate.
+15 -4
View File
@@ -2200,10 +2200,13 @@ every restart since the cutover.
to 200704`, and `62.9 GiB of weights locked … 11.0 GiB of KV pool, 5.7 GiB of
working memory, 79.6 GiB in all`; prompt cache ON (12 entries × 111 MiB at
ctx 200704 = 1.3 GiB), n-gram 47.68 GiB still `Pin::None`, no clamp line.
- The price is exactly the FLAGS.md collision: host `MemAvailable` fell
**26 → 22 GiB**, because every pool position is RAM the n-gram table's page
cache loses. The warm prompt-cache ratio (136.7× measured pre-raise) was NOT
re-measured — recorded in the runbook as the number to watch.
- Warm reuse survived the raise, measured right after the deploy: one
**101,987**-token wikitext prompt through `/v1/chat/completions` took
**81.5 s** cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats —
`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn
prefills nothing and the doubling evicted nothing the cache needed. The RAM
price is still real: host `MemAvailable` fell **26 → 22 GiB**, because every
pool position is RAM the n-gram table's page cache loses.
- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the
engine reserves **11.0 GiB** and its preflight bills **11.3**. Re-keyed in
AGENTS.md (topology row, KV-pool law, memory ledger), the `frank-dragoon.env`
@@ -2226,3 +2229,11 @@ every restart since the cutover.
200,704 → **262,144** with the 9B description; dragoon 262,144 → **200,704**,
which had kept the pre-lowering window while its own description said
200,704.
- Census of the halogen HTTP surface while doing it, because the fleet bench
law quoted the wrong engine: frank-dragoon answers **`/health`, `/metrics`,
`/v1/models`, `/v1/chat/completions`(POST)** and 404s `/ping`, `/props`,
`/completion`, `/tokenize`; the chat response carries **no `Server-Timing`**
header (`server: uvicorn`, only `date`/`content-*`). So the `/completion`
`timings` object and the `ttft;dur=` header are **llama.cpp** instruments —
frank-zealot's, not dragoon's. AGENTS.md's Benchmarking Policy re-keyed to
say which engine owns which instrument.