docs(fleet): halogen owns no timing instruments; warm-cache cost measured after the pool raise
Post-deploy probe on frank-dragoon: one 101,987-token prompt through /v1/chat/completions took 81.5 s cold (1,251 t/s of prefill) and 0.3 s on two repeats — HALOGEN_CACHE_INPLACE=1 keeps the KV rows in the pool, so doubling the pool evicted nothing the prompt cache needed. The RAM price is still real: MemAvailable 26 -> 22 GiB. Census of the same engine's surface corrected the Benchmarking Policy, which quoted llama.cpp instruments at a halogen endpoint: frank-dragoon answers /health, /metrics, /v1/models and POST /v1/chat/completions and 404s /ping, /props, /completion and /tokenize, and no response carries Server-Timing. The /completion `timings` object and the ttft;dur= header are frank-zealot's; on frank-dragoon the instrument is wall clock plus usage.prompt_tokens.
This commit is contained in:
@@ -231,8 +231,12 @@ archive.
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench
|
||||
requires asking first.
|
||||
endpoints; a live llama.cpp server (frank-zealot) also reports honest
|
||||
per-request numbers in its `/completion` `timings` object (prefill/decode
|
||||
rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured.
|
||||
halogen has NO `/completion` path at all — frank-dragoon's rates come from
|
||||
wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own
|
||||
bench requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
@@ -250,9 +254,13 @@ real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B,
|
||||
replay **real** text through the live server and read the `Server-Timing`
|
||||
header — but note the unit rides in the **name** half (`ttft;dur=402.485`),
|
||||
so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently
|
||||
returns nothing. Generate **at least 128 tokens** for the measurement: at 8
|
||||
tokens, `inter_token` is first-token noise and the run-to-run spread swamps
|
||||
every real difference.
|
||||
returns nothing. **That header is a llama.cpp feature**: halogen flash-server
|
||||
0.16.2 answers `/v1/chat/completions` with only `date`, `server` and
|
||||
`content-*` (verified 2026-10-18), and `/completion`, `/ping`, `/props` and
|
||||
`/tokenize` are all 404 on it — on frank-dragoon the A/B instrument is
|
||||
wall-clock around the POST plus `usage.prompt_tokens`. Generate **at least 128
|
||||
tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise
|
||||
and the run-to-run spread swamps every real difference.
|
||||
|
||||
This covers **any GPU experiment, not just benchmarks** — including a bare
|
||||
2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
|
||||
|
||||
Reference in New Issue
Block a user