Files
serverAppConfigs/serverAppConfigs-docs/LLM
Frank e7841bf9d6 docs(fleet): halogen owns no timing instruments; warm-cache cost measured after the pool raise
Post-deploy probe on frank-dragoon: one 101,987-token prompt through
/v1/chat/completions took 81.5 s cold (1,251 t/s of prefill) and 0.3 s on two
repeats — HALOGEN_CACHE_INPLACE=1 keeps the KV rows in the pool, so doubling the
pool evicted nothing the prompt cache needed. The RAM price is still real:
MemAvailable 26 -> 22 GiB.

Census of the same engine's surface corrected the Benchmarking Policy, which
quoted llama.cpp instruments at a halogen endpoint: frank-dragoon answers
/health, /metrics, /v1/models and POST /v1/chat/completions and 404s /ping,
/props, /completion and /tokenize, and no response carries Server-Timing. The
/completion `timings` object and the ttft;dur= header are frank-zealot's; on
frank-dragoon the instrument is wall clock plus usage.prompt_tokens.
2026-10-18 13:36:35 -07:00
..