diff --git a/AGENTS.md b/AGENTS.md index eabf47f..b1c1d0c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -231,8 +231,12 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp -endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench -requires asking first. +endpoints; a live llama.cpp server (frank-zealot) also reports honest +per-request numbers in its `/completion` `timings` object (prefill/decode +rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured. +halogen has NO `/completion` path at all — frank-dragoon's rates come from +wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own +bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. @@ -250,9 +254,13 @@ real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, replay **real** text through the live server and read the `Server-Timing` header — but note the unit rides in the **name** half (`ttft;dur=402.485`), so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently -returns nothing. Generate **at least 128 tokens** for the measurement: at 8 -tokens, `inter_token` is first-token noise and the run-to-run spread swamps -every real difference. +returns nothing. **That header is a llama.cpp feature**: halogen flash-server +0.16.2 answers `/v1/chat/completions` with only `date`, `server` and +`content-*` (verified 2026-10-18), and `/completion`, `/ping`, `/props` and +`/tokenize` are all 404 on it — on frank-dragoon the A/B instrument is +wall-clock around the POST plus `usage.prompt_tokens`. Generate **at least 128 +tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise +and the run-to-run spread swamps every real difference. This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run diff --git a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md index 66228c9..59701cb 100644 --- a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md +++ b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md @@ -152,9 +152,15 @@ of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`. Prompt cache ON (12 entries × 111 MiB at ctx 200704 = the 1.3 GiB in the KV-budget line), n-gram 47.68 GiB still disk-backed (`Pin::None`), no clamp line. Host `MemAvailable` fell 26 → 22 GiB across the raise, which is the FLAGS.md collision being paid -in page cache. **Not re-measured since the raise:** the warm prompt-cache ratio -(136.7× on the pre-raise boot) — that is the number to watch if a long warm -turn starts costing full prefill again. +in page cache. **Warm reuse survived it — measured after the deploy.** One +**101,987**-token wikitext prompt through `/v1/chat/completions` took **81.5 s** +cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats: +`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn prefills +nothing and the doubling evicted nothing the cache needed. (The pre-raise +136.7× figure was a `ttft` reading; this engine sends **no `Server-Timing`** +header and has **no `/completion` path** — `/ping`, `/props`, `/tokenize` are +404 too — so wall-clock around the POST plus `usage.prompt_tokens` is the only +instrument available here.) ## Measured A/B — same battery, both boots (2026-10-17) @@ -235,7 +241,12 @@ is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px probes; `response_format` unsupported; the chat template rejects `reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays mandatory); unknown sampling params are a 400; the prompt cache is on by -default, so a warm 100k follow-up costs seconds. See +default, so a warm 100k follow-up costs seconds (measured after the pool raise: +**81.5 s cold vs 0.3 s warm** at 101,987 prompt tokens). The HTTP surface is +thin — `/health`, `/metrics`, `/v1/models` and POST `/v1/chat/completions` +answer, while `/ping`, `/props`, `/completion` and `/tokenize` are 404 and no +response carries `Server-Timing`: llama.cpp's timing instruments do not exist +on this engine, so measure wall-clock plus `usage.prompt_tokens`. See [[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that carries verbatim, and [[deployment]] for the deploy/health gate. diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index a827b29..290c304 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2200,10 +2200,13 @@ every restart since the cutover. to 200704`, and `62.9 GiB of weights locked … 11.0 GiB of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`; prompt cache ON (12 entries × 111 MiB at ctx 200704 = 1.3 GiB), n-gram 47.68 GiB still `Pin::None`, no clamp line. -- The price is exactly the FLAGS.md collision: host `MemAvailable` fell - **26 → 22 GiB**, because every pool position is RAM the n-gram table's page - cache loses. The warm prompt-cache ratio (136.7× measured pre-raise) was NOT - re-measured — recorded in the runbook as the number to watch. +- Warm reuse survived the raise, measured right after the deploy: one + **101,987**-token wikitext prompt through `/v1/chat/completions` took + **81.5 s** cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats — + `HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn + prefills nothing and the doubling evicted nothing the cache needed. The RAM + price is still real: host `MemAvailable` fell **26 → 22 GiB**, because every + pool position is RAM the n-gram table's page cache loses. - Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the engine reserves **11.0 GiB** and its preflight bills **11.3**. Re-keyed in AGENTS.md (topology row, KV-pool law, memory ledger), the `frank-dragoon.env` @@ -2226,3 +2229,11 @@ every restart since the cutover. 200,704 → **262,144** with the 9B description; dragoon 262,144 → **200,704**, which had kept the pre-lowering window while its own description said 200,704. +- Census of the halogen HTTP surface while doing it, because the fleet bench + law quoted the wrong engine: frank-dragoon answers **`/health`, `/metrics`, + `/v1/models`, `/v1/chat/completions`(POST)** and 404s `/ping`, `/props`, + `/completion`, `/tokenize`; the chat response carries **no `Server-Timing`** + header (`server: uvicorn`, only `date`/`content-*`). So the `/completion` + `timings` object and the `ttft;dur=` header are **llama.cpp** instruments — + frank-zealot's, not dragoon's. AGENTS.md's Benchmarking Policy re-keyed to + say which engine owns which instrument.