From 67672dc7947c2253374b0d35b392fffc9c9034b2 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 29 Sep 2026 05:06:06 -0700 Subject: [PATCH] docs(llm): record a measured NEGATIVE tuning result for frank-large MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asked for iterative decode/prefill improvement at a 32k working point without touching the model or the quantization. There isn't one: gufo's served settings are already at this engine's tuned point, so no config change was made and none is proposed. Measured on the repo's own prose and shell sources at 32,762 prompt tokens, salted per run against a warm prefix cache, 3-5 repetitions: * --prefill-chunk 512 -> 8192 moves prefill 410.7-412.4 t/s. A 16x change in chunk size, 0.4% out. The lever is dead at this depth. * --draft-tokens 5 -> 10 gives 13.79-15.86 t/s decode, all inside the run-to-run noise. The 5-rep confirmation REVERSED the 3-rep leaders: dt9+chunk4096 looked best and came last (12.86), the deployed config came first (14.71). * --draft-policy fixed vs adaptive: no effect. --min-draft-tokens: rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for Q8 targets (token_embd.weight is Q8_0). * --context 262144 -> 40960, tested as a diagnostic: no gain, and it would break models.yml's declared contextWindow. Not shipped. What does move the number is speculation itself — 6.65 t/s off vs ~15 t/s on, 5.3x — and it is already enabled. The two traps that cost most of the wall time are now in AGENTS.md's benchmarking policy, because both produced FALSE results: 1. gufo bench cannot rank speculative settings and inverted the ranking. Its corpus reports acceptance=0.933 +/- 0.000 with an identical trace sha256 across draft widths — the same highly-predictable text every run. It reported 29-37 t/s where real traffic gives ~15, and made dt16 look 6% faster than dt7 when real text made it 14% slower. 2. A live A/B needs >=128 generated tokens. At 8, inter_token is first-token noise; the first pass produced a nonsense ordering that the 128-token pass reversed. Plus the Server-Timing header is `ttft;dur=402.485` — the unit rides in the NAME half, so splitting on "=" keys the dict "ttft;dur" and every lookup silently returns nothing. Four configs reported "no Server-Timing" while the header was present on all of them. --- AGENTS.md | 18 +++++++- .../LLM/frank-large-gufo-qwen38-27b.md | 34 ++++++++++++++ serverAppConfigs-docs/log.md | 45 +++++++++++++++++++ 3 files changed, 95 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f25c7fb..a902255 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -208,13 +208,27 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts -results. Use the runtime's own harness: `llama-bench` for every llama.cpp -endpoint (all three are llama.cpp now). Rolling your own bench +results. Use the runtime's own harness: `llama-bench` for the llama.cpp +endpoints, `gufo bench` for frank-large. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. +**`gufo bench` cannot rank speculative-decode settings, and will invert the +ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus +reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across +draft widths, i.e. the same highly-predictable text every run, and 93 % of +every proposal accepted. It reported 29–37 t/s decode where real traffic +gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when +real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, +replay **real** text through the live server and read the `Server-Timing` +header — but note the unit rides in the **name** half (`ttft;dur=402.485`), +so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently +returns nothing. Generate **at least 128 tokens** for the measurement: at 8 +tokens, `inter_token` is first-token noise and the run-to-run spread swamps +every real difference. + This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index cedbb77..c53f1b9 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -179,6 +179,40 @@ unchanged by the payload swap. Short-prompt decode is gufo's *winning* regime; see the depth table above before quoting it as the endpoint's number. +### Tuning: the served flags are already optimal (2026-09-28) + +Frank asked for iterative decode/prefill improvement at a 32k working point +without touching the model or the quantization. **Result: no win exists at +this engine's served settings, and nothing was changed.** Measured on real +repo prose and shell sources at **32,762 prompt tokens**, salted per run so +a warm prefix cache could not help, 3–5 repetitions each. + +| lever | range swept | outcome | +|---|---|---| +| `--prefill-chunk` | 512 → 8192 | **no effect** — 410.7–412.4 t/s prefill across a 16× change | +| `--draft-tokens` | 5 → 10 | **flat** — 13.79–15.86 t/s decode, inside run-to-run noise | +| `--draft-policy` | fixed vs adaptive | 35.90 vs 35.50 (bench) — no effect | +| `--min-draft-tokens` | — | **rejected by the engine**: "DFlash2 requires … 1" | +| MTP backend | — | **unsupported for Q8**: `token_embd.weight` is Q8_0 | +| `--context` 262144 → 40960 | diagnostic | 13.93 vs 14.71 t/s — no gain, and it would break the declared client context | + +The confirmation round at 5 reps **reversed** the 3-rep leaders: the pair +that looked best (`dt9 + chunk4096`) came last at 12.86 t/s, and the +**deployed configuration came first at 14.71 t/s**. Every difference sits +inside the noise band. + +What *does* move the number is speculative decoding itself: **6.65 t/s off → +~15 t/s on, a 5.3× step** — and that is already switched on. Prefill is +insensitive to everything tried, which is consistent with gufo's own +`EXPERIMENTS.md`, where dozens of kernel-level alternatives are recorded as +rejected for "no release throughput improvement". + +**Do not repeat this investigation without new information.** Two traps +cost most of the time, and both are in AGENTS.md's benchmarking policy: +`gufo bench` reports `acceptance=0.933 ± 0.000` on an identical trace hash +and **inverts** the draft-width ranking; and a live A/B needs ≥128 generated +tokens or `inter_token` is first-token noise. + ## The template law: Qwen-Sharp is gone `scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index c49a0f2..349caf6 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,6 +1,51 @@ # Bundle Change Log +## 2026-09-29 +- **frank-large tuning investigation: a measured NEGATIVE result, nothing + changed.** Frank asked for iterative decode/prefill improvement at a 32k + working point without touching the model or quantization. **No win exists + at gufo's served settings**, so no config change was shipped. frank-large + was stopped for the whole investigation (~4h) per the benchmarking policy + and restored healthy in 9s afterwards. + - Measured on the repo's own prose and shell sources at **32,762 prompt + tokens**, salted per run so a warm prefix cache could not flatter a + later run, 3–5 repetitions per configuration. + - **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s. + A 16x change in chunk size, 0.4% out. + - **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all + inside the run-to-run noise band. The 5-rep confirmation round + **reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came + **last** (12.86), while the **deployed config came first** (14.71). + - `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`: + **rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported + for Q8 targets** (`Unsupported Qwen MTP tensor type for + token_embd.weight: Q8_0`). + - `--context 262144 → 40960` was tested as a diagnostic (the deployed value + preallocates KV ~8x larger than a 32k conversation needs): **no gain** + (13.93 vs 14.71 t/s), and it would break `models.yml`'s declared + `contextWindow: 262144`. Not shipped. + - **What actually moves the number is speculation itself: 6.65 t/s off → + ~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to + everything tried, consistent with gufo's own `EXPERIMENTS.md` where + dozens of kernel variants are recorded rejected for "no release + throughput improvement". + - **Two traps are now in AGENTS.md's benchmarking policy**, because they + cost most of the wall time and both produced *false* results: + (1) **`gufo bench` cannot rank speculative settings and inverted the + ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an + identical trace sha256 across draft widths (the same highly-predictable + text every run, 93% of proposals accepted). It reported 29–37 t/s where + real traffic gives ~15, and made dt16 look 6% faster than dt7 when real + text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**; + at 8, `inter_token` is first-token noise and the first pass of the sweep + produced a nonsense ordering that the 128-token pass reversed. + - Also worth recording: the `Server-Timing` header is + `ttft;dur=402.485` — the unit is in the **name** half, so splitting on + `=` keys the dict `ttft;dur` and every lookup silently returns nothing. + Four configurations reported "no Server-Timing" while the header was + present on all of them. + ## 2026-09-28 - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to clean up old models. **510 G staged** to