diff --git a/AGENTS.md b/AGENTS.md index f25c7fb..a902255 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -208,13 +208,27 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts -results. Use the runtime's own harness: `llama-bench` for every llama.cpp -endpoint (all three are llama.cpp now). Rolling your own bench +results. Use the runtime's own harness: `llama-bench` for the llama.cpp +endpoints, `gufo bench` for frank-large. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. +**`gufo bench` cannot rank speculative-decode settings, and will invert the +ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus +reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across +draft widths, i.e. the same highly-predictable text every run, and 93 % of +every proposal accepted. It reported 29–37 t/s decode where real traffic +gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when +real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B, +replay **real** text through the live server and read the `Server-Timing` +header — but note the unit rides in the **name** half (`ttft;dur=402.485`), +so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently +returns nothing. Generate **at least 128 tokens** for the measurement: at 8 +tokens, `inter_token` is first-token noise and the run-to-run spread swamps +every real difference. + This covers **any GPU experiment, not just benchmarks** — including a bare 2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index cedbb77..c53f1b9 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -179,6 +179,40 @@ unchanged by the payload swap. Short-prompt decode is gufo's *winning* regime; see the depth table above before quoting it as the endpoint's number. +### Tuning: the served flags are already optimal (2026-09-28) + +Frank asked for iterative decode/prefill improvement at a 32k working point +without touching the model or the quantization. **Result: no win exists at +this engine's served settings, and nothing was changed.** Measured on real +repo prose and shell sources at **32,762 prompt tokens**, salted per run so +a warm prefix cache could not help, 3–5 repetitions each. + +| lever | range swept | outcome | +|---|---|---| +| `--prefill-chunk` | 512 → 8192 | **no effect** — 410.7–412.4 t/s prefill across a 16× change | +| `--draft-tokens` | 5 → 10 | **flat** — 13.79–15.86 t/s decode, inside run-to-run noise | +| `--draft-policy` | fixed vs adaptive | 35.90 vs 35.50 (bench) — no effect | +| `--min-draft-tokens` | — | **rejected by the engine**: "DFlash2 requires … 1" | +| MTP backend | — | **unsupported for Q8**: `token_embd.weight` is Q8_0 | +| `--context` 262144 → 40960 | diagnostic | 13.93 vs 14.71 t/s — no gain, and it would break the declared client context | + +The confirmation round at 5 reps **reversed** the 3-rep leaders: the pair +that looked best (`dt9 + chunk4096`) came last at 12.86 t/s, and the +**deployed configuration came first at 14.71 t/s**. Every difference sits +inside the noise band. + +What *does* move the number is speculative decoding itself: **6.65 t/s off → +~15 t/s on, a 5.3× step** — and that is already switched on. Prefill is +insensitive to everything tried, which is consistent with gufo's own +`EXPERIMENTS.md`, where dozens of kernel-level alternatives are recorded as +rejected for "no release throughput improvement". + +**Do not repeat this investigation without new information.** Two traps +cost most of the time, and both are in AGENTS.md's benchmarking policy: +`gufo bench` reports `acceptance=0.933 ± 0.000` on an identical trace hash +and **inverts** the draft-width ranking; and a live A/B needs ≥128 generated +tokens or `inter_token` is first-token noise. + ## The template law: Qwen-Sharp is gone `scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index c49a0f2..349caf6 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,6 +1,51 @@ # Bundle Change Log +## 2026-09-29 +- **frank-large tuning investigation: a measured NEGATIVE result, nothing + changed.** Frank asked for iterative decode/prefill improvement at a 32k + working point without touching the model or quantization. **No win exists + at gufo's served settings**, so no config change was shipped. frank-large + was stopped for the whole investigation (~4h) per the benchmarking policy + and restored healthy in 9s afterwards. + - Measured on the repo's own prose and shell sources at **32,762 prompt + tokens**, salted per run so a warm prefix cache could not flatter a + later run, 3–5 repetitions per configuration. + - **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s. + A 16x change in chunk size, 0.4% out. + - **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all + inside the run-to-run noise band. The 5-rep confirmation round + **reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came + **last** (12.86), while the **deployed config came first** (14.71). + - `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`: + **rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported + for Q8 targets** (`Unsupported Qwen MTP tensor type for + token_embd.weight: Q8_0`). + - `--context 262144 → 40960` was tested as a diagnostic (the deployed value + preallocates KV ~8x larger than a 32k conversation needs): **no gain** + (13.93 vs 14.71 t/s), and it would break `models.yml`'s declared + `contextWindow: 262144`. Not shipped. + - **What actually moves the number is speculation itself: 6.65 t/s off → + ~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to + everything tried, consistent with gufo's own `EXPERIMENTS.md` where + dozens of kernel variants are recorded rejected for "no release + throughput improvement". + - **Two traps are now in AGENTS.md's benchmarking policy**, because they + cost most of the wall time and both produced *false* results: + (1) **`gufo bench` cannot rank speculative settings and inverted the + ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an + identical trace sha256 across draft widths (the same highly-predictable + text every run, 93% of proposals accepted). It reported 29–37 t/s where + real traffic gives ~15, and made dt16 look 6% faster than dt7 when real + text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**; + at 8, `inter_token` is first-token noise and the first pass of the sweep + produced a nonsense ordering that the 128-token pass reversed. + - Also worth recording: the `Server-Timing` header is + `ttft;dur=402.485` — the unit is in the **name** half, so splitting on + `=` keys the dict `ttft;dur` and every lookup silently returns nothing. + Four configurations reported "no Server-Timing" while the header was + present on all of them. + ## 2026-09-28 - **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to clean up old models. **510 G staged** to