From 4d37f8ca65f7ae0ccf8a6486ae5e4fe14104a604 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 05:23:09 -0700 Subject: [PATCH] frank-medium: measure speculative depth on the repo's own code corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream ships dflash depth 7 as a content-dependent default measured for TP=2 on their corpus. Re-measured on one card here, two runs per setting, 16 repo-derived prompts at 512 output tokens: SPEC=7 conc1 46.2 tok/s, TPOT 20.8 ms, 3.86 tok/update conc4 118.7 / 127.0 tok/s SPEC=5 conc1 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update conc4 136.1 / 129.4 tok/s Step time is nearly identical at both depths (~79 ms vs ~80 ms single stream): 7 wins by landing more tokens per step, 5 wins at conc 4 by ~8% because more narrow verify steps pack better. Depth 7 stays — this fleet is interactive agent loops where per-turn latency is what is felt — and the runbook now records the numbers and the one-line switch for a batch-first workload. --- .../LLM/frank-medium-radiance-mxfp4.md | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 558c7c2..73e4c28 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -186,7 +186,7 @@ Both replicas, uniform except GPU index and port: | `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency | | `CHUNK` | 8192 | prefill chunk (upstream default), **coupled to the context**: the prefill arena grows with the chunk size and shares the device budget with the KV cache. Measured: `CHUNK=16384` leaves 4.22 GiB of KV against the 5.21 GiB that 114688 context needs, and the engine refuses to start (`To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)`). Raising CHUNK therefore means cutting `MAXLEN`; single-card prefill measured ≈2.5k tok/s at 8192 (~15k-token prompt, 5992 ms TTFT), so the trade was not worth taking | | `GPU_UTIL` | 0.98 | upstream default; the card is dedicated | -| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter, upstream's measured production depth | +| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter. Depth 7 is upstream's production default and was re-measured here on a code corpus built from this repo (see [Speculative depth](#speculative-depth)); depth 5 wins batch throughput, 7 wins single-stream latency, and this fleet is interactive | | `SNAP` / `DRAFTER` | `/models/Qwen3.8-27B-Uncensored-MXFP4-awq` / `/models/Qwen3.8-27B-DFlash2-FP8` | the served checkpoint and its drafter | Mounts are the TP=2 unit's minus nothing: `/models`, the HF cache, the shared @@ -258,6 +258,28 @@ Bench caveat: the client must be given `--tokenizer HuggingFace and die before sending a request — a green-looking run that produced no numbers at all. +### Speculative depth + +Upstream ships depth 7 for dflash and notes the choice is content-dependent; it +was measured for TP=2 on their corpus. Re-measured here on one card against a +code corpus built from this repository's own sources (16 prompts, 512 output +tokens each, two runs per setting): + +| Depth | conc 1 | conc 4 | +|---|--:|--:| +| `SPEC=7` (deployed) | **46.2 tok/s, TPOT 20.8 ms**, 3.86 tok/update | 118.7 / 127.0 tok/s, TPOT 25.1-29.3 ms | +| `SPEC=5` | 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update | **136.1 / 129.4 tok/s**, TPOT 25.3-26.6 ms | + +The step *time* is nearly identical at both depths (≈79 ms against ≈80 ms at +conc 1) — depth 7 wins single-stream because it lands more tokens per step +(3.86 against 3.41), and depth 5 wins at conc 4 by ≈8 % because more, narrower +verify steps pack better. Depth 7 stays: this fleet serves about three people +in interactive agent loops, where per-turn latency is the number that is felt, +and it is also the better default on code at low concurrency. Switch a replica +to `SPEC=5` if batch throughput ever matters more than turn latency — it is one +environment line in the unit, at the cost of a recompile (the spec depth is +part of the graph cache key). + ## Citations - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer