diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 558c7c2..73e4c28 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -186,7 +186,7 @@ Both replicas, uniform except GPU index and port: | `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency | | `CHUNK` | 8192 | prefill chunk (upstream default), **coupled to the context**: the prefill arena grows with the chunk size and shares the device budget with the KV cache. Measured: `CHUNK=16384` leaves 4.22 GiB of KV against the 5.21 GiB that 114688 context needs, and the engine refuses to start (`To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)`). Raising CHUNK therefore means cutting `MAXLEN`; single-card prefill measured ≈2.5k tok/s at 8192 (~15k-token prompt, 5992 ms TTFT), so the trade was not worth taking | | `GPU_UTIL` | 0.98 | upstream default; the card is dedicated | -| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter, upstream's measured production depth | +| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter. Depth 7 is upstream's production default and was re-measured here on a code corpus built from this repo (see [Speculative depth](#speculative-depth)); depth 5 wins batch throughput, 7 wins single-stream latency, and this fleet is interactive | | `SNAP` / `DRAFTER` | `/models/Qwen3.8-27B-Uncensored-MXFP4-awq` / `/models/Qwen3.8-27B-DFlash2-FP8` | the served checkpoint and its drafter | Mounts are the TP=2 unit's minus nothing: `/models`, the HF cache, the shared @@ -258,6 +258,28 @@ Bench caveat: the client must be given `--tokenizer HuggingFace and die before sending a request — a green-looking run that produced no numbers at all. +### Speculative depth + +Upstream ships depth 7 for dflash and notes the choice is content-dependent; it +was measured for TP=2 on their corpus. Re-measured here on one card against a +code corpus built from this repository's own sources (16 prompts, 512 output +tokens each, two runs per setting): + +| Depth | conc 1 | conc 4 | +|---|--:|--:| +| `SPEC=7` (deployed) | **46.2 tok/s, TPOT 20.8 ms**, 3.86 tok/update | 118.7 / 127.0 tok/s, TPOT 25.1-29.3 ms | +| `SPEC=5` | 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update | **136.1 / 129.4 tok/s**, TPOT 25.3-26.6 ms | + +The step *time* is nearly identical at both depths (≈79 ms against ≈80 ms at +conc 1) — depth 7 wins single-stream because it lands more tokens per step +(3.86 against 3.41), and depth 5 wins at conc 4 by ≈8 % because more, narrower +verify steps pack better. Depth 7 stays: this fleet serves about three people +in interactive agent loops, where per-turn latency is the number that is felt, +and it is also the better default on code at low concurrency. Switch a replica +to `SPEC=5` if batch throughput ever matters more than turn latency — it is one +environment line in the unit, at the cost of a recompile (the spec depth is +part of the graph cache key). + ## Citations - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer