diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index 6e1db27..81f3422 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -320,6 +320,60 @@ only available while the container lives). Rollback to the 2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there). +## Parallelism options on this box (2026-10-01 survey) + +Surveyed from the running image's own CLI surface. Verdicts are specific to +dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model +with only 2 KV heads. + +| option | flag | verdict on this box | +|---|---|---| +| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` | +| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below | +| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) | +| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) | +| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all | +| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards | + +### DCP is structurally unavailable here — do not re-test on 2 GPUs +`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot: + +``` +Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be +greater than the model's total number of KV heads (2). Increase +`--tensor-parallel-size` or set `--decode-context-parallel-size 1`. +``` + +DCP shards the KV along the *sequence* dimension after KV heads have been spread +across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV +heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related +fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not +`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}` +(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real — +each rank would hold half the sequence's KV — but it is unreachable without two +more cards. + +### Measuring decode: the harness lives INSIDE the container +`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must +run in the engine's namespace and the files must be copied in first: + +``` +podman cp /home/dank/mtp_measure.py frank-primary-qwen:/ +podman cp /home/dank/bench_iter.py frank-primary-qwen:/ +podman exec frank-primary-qwen python3 /mtp_measure.py +``` + +Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2] +No such file or directory` — measured 2026-10-01, which silently cost a whole +arm's numbers. A window script must do the copy, not assume the path exists. + +### The deployed shape, measured again (2026-10-01 A/B window) +A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT +2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum +concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the +deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is +reproducible across boots. + ## See Also - [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page