docs(frank-primary): parallelism survey — DCP rejected on 2 GPUs (GQA constraint)
--decode-context-parallel-size 2 fails VllmConfig validation at boot: 'Decode context parallelism for GQA/MQA requires --tensor-parallel-size (2) to be greater than the model's total number of KV heads (2)'. DCP shards KV along the sequence dimension AFTER KV heads spread across TP ranks, so it needs TP > num_kv_heads; this model has 2 KV heads, so TP >= 4 — four GPUs, not two. Recorded with the rest of the survey (PP/DP/EP/multi-node verdicts for this box), the container-injection requirement for /home/dank/mtp_measure.py (it imports /bench_iter.py by absolute path and must be copied in), and the re-measured deployed capacity (1,079,582 tokens / 4.12x).
This commit is contained in:
@@ -320,6 +320,60 @@ only available while the container lives).
|
||||
Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
|
||||
`frank-primary-fp8-mtp-20261001` (env trio + seam re-pinned there).
|
||||
|
||||
## Parallelism options on this box (2026-10-01 survey)
|
||||
|
||||
Surveyed from the running image's own CLI surface. Verdicts are specific to
|
||||
dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
|
||||
with only 2 KV heads.
|
||||
|
||||
| option | flag | verdict on this box |
|
||||
|---|---|---|
|
||||
| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
|
||||
| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
|
||||
| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
|
||||
| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
|
||||
| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
|
||||
| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
|
||||
|
||||
### DCP is structurally unavailable here — do not re-test on 2 GPUs
|
||||
`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
|
||||
|
||||
```
|
||||
Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
|
||||
greater than the model's total number of KV heads (2). Increase
|
||||
`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
|
||||
```
|
||||
|
||||
DCP shards the KV along the *sequence* dimension after KV heads have been spread
|
||||
across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
|
||||
heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
|
||||
fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
|
||||
`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
|
||||
(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
|
||||
each rank would hold half the sequence's KV — but it is unreachable without two
|
||||
more cards.
|
||||
|
||||
### Measuring decode: the harness lives INSIDE the container
|
||||
`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
|
||||
run in the engine's namespace and the files must be copied in first:
|
||||
|
||||
```
|
||||
podman cp /home/dank/mtp_measure.py frank-primary-qwen:/
|
||||
podman cp /home/dank/bench_iter.py frank-primary-qwen:/
|
||||
podman exec frank-primary-qwen python3 /mtp_measure.py
|
||||
```
|
||||
|
||||
Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
|
||||
No such file or directory` — measured 2026-10-01, which silently cost a whole
|
||||
arm's numbers. A window script must do the copy, not assume the path exists.
|
||||
|
||||
### The deployed shape, measured again (2026-10-01 A/B window)
|
||||
A clean production-environment boot of the deployed config (4 x 262144 fp8, MNBT
|
||||
2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
|
||||
concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
|
||||
deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
|
||||
reproducible across boots.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
|
||||
|
||||
Reference in New Issue
Block a user