frank-medium: ubatch 512, and rebalance the split to 35:30

The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau: ubatch 256 384 512 768 1024 1280 1536 1792 2048 tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078 The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 / 1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out 1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the shape, not noise. The logical batch is throughput-neutral (2048..8192 all within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell. So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2): 36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed) 36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone) 35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1 holds the output layers and the drafter's graphs -- so it shifted the balance toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split can be tuned finer than a layer, which is why 34,31 is not better. Re-fitted the seam with both ubatch coefficients measured at this shape and re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to 0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 -> 0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB. Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the shape it actually cares about -- two shares that add up to the model's 65 blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer breaks a test about parallelism. The numeric contract lives in the seam test.
2026-09-17 00:21:48 -07:00
parent 2f012fba39
commit a11fae07d8
3 changed files with 142 additions and 46 deletions
@@ -31,6 +31,16 @@ P2P operation can be dispatched on card #2. Two paths restore cooperation:
 2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class,
   ~$250-400). Best performance. This board has no x8/x8 mode.

+The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS
+lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as
+chipset, while an empty slot does not enumerate — so nothing on a running
+system can confirm it until a card is in it. And CPU-attached does not imply
+`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth
+trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what
+the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one
+model" — the layer split already delivers that, so this is a performance
+project, not a capability one.
+
 ## The gate, in source

 ```cpp
@@ -59,6 +69,37 @@ CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path
 includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability
 bit**, so no software can enable it.

+### The same gate, as this board reports it (2026-09-16)
+
+`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` →
+`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` →
+`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`:
+
+| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
+|---|---|---|---|---|
+| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 |
+| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 |
+| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 |
+| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` |
+| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) |
+| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 |
+
+Three things fall out of that table, none of them in the source alone:
+
+- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the
+  chipset, not the platform, which is why a slot on CPU lanes is the fix.
+- **The endpoint's own bit differs between the cards, and that is the kernel's
+  doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses
+  to set Requester Enable when any bridge cannot route atomics. ROCclr's
+  `pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it
+  makes `lspci -vv -s <bdf> | grep AtomicOpsCtl` a one-command diagnosis: on
+  this host card 0 reads `ReqEn+` and card 1 `ReqEn−`.
+- **Card 1's effective link is x1**, not the x16 its own capability block
+  advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a
+  bandwidth fact rather than an atomics one, and it only matters for
+  collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
+  x4. The layer split's boundary copy does not care.
+
 ## Measured matrix

 Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no