Frank
71a7e5cb9a
Per-card measurement was unreachable for every card but ordinal 0: the driver set HIP_VISIBLE_DEVICES=N and still passed N as the device argument, so the probe called hipSetDevice(N) on a runtime where the mask left only ordinal 0. The bats stub reported the env var rather than the argument it received, so the suite asserted the mask was set and never noticed the collision. - one selector: the ordinal is passed as argv, the mask is never set - a probe that fails is now exit 5 and names probe + device; the sweep still measures the remaining cards instead of swallowing the failure - the guard fails closed when `fuser` is missing (gpu_busy_percent reads 0 for a resident-but-idle service), with PCIE_PROBE_FUSER as the test seam - stub probes now model the HIP mask rule; the pre-existing per-device test goes red against the old driver, which is the regression proof Refs #60
4.9 KiB
type, title, description, tags, verified, timestamp
| type | title | description | tags | verified | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| reference | pcie-probe — link or read path | HIP instruments that separate PCIe link bandwidth from the engine's read path at the real expert-slab size, plus the refuse-by-default guard that stops a benchmark from running beside a live service. |
|
2026-10-21 | 2026-10-20 |
pcie-probe
scripts/pcie-probe.sh answers one question, which comes up every time MoE
expert offload is discussed on the R9700 box
(dankserver): is the stall the PCIe link, or is it
the engine's read path? The two look identical from the outside — decode is
slow while weights stream in — and they have opposite fixes.
Instruments
Sources in scripts/probes/, built with the host hipcc:
| Probe | Question it answers |
|---|---|
hip_smoke.hip |
is host ROCm usable from a plain process at all (count, alloc, copy) |
pcie_bw.hip |
host→device bandwidth at the engine's exact slab size, pinned vs pageable |
pcie_probe.hip |
the size sweep — where throughput collapses as transfers shrink |
scripts/pcie-probe.sh # guard, build, topology, measure
scripts/pcie-probe.sh --device 0 --device 1
scripts/pcie-probe.sh --build-only # compile, never touch the cards
The law: never benchmark beside a live service
The guard runs before anything is built or run and refuses when the GPU is
occupied — gpu_busy_percent above PCIE_PROBE_MAX_BUSY (default 5 %) or any
process holding /dev/kfd. Refusal is exit 3, and it stops nothing. If the holder
probe is missing the guard refuses too rather than assuming idle — see
seams; gpu_busy_percent reads 0 for a service that is
resident and idle, so a silent fallback is exactly the benchmark this law forbids.
Auto-stopping a production endpoint is not this tool's decision to make, and a
measurement taken as noise beside serving traffic is worthless anyway: it is
both disruptive and wrong. Take the service down deliberately, or pass --force
and mean it. tests/pcie-probe.bats pins that the refusal happens before the
build step, so a contended box cannot be measured by accident.
Reading the numbers
The reference is the link, printed as a topology block from sysfs
(current_link_width / current_link_speed against max_*): a card trained to
fewer lanes or a lower generation than it is wired for caps everything
downstream, and no engine change will recover it.
Then the sweep table, whose first row is the engine's actual expert-slab size — throughput at that size is what matters, not the big-transfer figure:
- Small transfers reach the trained-link ceiling → the link is healthy, the engine's read path (fetch granularity, pipelining, pinning) is the limit.
- Small transfers stall far below a large-transfer figure → the wall is the link or the IOMMU path, not the engine.
- Pinned versus pageable columns apart means pinning is load-bearing; together means the copies are already pinned by someone else.
Measure one device at a time (--device, a HIP ordinal). A run that lets the runtime
pick several devices cannot attribute bandwidth to a card at all. The ordinal goes to
the probe as an argument and the driver never sets HIP_VISIBLE_DEVICES: masking
the device and passing its real ordinal asks hipSetDevice for a card the runtime
has hidden, so every card but ordinal 0 fails to measure at all. One selector only.
The trap carried in the sources
hipGetDeviceProperties segfaults on the ROCm build these probes were
written against, while hipGetDeviceCount works fine and reports every device.
None of the three sources calls it; device identity comes from sysfs instead.
The setvbuf call in hip_smoke.hip is a fossil of discovering that — an
unflushed stdout hid the crash point for an afternoon. Keep both facts intact
when touching these files.
Exit status and seams
0 measured · 2 usage or no sources · 3 refused, GPU busy or holders unknowable ·
4 no HIP toolchain or build failed · 5 a probe ran and failed. A client box without
the SDK exits 4 loudly rather than reporting an empty success; the probes belong on
the box with the cards. A probe that dies (the segfault above, an unfundable
allocation) exits 5 and names which probe on which device — a sweep that quietly
swallowed the failure would report success with half its rows missing.
Test seams (all documented in the script header): PCIE_PROBE_SRC,
PCIE_PROBE_BUILD, PCIE_PROBE_SYSFS, PCIE_PROBE_KFD, PCIE_PROBE_HIPCC,
PCIE_PROBE_FUSER, PCIE_PROBE_MAX_BUSY — the sysfs/kfd paths are what make a
refusal testable on a machine with no cards, which is the only honest way to test a
refusal.
Measured campaign numbers are not kept here; they belong with the endpoint they were taken on, in the serverAppConfigs LLM runbook vault.