Frank
71a7e5cb9a
Per-card measurement was unreachable for every card but ordinal 0: the driver set HIP_VISIBLE_DEVICES=N and still passed N as the device argument, so the probe called hipSetDevice(N) on a runtime where the mask left only ordinal 0. The bats stub reported the env var rather than the argument it received, so the suite asserted the mask was set and never noticed the collision. - one selector: the ordinal is passed as argv, the mask is never set - a probe that fails is now exit 5 and names probe + device; the sweep still measures the remaining cards instead of swallowing the failure - the guard fails closed when `fuser` is missing (gpu_busy_percent reads 0 for a resident-but-idle service), with PCIE_PROBE_FUSER as the test seam - stub probes now model the HIP mask rule; the pre-existing per-device test goes red against the old driver, which is the regression proof Refs #60
103 lines
4.9 KiB
Markdown
103 lines
4.9 KiB
Markdown
---
|
|
type: reference
|
|
title: pcie-probe — link or read path
|
|
description: HIP instruments that separate PCIe link bandwidth from the engine's read path at the real expert-slab size, plus the refuse-by-default guard that stops a benchmark from running beside a live service.
|
|
tags:
|
|
- gpu
|
|
- rocm
|
|
- rdna4
|
|
- benchmarking
|
|
- dankserver
|
|
verified: 2026-10-21
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# pcie-probe
|
|
|
|
`scripts/pcie-probe.sh` answers one question, which comes up every time MoE
|
|
expert offload is discussed on the R9700 box
|
|
([dankserver](../system/dankserver.md)): **is the stall the PCIe link, or is it
|
|
the engine's read path?** The two look identical from the outside — decode is
|
|
slow while weights stream in — and they have opposite fixes.
|
|
|
|
## Instruments
|
|
|
|
Sources in `scripts/probes/`, built with the host `hipcc`:
|
|
|
|
| Probe | Question it answers |
|
|
|-------|---------------------|
|
|
| `hip_smoke.hip` | is host ROCm usable from a plain process at all (count, alloc, copy) |
|
|
| `pcie_bw.hip` | host→device bandwidth at the engine's exact slab size, pinned vs pageable |
|
|
| `pcie_probe.hip` | the size sweep — where throughput collapses as transfers shrink |
|
|
|
|
```bash
|
|
scripts/pcie-probe.sh # guard, build, topology, measure
|
|
scripts/pcie-probe.sh --device 0 --device 1
|
|
scripts/pcie-probe.sh --build-only # compile, never touch the cards
|
|
```
|
|
|
|
## The law: never benchmark beside a live service
|
|
|
|
The guard runs **before anything is built or run** and refuses when the GPU is
|
|
occupied — `gpu_busy_percent` above `PCIE_PROBE_MAX_BUSY` (default 5 %) or any
|
|
process holding `/dev/kfd`. Refusal is exit `3`, and it stops nothing. If the holder
|
|
probe is missing the guard refuses too rather than assuming idle — see
|
|
[seams](#exit-status-and-seams); `gpu_busy_percent` reads `0` for a service that is
|
|
resident and idle, so a silent fallback is exactly the benchmark this law forbids.
|
|
|
|
Auto-stopping a production endpoint is not this tool's decision to make, and a
|
|
measurement taken as noise beside serving traffic is worthless anyway: it is
|
|
both disruptive and wrong. Take the service down deliberately, or pass `--force`
|
|
and mean it. `tests/pcie-probe.bats` pins that the refusal happens before the
|
|
build step, so a contended box cannot be measured by accident.
|
|
|
|
## Reading the numbers
|
|
|
|
The reference is the **link**, printed as a topology block from sysfs
|
|
(`current_link_width` / `current_link_speed` against `max_*`): a card trained to
|
|
fewer lanes or a lower generation than it is wired for caps everything
|
|
downstream, and no engine change will recover it.
|
|
|
|
Then the sweep table, whose first row is the engine's actual expert-slab size —
|
|
throughput at *that* size is what matters, not the big-transfer figure:
|
|
|
|
- Small transfers reach the trained-link ceiling → the link is healthy, the
|
|
engine's read path (fetch granularity, pipelining, pinning) is the limit.
|
|
- Small transfers stall far below a large-transfer figure → the wall is the
|
|
link or the IOMMU path, not the engine.
|
|
- Pinned versus pageable columns apart means pinning is load-bearing; together
|
|
means the copies are already pinned by someone else.
|
|
|
|
Measure one device at a time (`--device`, a HIP ordinal). A run that lets the runtime
|
|
pick several devices cannot attribute bandwidth to a card at all. The ordinal goes to
|
|
the probe as an argument and **the driver never sets `HIP_VISIBLE_DEVICES`**: masking
|
|
the device *and* passing its real ordinal asks `hipSetDevice` for a card the runtime
|
|
has hidden, so every card but ordinal 0 fails to measure at all. One selector only.
|
|
|
|
## The trap carried in the sources
|
|
|
|
`hipGetDeviceProperties` **segfaults** on the ROCm build these probes were
|
|
written against, while `hipGetDeviceCount` works fine and reports every device.
|
|
None of the three sources calls it; device identity comes from sysfs instead.
|
|
The setvbuf call in `hip_smoke.hip` is a fossil of discovering that — an
|
|
unflushed stdout hid the crash point for an afternoon. Keep both facts intact
|
|
when touching these files.
|
|
|
|
## Exit status and seams
|
|
|
|
`0` measured · `2` usage or no sources · `3` refused, GPU busy or holders unknowable ·
|
|
`4` no HIP toolchain or build failed · `5` a probe ran and failed. A client box without
|
|
the SDK exits `4` loudly rather than reporting an empty success; the probes belong on
|
|
the box with the cards. A probe that dies (the segfault above, an unfundable
|
|
allocation) exits `5` and names which probe on which device — a sweep that quietly
|
|
swallowed the failure would report success with half its rows missing.
|
|
|
|
Test seams (all documented in the script header): `PCIE_PROBE_SRC`,
|
|
`PCIE_PROBE_BUILD`, `PCIE_PROBE_SYSFS`, `PCIE_PROBE_KFD`, `PCIE_PROBE_HIPCC`,
|
|
`PCIE_PROBE_FUSER`, `PCIE_PROBE_MAX_BUSY` — the sysfs/kfd paths are what make a
|
|
refusal testable on a machine with no cards, which is the only honest way to test a
|
|
refusal.
|
|
|
|
Measured campaign numbers are not kept here; they belong with the endpoint they
|
|
were taken on, in the serverAppConfigs LLM runbook vault.
|