Frank
56fc899763
The RDNA4 offload campaign left three HIP probes and a driver sitting loose in $HOME, where the next agent cannot find them and a stray `rm -rf` ends them. They answer a recurring question on the R9700 box — is decode limited by the link or by the engine's read path — so they move into scripts/probes/ with a driver, tests, and a page. Two things changed in the code, not just the address: - The driver measures only after a guard that runs BEFORE anything is built: gpu_busy_percent over PCIE_PROBE_MAX_BUSY, or any process holding /dev/kfd, is a refusal (exit 3) rather than a measurement. A benchmark taken beside a live endpoint is both disruptive and worthless, and silently unloading a production service is not a tool's decision — --force is the human saying otherwise. The refusal-before-build ordering is pinned by a test, because the sysfs and kfd paths are overridable and a refusal is otherwise untestable off the card box. - hip_smoke.hip and pcie_probe.hip called hipGetDeviceProperties, which SEGFAULTS on this ROCm build even though hipGetDeviceCount works; only pcie_bw.hip avoided it, and that is the one that produced the campaign's numbers. Device identity now comes from the sysfs topology block the driver prints, so all three probes are runnable on the build we ship. Verified here: refusal on a real busy APU (exit 3), missing toolchain (exit 4), missing sources (exit 2). Compile-and-run still needs the box with hipcc. Refs #53
4.1 KiB
type, title, description, tags, verified, timestamp
| type | title | description | tags | verified | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| reference | pcie-probe — link or read path | HIP instruments that separate PCIe link bandwidth from the engine's read path at the real expert-slab size, plus the refuse-by-default guard that stops a benchmark from running beside a live service. |
|
2026-10-20 | 2026-10-20 |
pcie-probe
scripts/pcie-probe.sh answers one question, which comes up every time MoE
expert offload is discussed on the R9700 box
(dankserver): is the stall the PCIe link, or is it
the engine's read path? The two look identical from the outside — decode is
slow while weights stream in — and they have opposite fixes.
Instruments
Sources in scripts/probes/, built with the host hipcc:
| Probe | Question it answers |
|---|---|
hip_smoke.hip |
is host ROCm usable from a plain process at all (count, alloc, copy) |
pcie_bw.hip |
host→device bandwidth at the engine's exact slab size, pinned vs pageable |
pcie_probe.hip |
the size sweep — where throughput collapses as transfers shrink |
scripts/pcie-probe.sh # guard, build, topology, measure
scripts/pcie-probe.sh --device 0 --device 1
scripts/pcie-probe.sh --build-only # compile, never touch the cards
The law: never benchmark beside a live service
The guard runs before anything is built or run and refuses when the GPU is
occupied — gpu_busy_percent above PCIE_PROBE_MAX_BUSY (default 5 %) or any
process holding /dev/kfd. Refusal is exit 3, and it stops nothing.
Auto-stopping a production endpoint is not this tool's decision to make, and a
measurement taken as noise beside serving traffic is worthless anyway: it is
both disruptive and wrong. Take the service down deliberately, or pass --force
and mean it. tests/pcie-probe.bats pins that the refusal happens before the
build step, so a contended box cannot be measured by accident.
Reading the numbers
The reference is the link, printed as a topology block from sysfs
(current_link_width / current_link_speed against max_*): a card trained to
fewer lanes or a lower generation than it is wired for caps everything
downstream, and no engine change will recover it.
Then the sweep table, whose first row is the engine's actual expert-slab size — throughput at that size is what matters, not the big-transfer figure:
- Small transfers reach the trained-link ceiling → the link is healthy, the engine's read path (fetch granularity, pipelining, pinning) is the limit.
- Small transfers stall far below a large-transfer figure → the wall is the link or the IOMMU path, not the engine.
- Pinned versus pageable columns apart means pinning is load-bearing; together means the copies are already pinned by someone else.
Measure one device at a time (--device). A run that lets the runtime pick
several devices cannot attribute bandwidth to a card at all.
The trap carried in the sources
hipGetDeviceProperties segfaults on the ROCm build these probes were
written against, while hipGetDeviceCount works fine and reports every device.
None of the three sources calls it; device identity comes from sysfs instead.
The setvbuf call in hip_smoke.hip is a fossil of discovering that — an
unflushed stdout hid the crash point for an afternoon. Keep both facts intact
when touching these files.
Exit status and seams
0 measured · 2 usage or no sources · 3 refused, GPU busy · 4 no HIP
toolchain or build failed. A client box without the SDK exits 4 loudly rather
than reporting an empty success; the probes belong on the box with the cards.
Test seams (all documented in the script header): PCIE_PROBE_SRC,
PCIE_PROBE_BUILD, PCIE_PROBE_SYSFS, PCIE_PROBE_KFD, PCIE_PROBE_HIPCC,
PCIE_PROBE_MAX_BUSY — the sysfs/kfd paths are what make a refusal testable on
a machine with no cards, which is the only honest way to test a refusal.
Measured campaign numbers are not kept here; they belong with the endpoint they were taken on, in the serverAppConfigs LLM runbook vault.