---
type: reference
title: pcie-probe — link or read path
description: HIP instruments that separate PCIe link bandwidth from the engine's read path at the real expert-slab size, plus the refuse-by-default guard that stops a benchmark from running beside a live service.
tags:
  - gpu
  - rocm
  - rdna4
  - benchmarking
  - dankserver
verified: 2026-10-21
timestamp: 2026-10-20
---

# pcie-probe

`scripts/pcie-probe.sh` answers one question, which comes up every time MoE
expert offload is discussed on the R9700 box
([dankserver](../system/dankserver.md)): **is the stall the PCIe link, or is it
the engine's read path?** The two look identical from the outside — decode is
slow while weights stream in — and they have opposite fixes.

## Instruments

Sources in `scripts/probes/`, built with the host `hipcc`:

| Probe | Question it answers |
|-------|---------------------|
| `hip_smoke.hip` | is host ROCm usable from a plain process at all (count, alloc, copy) |
| `pcie_bw.hip` | host→device bandwidth at the engine's exact slab size, pinned vs pageable |
| `pcie_probe.hip` | the size sweep — where throughput collapses as transfers shrink |

```bash
scripts/pcie-probe.sh                      # guard, build, topology, measure
scripts/pcie-probe.sh --device 0 --device 1
scripts/pcie-probe.sh --build-only         # compile, never touch the cards
```

## The law: never benchmark beside a live service

The guard runs **before anything is built or run** and refuses when the GPU is
occupied — `gpu_busy_percent` above `PCIE_PROBE_MAX_BUSY` (default 5 %) or any
process holding `/dev/kfd`. Refusal is exit `3`, and it stops nothing. If the holder
probe is missing the guard refuses too rather than assuming idle — see
[seams](#exit-status-and-seams); `gpu_busy_percent` reads `0` for a service that is
resident and idle, so a silent fallback is exactly the benchmark this law forbids.

Auto-stopping a production endpoint is not this tool's decision to make, and a
measurement taken as noise beside serving traffic is worthless anyway: it is
both disruptive and wrong. Take the service down deliberately, or pass `--force`
and mean it. `tests/pcie-probe.bats` pins that the refusal happens before the
build step, so a contended box cannot be measured by accident.

## Reading the numbers

The reference is the **link**, printed as a topology block from sysfs
(`current_link_width` / `current_link_speed` against `max_*`): a card trained to
fewer lanes or a lower generation than it is wired for caps everything
downstream, and no engine change will recover it.

Then the sweep table, whose first row is the engine's actual expert-slab size —
throughput at *that* size is what matters, not the big-transfer figure:

- Small transfers reach the trained-link ceiling → the link is healthy, the
  engine's read path (fetch granularity, pipelining, pinning) is the limit.
- Small transfers stall far below a large-transfer figure → the wall is the
  link or the IOMMU path, not the engine.
- Pinned versus pageable columns apart means pinning is load-bearing; together
  means the copies are already pinned by someone else.

Measure one device at a time (`--device`, a HIP ordinal). A run that lets the runtime
pick several devices cannot attribute bandwidth to a card at all. The ordinal goes to
the probe as an argument and **the driver never sets `HIP_VISIBLE_DEVICES`**: masking
the device *and* passing its real ordinal asks `hipSetDevice` for a card the runtime
has hidden, so every card but ordinal 0 fails to measure at all. One selector only.

## The trap carried in the sources

`hipGetDeviceProperties` **segfaults** on the ROCm build these probes were
written against, while `hipGetDeviceCount` works fine and reports every device.
None of the three sources calls it; device identity comes from sysfs instead.
The setvbuf call in `hip_smoke.hip` is a fossil of discovering that — an
unflushed stdout hid the crash point for an afternoon. Keep both facts intact
when touching these files.

## Exit status and seams

`0` measured · `2` usage or no sources · `3` refused, GPU busy or holders unknowable ·
`4` no HIP toolchain or build failed · `5` a probe ran and failed. A client box without
the SDK exits `4` loudly rather than reporting an empty success; the probes belong on
the box with the cards. A probe that dies (the segfault above, an unfundable
allocation) exits `5` and names which probe on which device — a sweep that quietly
swallowed the failure would report success with half its rows missing.

Test seams (all documented in the script header): `PCIE_PROBE_SRC`,
`PCIE_PROBE_BUILD`, `PCIE_PROBE_SYSFS`, `PCIE_PROBE_KFD`, `PCIE_PROBE_HIPCC`,
`PCIE_PROBE_FUSER`, `PCIE_PROBE_MAX_BUSY` — the sysfs/kfd paths are what make a
refusal testable on a machine with no cards, which is the only honest way to test a
refusal.

Measured campaign numbers are not kept here; they belong with the endpoint they
were taken on, in the serverAppConfigs LLM runbook vault.