Ship the PCIe probe instruments with a guard that refuses to contend with a service

The RDNA4 offload campaign left three HIP probes and a driver sitting loose in
$HOME, where the next agent cannot find them and a stray `rm -rf` ends them. They
answer a recurring question on the R9700 box — is decode limited by the link or by
the engine's read path — so they move into scripts/probes/ with a driver, tests,
and a page.

Two things changed in the code, not just the address:

- The driver measures only after a guard that runs BEFORE anything is built:
  gpu_busy_percent over PCIE_PROBE_MAX_BUSY, or any process holding /dev/kfd, is a
  refusal (exit 3) rather than a measurement. A benchmark taken beside a live
  endpoint is both disruptive and worthless, and silently unloading a production
  service is not a tool's decision — --force is the human saying otherwise. The
  refusal-before-build ordering is pinned by a test, because the sysfs and kfd
  paths are overridable and a refusal is otherwise untestable off the card box.
- hip_smoke.hip and pcie_probe.hip called hipGetDeviceProperties, which SEGFAULTS
  on this ROCm build even though hipGetDeviceCount works; only pcie_bw.hip avoided
  it, and that is the one that produced the campaign's numbers. Device identity now
  comes from the sysfs topology block the driver prints, so all three probes are
  runnable on the build we ship. Verified here: refusal on a real busy APU (exit 3),
  missing toolchain (exit 4), missing sources (exit 2). Compile-and-run still needs
  the box with hipcc.

Refs #53

omp-docs/log.md

@@ -16,6 +16,29 @@ lesson, a few lines. Full incident narratives are recoverable from git
 (`git log --follow omp-docs/log.md`); knowledge that outlives an entry lives
 on its concept page, linked here. Never re-expand an entry in place.
 
+## 2026-10-20 — two instruments rescued from `$HOME` script sediment
+
+`$HOME` had accumulated 123 loose scripts at its root (archived to
+`.AGENTTRASH/home-scripts-2026-10-20`, ~1.8 GiB of stray audio consolidated into
+`~/audiosamples`). Two were durable instruments and now live in the repo with
+black-box tests and pages: [tools/remote-job](tools/remote-job.md) and
+[tools/pcie-probe](tools/pcie-probe.md). Refs #53. Laws learned:
+
+- **A readiness pattern in a log is not evidence of completion.** `ssh host 'cmd |
+  tail -f log | sed -n /ready/q'` fails three independent ways — the broker aborts
+  the long connection, the pattern can match a header printed before the work
+  starts, and `sed -n /x/q` exits the moment it matches, tearing down the job it was
+  watching. The only honest signal is a sentinel the job writes last, read over a
+  short-lived connection.
+- **Never benchmark beside a live service, and never let a tool decide to stop one.**
+  The guard refuses before anything is built; unloading a production endpoint is a
+  human's call. A number taken as noise beside serving traffic is both disruptive and
+  wrong.
+- **An instrument that segfaults is not an instrument.** Two of the three shipped
+  `.hip` probes called `hipGetDeviceProperties`, which crashes on this ROCm build
+  while `hipGetDeviceCount` works fine; device identity now comes from sysfs, and the
+  trap is recorded in the sources so it is not reintroduced.
+
 ## 2026-10-20 — the secret gate was reading the wrong files, and one credential had already escaped
 
 Found while deciding what else belongs in `dotfiles/`. (1) The gate whitelisted

omp-docs/tools/index.md

@@ -20,6 +20,7 @@ Configuration guides and reference material for installed software. Each tool ha
 * [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation.
 * [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management.
 * [packages](packages.md) — `packages/*.pkgs` desired state and the two tools that keep a machine and its manifest agreeing: `install.sh` installs the delta, `pkg-inventory.sh` reports what the machine has outgrown.
+* [pcie-probe](pcie-probe.md) — HIP probes that separate PCIe link bandwidth from the engine's read path at the expert-slab size, and the refuse-by-default guard against benchmarking beside a live service.
 * [remote-job](remote-job.md) — Detached job runner for `ssh` work: completion sentinels on the host replace the `tail -f | sed -n /ready/q` pattern that reported success on jobs that never finished.
 * [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures.
 

omp-docs/tools/pcie-probe.md

@@ -0,0 +1,92 @@
+---
+type: reference
+title: pcie-probe — link or read path
+description: HIP instruments that separate PCIe link bandwidth from the engine's read path at the real expert-slab size, plus the refuse-by-default guard that stops a benchmark from running beside a live service.
+tags:
+  - gpu
+  - rocm
+  - rdna4
+  - benchmarking
+  - dankserver
+verified: 2026-10-20
+timestamp: 2026-10-20
+---
+
+# pcie-probe
+
+`scripts/pcie-probe.sh` answers one question, which comes up every time MoE
+expert offload is discussed on the R9700 box
+([dankserver](../system/dankserver.md)): **is the stall the PCIe link, or is it
+the engine's read path?** The two look identical from the outside — decode is
+slow while weights stream in — and they have opposite fixes.
+
+## Instruments
+
+Sources in `scripts/probes/`, built with the host `hipcc`:
+
+| Probe | Question it answers |
+|-------|---------------------|
+| `hip_smoke.hip` | is host ROCm usable from a plain process at all (count, alloc, copy) |
+| `pcie_bw.hip` | host→device bandwidth at the engine's exact slab size, pinned vs pageable |
+| `pcie_probe.hip` | the size sweep — where throughput collapses as transfers shrink |
+
+```bash
+scripts/pcie-probe.sh                      # guard, build, topology, measure
+scripts/pcie-probe.sh --device 0 --device 1
+scripts/pcie-probe.sh --build-only         # compile, never touch the cards
+```
+
+## The law: never benchmark beside a live service
+
+The guard runs **before anything is built or run** and refuses when the GPU is
+occupied — `gpu_busy_percent` above `PCIE_PROBE_MAX_BUSY` (default 5 %) or any
+process holding `/dev/kfd`. Refusal is exit `3`, and it stops nothing.
+
+Auto-stopping a production endpoint is not this tool's decision to make, and a
+measurement taken as noise beside serving traffic is worthless anyway: it is
+both disruptive and wrong. Take the service down deliberately, or pass `--force`
+and mean it. `tests/pcie-probe.bats` pins that the refusal happens before the
+build step, so a contended box cannot be measured by accident.
+
+## Reading the numbers
+
+The reference is the **link**, printed as a topology block from sysfs
+(`current_link_width` / `current_link_speed` against `max_*`): a card trained to
+fewer lanes or a lower generation than it is wired for caps everything
+downstream, and no engine change will recover it.
+
+Then the sweep table, whose first row is the engine's actual expert-slab size —
+throughput at *that* size is what matters, not the big-transfer figure:
+
+- Small transfers reach the trained-link ceiling → the link is healthy, the
+  engine's read path (fetch granularity, pipelining, pinning) is the limit.
+- Small transfers stall far below a large-transfer figure → the wall is the
+  link or the IOMMU path, not the engine.
+- Pinned versus pageable columns apart means pinning is load-bearing; together
+  means the copies are already pinned by someone else.
+
+Measure one device at a time (`--device`). A run that lets the runtime pick
+several devices cannot attribute bandwidth to a card at all.
+
+## The trap carried in the sources
+
+`hipGetDeviceProperties` **segfaults** on the ROCm build these probes were
+written against, while `hipGetDeviceCount` works fine and reports every device.
+None of the three sources calls it; device identity comes from sysfs instead.
+The setvbuf call in `hip_smoke.hip` is a fossil of discovering that — an
+unflushed stdout hid the crash point for an afternoon. Keep both facts intact
+when touching these files.
+
+## Exit status and seams
+
+`0` measured · `2` usage or no sources · `3` refused, GPU busy · `4` no HIP
+toolchain or build failed. A client box without the SDK exits `4` loudly rather
+than reporting an empty success; the probes belong on the box with the cards.
+
+Test seams (all documented in the script header): `PCIE_PROBE_SRC`,
+`PCIE_PROBE_BUILD`, `PCIE_PROBE_SYSFS`, `PCIE_PROBE_KFD`, `PCIE_PROBE_HIPCC`,
+`PCIE_PROBE_MAX_BUSY` — the sysfs/kfd paths are what make a refusal testable on
+a machine with no cards, which is the only honest way to test a refusal.
+
+Measured campaign numbers are not kept here; they belong with the endpoint they
+were taken on, in the serverAppConfigs LLM runbook vault.