Ship the PCIe probe instruments with a guard that refuses to contend with a service
The RDNA4 offload campaign left three HIP probes and a driver sitting loose in $HOME, where the next agent cannot find them and a stray `rm -rf` ends them. They answer a recurring question on the R9700 box — is decode limited by the link or by the engine's read path — so they move into scripts/probes/ with a driver, tests, and a page. Two things changed in the code, not just the address: - The driver measures only after a guard that runs BEFORE anything is built: gpu_busy_percent over PCIE_PROBE_MAX_BUSY, or any process holding /dev/kfd, is a refusal (exit 3) rather than a measurement. A benchmark taken beside a live endpoint is both disruptive and worthless, and silently unloading a production service is not a tool's decision — --force is the human saying otherwise. The refusal-before-build ordering is pinned by a test, because the sysfs and kfd paths are overridable and a refusal is otherwise untestable off the card box. - hip_smoke.hip and pcie_probe.hip called hipGetDeviceProperties, which SEGFAULTS on this ROCm build even though hipGetDeviceCount works; only pcie_bw.hip avoided it, and that is the one that produced the campaign's numbers. Device identity now comes from the sysfs topology block the driver prints, so all three probes are runnable on the build we ship. Verified here: refusal on a real busy APU (exit 3), missing toolchain (exit 4), missing sources (exit 2). Compile-and-run still needs the box with hipcc. Refs #53
This commit is contained in:
1 parent
35ceae6d7b
commit
56fc899763
8 files changed
+573
No files matched your search
@@ -16,6 +16,29 @@ lesson, a few lines. Full incident narratives are recoverable from git
|
||||
(`git log --follow omp-docs/log.md`); knowledge that outlives an entry lives
|
||||
on its concept page, linked here. Never re-expand an entry in place.
|
||||
|
||||
## 2026-10-20 — two instruments rescued from `$HOME` script sediment
|
||||
|
||||
`$HOME` had accumulated 123 loose scripts at its root (archived to
|
||||
`.AGENTTRASH/home-scripts-2026-10-20`, ~1.8 GiB of stray audio consolidated into
|
||||
`~/audiosamples`). Two were durable instruments and now live in the repo with
|
||||
black-box tests and pages: [tools/remote-job](tools/remote-job.md) and
|
||||
[tools/pcie-probe](tools/pcie-probe.md). Refs #53. Laws learned:
|
||||
|
||||
- **A readiness pattern in a log is not evidence of completion.** `ssh host 'cmd |
|
||||
tail -f log | sed -n /ready/q'` fails three independent ways — the broker aborts
|
||||
the long connection, the pattern can match a header printed before the work
|
||||
starts, and `sed -n /x/q` exits the moment it matches, tearing down the job it was
|
||||
watching. The only honest signal is a sentinel the job writes last, read over a
|
||||
short-lived connection.
|
||||
- **Never benchmark beside a live service, and never let a tool decide to stop one.**
|
||||
The guard refuses before anything is built; unloading a production endpoint is a
|
||||
human's call. A number taken as noise beside serving traffic is both disruptive and
|
||||
wrong.
|
||||
- **An instrument that segfaults is not an instrument.** Two of the three shipped
|
||||
`.hip` probes called `hipGetDeviceProperties`, which crashes on this ROCm build
|
||||
while `hipGetDeviceCount` works fine; device identity now comes from sysfs, and the
|
||||
trap is recorded in the sources so it is not reintroduced.
|
||||
|
||||
## 2026-10-20 — the secret gate was reading the wrong files, and one credential had already escaped
|
||||
|
||||
Found while deciding what else belongs in `dotfiles/`. (1) The gate whitelisted
|
||||
|
||||
Reference in new issue
Block a user