Two durable instruments rescued from the 123 loose scripts that had piled up at $HOME root (rest archived to ~/.AGENTTRASH/home-scripts-2026-10-20; ~1.8 GiB of stray audio consolidated into ~/audiosamples). Refs #53.
scripts/remote-job.sh — the pipe lied; this doesn't
ssh host 'cmd | tail -f log | sed -n /ready/q' reported success on jobs that never finished, three independent structural ways: the broker aborts the long-lived connection; a readiness pattern can match a log header printed before the work starts; and sed -n /x/q exits the instant it matches, tearing down the very command it was watching. The runner starts the job detached on the host and writes NAME.out, NAME.rc, then NAME.donelast, so status is a cheap read of files over a short-lived connection — one that survives the connection, the reboot, and the job that restarts the endpoint this agent is itself running on.
scripts/pcie-probe.sh + scripts/probes/*.hip — measure the link, not the noise
Two real changes, not just an address change:
Refuse by default. The guard runs before anything is built or run: gpu_busy_percent above PCIE_PROBE_MAX_BUSY (5 %) or any process holding /dev/kfd exits 3, and it stops nothing — putting a production endpoint down is a human's decision, made with --force. A measurement taken as noise beside serving traffic is both disruptive and worthless.
The shipped probes could not actually run.hip_smoke.hip and pcie_probe.hip called hipGetDeviceProperties, which SEGFAULTS on this ROCm build while hipGetDeviceCount is fine; only pcie_bw.hip avoided it — and that is the one that produced the campaign's numbers. Device identity now comes from the sysfs topology block the driver prints (current against max link width/speed), which is what the sweep numbers have to be read against anyway.
Verification
22 new black-box tests (tests/remote-job.bats, tests/pcie-probe.bats); scripts/kb-lint.sh = 0 violations.
Correction, same day: this line originally claimed "full bats tests/ = 109 ok, 0 failures". That came from reading a tail window instead of grepping for failures, and it was wrong: tests/gen-keybinds.bats "section comments become headers" is red under LC_ALL=C. Nothing here touched it — the defect pre-dates this PR, and is fixed in the companion PR #55. Re-run after that one lands: LC_ALL=C bats tests/ = 110 ok, 0 failures.
Both tools exercised for real here: remote-job start → RUN 1 lines so far → DONE demo rc=3 → wait exit 0 with the output tail; pcie-probe refused on this laptop's own APU at 32 % busy before building anything (exit 3), refused on a fixture card at 91 %, exit 4 for a missing toolchain, exit 2 for missing sources. The PCIE_PROBE_SYSFS/KFD/HIPCC seams are what make a refusal testable off the card box — the only honest way to test a refusal.
Not done here
Compile-and-run of the three .hip probes needs the box with hipcc (Dankserver). blankpad has no SDK by design, so the driver exits 4 there instead of reporting an empty success. On request I will compile-verify over ssh: scratch directory only, no GPU work, no service touched.
Nothing merged. The $HOME originals were archived, not deleted, per the trash convention.
cwd: /home/blank/.omp/wt/wt-20261019-125509-7c8a99d · session: 01a155bb-6474-7366-be9d-629f74af0e05 (built in the worktree ~/.omp/wt/wt-probes)
Two durable instruments rescued from the 123 loose scripts that had piled up at `$HOME` root (rest archived to `~/.AGENTTRASH/home-scripts-2026-10-20`; ~1.8 GiB of stray audio consolidated into `~/audiosamples`). Refs #53.
## scripts/remote-job.sh — the pipe lied; this doesn't
`ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that never finished, three independent structural ways: the broker aborts the long-lived connection; a readiness pattern can match a log header printed before the work starts; and `sed -n /x/q` exits the instant it matches, tearing down the very command it was watching. The runner starts the job detached on the host and writes `NAME.out`, `NAME.rc`, then `NAME.done` **last**, so status is a cheap read of files over a short-lived connection — one that survives the connection, the reboot, and the job that restarts the endpoint this agent is itself running on.
## scripts/pcie-probe.sh + scripts/probes/*.hip — measure the link, not the noise
Two real changes, not just an address change:
- **Refuse by default.** The guard runs *before anything is built or run*: `gpu_busy_percent` above `PCIE_PROBE_MAX_BUSY` (5 %) or any process holding `/dev/kfd` exits 3, and it stops nothing — putting a production endpoint down is a human's decision, made with `--force`. A measurement taken as noise beside serving traffic is both disruptive and worthless.
- **The shipped probes could not actually run.** `hip_smoke.hip` and `pcie_probe.hip` called `hipGetDeviceProperties`, which SEGFAULTS on this ROCm build while `hipGetDeviceCount` is fine; only `pcie_bw.hip` avoided it — and that is the one that produced the campaign's numbers. Device identity now comes from the sysfs topology block the driver prints (current against max link width/speed), which is what the sweep numbers have to be read against anyway.
## Verification
- 22 new black-box tests (`tests/remote-job.bats`, `tests/pcie-probe.bats`); `scripts/kb-lint.sh` = 0 violations.
- **Correction, same day:** this line originally claimed "full `bats tests/` = 109 ok, 0 failures". That came from reading a `tail` window instead of grepping for failures, and it was wrong: `tests/gen-keybinds.bats` "section comments become headers" is red under `LC_ALL=C`. Nothing here touched it — the defect pre-dates this PR, and is fixed in the companion PR https://gitea.franksdank.cloud/dank/command-center/pulls/55. Re-run after that one lands: `LC_ALL=C bats tests/` = 110 ok, 0 failures.
- Both tools exercised for real here: `remote-job` start → `RUN 1 lines so far` → `DONE demo rc=3` → `wait` exit 0 with the output tail; `pcie-probe` refused on this laptop's own APU at 32 % busy **before building anything** (exit 3), refused on a fixture card at 91 %, exit 4 for a missing toolchain, exit 2 for missing sources. The `PCIE_PROBE_SYSFS/KFD/HIPCC` seams are what make a refusal testable off the card box — the only honest way to test a refusal.
## Not done here
- **Compile-and-run of the three `.hip` probes needs the box with `hipcc`** (Dankserver). blankpad has no SDK by design, so the driver exits 4 there instead of reporting an empty success. On request I will compile-verify over ssh: scratch directory only, no GPU work, no service touched.
- Nothing merged. The `$HOME` originals were archived, not deleted, per the trash convention.
---
cwd: /home/blank/.omp/wt/wt-20261019-125509-7c8a99d · session: 01a155bb-6474-7366-be9d-629f74af0e05 (built in the worktree `~/.omp/wt/wt-probes`)
`ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that
never finished, three structural ways: the broker aborts the long connection, a
readiness pattern can match a log header printed before the work starts, and
`sed -n /x/q` exits the instant it matches — tearing down the very command it
was watching. The replacement runs the job detached on the host and writes
NAME.out, NAME.rc, then NAME.done LAST, so status is a cheap read of files over
a short-lived connection. That survives the connection, the restart, and the job
that restarts the endpoint the agent itself is running on.
Black-box tests pin the sentinel ordering (rc before done), that a finished-but-
failed job is DONE rc=N with the report still exiting 0, and that a name with no
state is never reported DONE.
Refs #53
The RDNA4 offload campaign left three HIP probes and a driver sitting loose in
$HOME, where the next agent cannot find them and a stray `rm -rf` ends them. They
answer a recurring question on the R9700 box — is decode limited by the link or by
the engine's read path — so they move into scripts/probes/ with a driver, tests,
and a page.
Two things changed in the code, not just the address:
- The driver measures only after a guard that runs BEFORE anything is built:
gpu_busy_percent over PCIE_PROBE_MAX_BUSY, or any process holding /dev/kfd, is a
refusal (exit 3) rather than a measurement. A benchmark taken beside a live
endpoint is both disruptive and worthless, and silently unloading a production
service is not a tool's decision — --force is the human saying otherwise. The
refusal-before-build ordering is pinned by a test, because the sysfs and kfd
paths are overridable and a refusal is otherwise untestable off the card box.
- hip_smoke.hip and pcie_probe.hip called hipGetDeviceProperties, which SEGFAULTS
on this ROCm build even though hipGetDeviceCount works; only pcie_bw.hip avoided
it, and that is the one that produced the campaign's numbers. Device identity now
comes from the sysfs topology block the driver prints, so all three probes are
runnable on the build we ship. Verified here: refusal on a real busy APU (exit 3),
missing toolchain (exit 4), missing sources (exit 2). Compile-and-run still needs
the box with hipcc.
Refs #53
dank
merged commit 004962ffce into main2026-10-20 19:39:08 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Two durable instruments rescued from the 123 loose scripts that had piled up at
$HOMEroot (rest archived to~/.AGENTTRASH/home-scripts-2026-10-20; ~1.8 GiB of stray audio consolidated into~/audiosamples). Refs #53.scripts/remote-job.sh — the pipe lied; this doesn't
ssh host 'cmd | tail -f log | sed -n /ready/q'reported success on jobs that never finished, three independent structural ways: the broker aborts the long-lived connection; a readiness pattern can match a log header printed before the work starts; andsed -n /x/qexits the instant it matches, tearing down the very command it was watching. The runner starts the job detached on the host and writesNAME.out,NAME.rc, thenNAME.donelast, so status is a cheap read of files over a short-lived connection — one that survives the connection, the reboot, and the job that restarts the endpoint this agent is itself running on.scripts/pcie-probe.sh + scripts/probes/*.hip — measure the link, not the noise
Two real changes, not just an address change:
gpu_busy_percentabovePCIE_PROBE_MAX_BUSY(5 %) or any process holding/dev/kfdexits 3, and it stops nothing — putting a production endpoint down is a human's decision, made with--force. A measurement taken as noise beside serving traffic is both disruptive and worthless.hip_smoke.hipandpcie_probe.hipcalledhipGetDeviceProperties, which SEGFAULTS on this ROCm build whilehipGetDeviceCountis fine; onlypcie_bw.hipavoided it — and that is the one that produced the campaign's numbers. Device identity now comes from the sysfs topology block the driver prints (current against max link width/speed), which is what the sweep numbers have to be read against anyway.Verification
tests/remote-job.bats,tests/pcie-probe.bats);scripts/kb-lint.sh= 0 violations.bats tests/= 109 ok, 0 failures". That came from reading atailwindow instead of grepping for failures, and it was wrong:tests/gen-keybinds.bats"section comments become headers" is red underLC_ALL=C. Nothing here touched it — the defect pre-dates this PR, and is fixed in the companion PR #55. Re-run after that one lands:LC_ALL=C bats tests/= 110 ok, 0 failures.remote-jobstart →RUN 1 lines so far→DONE demo rc=3→waitexit 0 with the output tail;pcie-proberefused on this laptop's own APU at 32 % busy before building anything (exit 3), refused on a fixture card at 91 %, exit 4 for a missing toolchain, exit 2 for missing sources. ThePCIE_PROBE_SYSFS/KFD/HIPCCseams are what make a refusal testable off the card box — the only honest way to test a refusal.Not done here
.hipprobes needs the box withhipcc(Dankserver). blankpad has no SDK by design, so the driver exits 4 there instead of reporting an empty success. On request I will compile-verify over ssh: scratch directory only, no GPU work, no service touched.$HOMEoriginals were archived, not deleted, per the trash convention.cwd: /home/blank/.omp/wt/wt-20261019-125509-7c8a99d · session: 01a155bb-6474-7366-be9d-629f74af0e05 (built in the worktree
~/.omp/wt/wt-probes)