Rescue two durable instruments from $HOME script sediment
Clearing 106 loose scripts from $HOME (archived to ~/.AGENTTRASH/home-scripts-2026-10-20/,
123 files with their logs) left two things that are not campaign scaffolding and have nowhere to
live. Everything else in that pile is either the abandoned GLM-5.3-on-vLLM-0.28 port (92 files,
zero mentions in the KB or vault, targets the sold dankware box) or superseded bench probes.
1. The detached job runner (job.sh)
Written to kill the ssh host 'tail -f | sed' pattern, which lied about success for three
structural reasons: the broker aborts long-lived ssh, a readiness pattern can match a log header
instantly, and a sed filter exits the moment it matches — so the supervisor reported a green run
that had not happened. The replacement runs the job detached on the host, writes output, then the
exit code, then touches a .done sentinel last, so status is a fast independent ls and the
result survives the session. That is a fleet-wide pattern, not a Dankserver one. Only the hardcoded /home/dank/jobs directory ties it to one box.
2. The RDNA4 PCIe/HIP probes (pcie_probe.hip, pcie_bw.hip, hip_smoke.hip)
hip_smoke.hip answers "is the host ROCm usable from a plain process"; pcie_bw.hip measures
host-to-device bandwidth at the engine's actual expert-slab size; pcie_probe.hip answers the
question that decides tuning direction — is the decode bottleneck the LINK or the engine's read
PATH? Same GPUs are in Dankserver today, so this is a live instrument, not archaeology.
Two traps are encoded in the source comments and must survive the move:
hipGetDeviceProperties segfaults on this ROCm build while hipGetDeviceCount works fine and
reports three devices. The probes deliberately avoid the former.
Never benchmark beside a live service. The original driver stopped the serving container,
measured, and relaunched it — with hardcoded campaign paths.
Shape
scripts/remote-job.sh — start|status|wait|out|tail|watch, job dir from REMOTE_JOBS_DIR
(default $HOME/jobs). status/wait exit non-zero while a job is unfinished, so a supervisor
can actually branch on it. The sentinel ordering (.rc before .done) is the contract.
scripts/pcie-probe.sh + scripts/probes/*.hip — builds with hipcc, prints per-device results
plus the sysfs link topology, and refuses to measure while the GPU is held (/dev/kfd in use
or gpu_busy_percent above a threshold). It stops nothing and starts nothing: auto-stopping a
production service is exactly what the tool must not do unasked. Missing hipcc is a loud exit,
not a silent skip.
Tests: tests/remote-job.bats, tests/pcie-probe.bats — real local job lifecycles (a job that
exits non-zero must report that code; .done must never exist without a readable .rc; a bounded wait must time out) and stubbed hipcc/systemctl/sysfs for the probe guards. Verified on
blankpad for the guard paths; the HIP build itself runs only on a box with the SDK.
# Rescue two durable instruments from $HOME script sediment
Clearing 106 loose scripts from `$HOME` (archived to `~/.AGENTTRASH/home-scripts-2026-10-20/`,
123 files with their logs) left two things that are not campaign scaffolding and have nowhere to
live. Everything else in that pile is either the abandoned GLM-5.3-on-vLLM-0.28 port (92 files,
zero mentions in the KB or vault, targets the sold `dankware` box) or superseded bench probes.
## 1. The detached job runner (`job.sh`)
Written to kill the `ssh host 'tail -f | sed'` pattern, which lied about success for three
structural reasons: the broker aborts long-lived ssh, a readiness pattern can match a log header
instantly, and a `sed` filter exits the moment it matches — so the supervisor reported a green run
that had not happened. The replacement runs the job detached on the host, writes output, then the
exit code, then touches a `.done` sentinel **last**, so status is a fast independent `ls` and the
result survives the session. That is a fleet-wide pattern, not a Dankserver one. Only the hardcoded
`/home/dank/jobs` directory ties it to one box.
## 2. The RDNA4 PCIe/HIP probes (`pcie_probe.hip`, `pcie_bw.hip`, `hip_smoke.hip`)
`hip_smoke.hip` answers "is the host ROCm usable from a plain process"; `pcie_bw.hip` measures
host-to-device bandwidth at the engine's actual expert-slab size; `pcie_probe.hip` answers the
question that decides tuning direction — *is the decode bottleneck the LINK or the engine's read
PATH?* Same GPUs are in Dankserver today, so this is a live instrument, not archaeology.
Two traps are encoded in the source comments and must survive the move:
- **`hipGetDeviceProperties` segfaults on this ROCm build** while `hipGetDeviceCount` works fine and
reports three devices. The probes deliberately avoid the former.
- **Never benchmark beside a live service.** The original driver stopped the serving container,
measured, and relaunched it — with hardcoded campaign paths.
## Shape
- `scripts/remote-job.sh` — `start|status|wait|out|tail|watch`, job dir from `REMOTE_JOBS_DIR`
(default `$HOME/jobs`). `status`/`wait` exit non-zero while a job is unfinished, so a supervisor
can actually branch on it. The sentinel ordering (`.rc` before `.done`) is the contract.
- `scripts/pcie-probe.sh` + `scripts/probes/*.hip` — builds with `hipcc`, prints per-device results
plus the sysfs link topology, and **refuses to measure while the GPU is held** (`/dev/kfd` in use
or `gpu_busy_percent` above a threshold). It stops nothing and starts nothing: auto-stopping a
production service is exactly what the tool must not do unasked. Missing `hipcc` is a loud exit,
not a silent skip.
Tests: `tests/remote-job.bats`, `tests/pcie-probe.bats` — real local job lifecycles (a job that
exits non-zero must report that code; `.done` must never exist without a readable `.rc`; a bounded
`wait` must time out) and stubbed `hipcc`/`systemctl`/sysfs for the probe guards. Verified on
blankpad for the guard paths; the HIP build itself runs only on a box with the SDK.
---
cwd: /home/blank/.omp/wt/wt-20261019-125509-7c8a99d · session: 01a155bb-6474-7366-be9d-629f74af0e05
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Rescue two durable instruments from $HOME script sediment
Clearing 106 loose scripts from
$HOME(archived to~/.AGENTTRASH/home-scripts-2026-10-20/,123 files with their logs) left two things that are not campaign scaffolding and have nowhere to
live. Everything else in that pile is either the abandoned GLM-5.3-on-vLLM-0.28 port (92 files,
zero mentions in the KB or vault, targets the sold
dankwarebox) or superseded bench probes.1. The detached job runner (
job.sh)Written to kill the
ssh host 'tail -f | sed'pattern, which lied about success for threestructural reasons: the broker aborts long-lived ssh, a readiness pattern can match a log header
instantly, and a
sedfilter exits the moment it matches — so the supervisor reported a green runthat had not happened. The replacement runs the job detached on the host, writes output, then the
exit code, then touches a
.donesentinel last, so status is a fast independentlsand theresult survives the session. That is a fleet-wide pattern, not a Dankserver one. Only the hardcoded
/home/dank/jobsdirectory ties it to one box.2. The RDNA4 PCIe/HIP probes (
pcie_probe.hip,pcie_bw.hip,hip_smoke.hip)hip_smoke.hipanswers "is the host ROCm usable from a plain process";pcie_bw.hipmeasureshost-to-device bandwidth at the engine's actual expert-slab size;
pcie_probe.hipanswers thequestion that decides tuning direction — is the decode bottleneck the LINK or the engine's read
PATH? Same GPUs are in Dankserver today, so this is a live instrument, not archaeology.
Two traps are encoded in the source comments and must survive the move:
hipGetDevicePropertiessegfaults on this ROCm build whilehipGetDeviceCountworks fine andreports three devices. The probes deliberately avoid the former.
measured, and relaunched it — with hardcoded campaign paths.
Shape
scripts/remote-job.sh—start|status|wait|out|tail|watch, job dir fromREMOTE_JOBS_DIR(default
$HOME/jobs).status/waitexit non-zero while a job is unfinished, so a supervisorcan actually branch on it. The sentinel ordering (
.rcbefore.done) is the contract.scripts/pcie-probe.sh+scripts/probes/*.hip— builds withhipcc, prints per-device resultsplus the sysfs link topology, and refuses to measure while the GPU is held (
/dev/kfdin useor
gpu_busy_percentabove a threshold). It stops nothing and starts nothing: auto-stopping aproduction service is exactly what the tool must not do unasked. Missing
hipccis a loud exit,not a silent skip.
Tests:
tests/remote-job.bats,tests/pcie-probe.bats— real local job lifecycles (a job thatexits non-zero must report that code;
.donemust never exist without a readable.rc; a boundedwaitmust time out) and stubbedhipcc/systemctl/sysfs for the probe guards. Verified onblankpad for the guard paths; the HIP build itself runs only on a box with the SDK.
cwd: /home/blank/.omp/wt/wt-20261019-125509-7c8a99d · session: 01a155bb-6474-7366-be9d-629f74af0e05