Frank
8393dc1090
`status` built its job list with `for n in $names`, which word-splits AND globs. The second half is the bug: a job named `x*` was expanded against the caller's working directory, so the report described unrelated files there as jobs and never mentioned the one that ran — on a harness whose whole purpose is a report you can trust after the session that started the job is gone. Splitting stays (several names in one argument is the documented call shape); expansion goes. `tail` was honest-by-accident in the wrong direction: with no jobs it handed a literal `$JOBS/*.out` to tail, which reports a missing *file* — indistinguishable from a broken tool — and a missing name produced tail's errno rather than "no job called that". Both now say what is missing and exit 1. Red first: the name test reads the report from a directory containing an unrelated file, so the old glob is visible as a phantom job; both new tests fail against the previous code and pass against this one. Refs #60
3.6 KiB
type, title, description, tags, verified, timestamp
| type | title | description | tags | verified | timestamp | ||||
|---|---|---|---|---|---|---|---|---|---|
| reference | remote-job — detached jobs that outlive the session | Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished. |
|
2026-10-20 | 2026-10-20 |
remote-job
scripts/remote-job.sh starts a command detached on the host, and reports
what actually happened from files on disk. It exists because
ssh host 'cmd | tail -f log | sed -n /ready/q' lied repeatedly during the
worktree campaign.
Why the pipe lies
Three independent failure modes, all of them structural:
- The broker kills long-lived ssh. A job longer than the connection's patience dies mid-run; the caller sees a closed pipe, not a failure.
- A readiness pattern can match a log header.
sed -n /building/qfires on a banner printed before the work starts — the job is at line 1 and the caller already moved on. sed -n /x/qexits the instant it matches. Even when the pattern is genuinely ready-shaped, the pipeline tears down the remote command it was watching. Result: green report, dead job.
The only reliable signal is a completion sentinel written by the job itself, read afterwards by a short-lived connection. That is what this is.
Contract
remote-job.sh start NAME [--] CMD... # launch detached, return at once
remote-job.sh status [NAME] # one line per job; names literal, never globs
remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines
remote-job.sh out NAME [N] # tail a job's output
remote-job.sh tail [NAME] # follow one job, or all; exit 1 if nothing to follow
remote-job.sh watch # refreshing status, for a spare pane
Job dir $REMOTE_JOBS_DIR (default $HOME/jobs). Per job: NAME.out (stdout+
stderr), NAME.rc (exit code), NAME.done (sentinel, written last).
Exit status of the report: 0 all reported jobs finished · 1 unfinished or
timed out · 2 usage. The rc= column is the job's own code — a job that
failed is still DONE rc=7 and status exits 0, because "is it finished" and
"did it work" are different questions a supervisor must be able to ask
separately.
Written for the case where the job restarts the endpoint the agent itself is
running on: the ssh connection dies, the session may die, and the answer is
still sitting in $JOBS when a session comes back.
Use over ssh
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect
Each ssh call is short, so there is nothing there to abort, and the second call can be made from a fresh session on a machine that rebooted underneath it.
Failure modes
| Event | Result |
|---|---|
| Session/connection dies mid-job | job keeps running; status from anywhere reports it |
| Command name is wrong | PENDING (no output yet), exit 1 — never a false DONE |
| Job exits non-zero | DONE rc=N; the code is data, not the report's status |
| Two jobs, same NAME | the second replaces the first's state files |
tail with no job, or a name that is a glob |
exit 1 saying there is nothing to follow; names are matched literally, never expanded |
| Job dir missing | created on first use |
Pinned by tests/remote-job.bats (black-box: output, exit codes, sentinel
ordering — never internals).