Frank
35ceae6d7b
`ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that never finished, three structural ways: the broker aborts the long connection, a readiness pattern can match a log header printed before the work starts, and `sed -n /x/q` exits the instant it matches — tearing down the very command it was watching. The replacement runs the job detached on the host and writes NAME.out, NAME.rc, then NAME.done LAST, so status is a cheap read of files over a short-lived connection. That survives the connection, the restart, and the job that restarts the endpoint the agent itself is running on. Black-box tests pin the sentinel ordering (rc before done), that a finished-but- failed job is DONE rc=N with the report still exiting 0, and that a name with no state is never reported DONE. Refs #53
3.5 KiB
type, title, description, tags, verified, timestamp
| type | title | description | tags | verified | timestamp | ||||
|---|---|---|---|---|---|---|---|---|---|
| reference | remote-job — detached jobs that outlive the session | Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished. |
|
2026-10-20 | 2026-10-20 |
remote-job
scripts/remote-job.sh starts a command detached on the host, and reports
what actually happened from files on disk. It exists because
ssh host 'cmd | tail -f log | sed -n /ready/q' lied repeatedly during the
worktree campaign.
Why the pipe lies
Three independent failure modes, all of them structural:
- The broker kills long-lived ssh. A job longer than the connection's patience dies mid-run; the caller sees a closed pipe, not a failure.
- A readiness pattern can match a log header.
sed -n /building/qfires on a banner printed before the work starts — the job is at line 1 and the caller already moved on. sed -n /x/qexits the instant it matches. Even when the pattern is genuinely ready-shaped, the pipeline tears down the remote command it was watching. Result: green report, dead job.
The only reliable signal is a completion sentinel written by the job itself, read afterwards by a short-lived connection. That is what this is.
Contract
remote-job.sh start NAME [--] CMD... # launch detached, return at once
remote-job.sh status [NAME] # one line per job; exit 0 only if all DONE
remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines
remote-job.sh out NAME [N] # tail a job's output
remote-job.sh tail [NAME] # follow one job, or every job at once
remote-job.sh watch # refreshing status, for a spare pane
Job dir $REMOTE_JOBS_DIR (default $HOME/jobs). Per job: NAME.out (stdout+
stderr), NAME.rc (exit code), NAME.done (sentinel, written last).
Exit status of the report: 0 all reported jobs finished · 1 unfinished or
timed out · 2 usage. The rc= column is the job's own code — a job that
failed is still DONE rc=7 and status exits 0, because "is it finished" and
"did it work" are different questions a supervisor must be able to ask
separately.
Written for the case where the job restarts the endpoint the agent itself is
running on: the ssh connection dies, the session may die, and the answer is
still sitting in $JOBS when a session comes back.
Use over ssh
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect
Each ssh call is short, so there is nothing there to abort, and the second call can be made from a fresh session on a machine that rebooted underneath it.
Failure modes
| Event | Result |
|---|---|
| Session/connection dies mid-job | job keeps running; status from anywhere reports it |
| Command name is wrong | PENDING (no output yet), exit 1 — never a false DONE |
| Job exits non-zero | DONE rc=N; the code is data, not the report's status |
| Two jobs, same NAME | the second replaces the first's state files |
| Job dir missing | created on first use |
Pinned by tests/remote-job.bats (black-box: output, exit codes, sentinel
ordering — never internals).