Add remote-job.sh: a detached job whose result outlives the session

`ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that
never finished, three structural ways: the broker aborts the long connection, a
readiness pattern can match a log header printed before the work starts, and
`sed -n /x/q` exits the instant it matches — tearing down the very command it
was watching. The replacement runs the job detached on the host and writes
NAME.out, NAME.rc, then NAME.done LAST, so status is a cheap read of files over
a short-lived connection. That survives the connection, the restart, and the job
that restarts the endpoint the agent itself is running on.

Black-box tests pin the sentinel ordering (rc before done), that a finished-but-
failed job is DONE rc=N with the report still exiting 0, and that a name with no
state is never reported DONE.

Refs #53

omp-docs/tools/remote-job.md

@@ -0,0 +1,83 @@
+---
+type: reference
+title: remote-job — detached jobs that outlive the session
+description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished.
+tags:
+  - ssh
+  - fleet
+  - automation
+  - verification
+verified: 2026-10-20
+timestamp: 2026-10-20
+---
+
+# remote-job
+
+`scripts/remote-job.sh` starts a command **detached on the host**, and reports
+what actually happened from files on disk. It exists because
+`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the
+worktree campaign.
+
+## Why the pipe lies
+
+Three independent failure modes, all of them structural:
+
+- **The broker kills long-lived ssh.** A job longer than the connection's
+  patience dies mid-run; the caller sees a closed pipe, not a failure.
+- **A readiness pattern can match a log header.** `sed -n /building/q` fires on
+  a banner printed before the work starts — the job is at line 1 and the caller
+  already moved on.
+- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is
+  genuinely ready-shaped, the pipeline tears down the remote command it was
+  watching. Result: green report, dead job.
+
+The only reliable signal is a **completion sentinel written by the job itself**,
+read afterwards by a short-lived connection. That is what this is.
+
+## Contract
+
+```bash
+remote-job.sh start NAME [--] CMD...   # launch detached, return at once
+remote-job.sh status [NAME]            # one line per job; exit 0 only if all DONE
+remote-job.sh wait NAME [SEC] [N]      # block until DONE or timeout, then tail N lines
+remote-job.sh out NAME [N]             # tail a job's output
+remote-job.sh tail [NAME]              # follow one job, or every job at once
+remote-job.sh watch                    # refreshing status, for a spare pane
+```
+
+Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+
+stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**).
+
+Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or
+timed out · `2` usage. The `rc=` column is the job's own code — a job that
+failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and
+"did it work" are different questions a supervisor must be able to ask
+separately.
+
+Written for the case where the job **restarts the endpoint the agent itself is
+running on**: the ssh connection dies, the session may die, and the answer is
+still sitting in `$JOBS` when a session comes back.
+
+## Use over ssh
+
+```bash
+ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
+  bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
+ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20   # short-lived reconnect
+```
+
+Each ssh call is short, so there is nothing there to abort, and the second call
+can be made from a fresh session on a machine that rebooted underneath it.
+
+## Failure modes
+
+| Event | Result |
+|-------|--------|
+| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it |
+| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE |
+| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status |
+| Two jobs, same NAME | the second replaces the first's state files |
+| Job dir missing | created on first use |
+
+Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel
+ordering — never internals).