Frank
35ceae6d7b
`ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that never finished, three structural ways: the broker aborts the long connection, a readiness pattern can match a log header printed before the work starts, and `sed -n /x/q` exits the instant it matches — tearing down the very command it was watching. The replacement runs the job detached on the host and writes NAME.out, NAME.rc, then NAME.done LAST, so status is a cheap read of files over a short-lived connection. That survives the connection, the restart, and the job that restarts the endpoint the agent itself is running on. Black-box tests pin the sentinel ordering (rc before done), that a finished-but- failed job is DONE rc=N with the report still exiting 0, and that a name with no state is never reported DONE. Refs #53
84 lines
3.5 KiB
Markdown
84 lines
3.5 KiB
Markdown
---
|
|
type: reference
|
|
title: remote-job — detached jobs that outlive the session
|
|
description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished.
|
|
tags:
|
|
- ssh
|
|
- fleet
|
|
- automation
|
|
- verification
|
|
verified: 2026-10-20
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# remote-job
|
|
|
|
`scripts/remote-job.sh` starts a command **detached on the host**, and reports
|
|
what actually happened from files on disk. It exists because
|
|
`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the
|
|
worktree campaign.
|
|
|
|
## Why the pipe lies
|
|
|
|
Three independent failure modes, all of them structural:
|
|
|
|
- **The broker kills long-lived ssh.** A job longer than the connection's
|
|
patience dies mid-run; the caller sees a closed pipe, not a failure.
|
|
- **A readiness pattern can match a log header.** `sed -n /building/q` fires on
|
|
a banner printed before the work starts — the job is at line 1 and the caller
|
|
already moved on.
|
|
- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is
|
|
genuinely ready-shaped, the pipeline tears down the remote command it was
|
|
watching. Result: green report, dead job.
|
|
|
|
The only reliable signal is a **completion sentinel written by the job itself**,
|
|
read afterwards by a short-lived connection. That is what this is.
|
|
|
|
## Contract
|
|
|
|
```bash
|
|
remote-job.sh start NAME [--] CMD... # launch detached, return at once
|
|
remote-job.sh status [NAME] # one line per job; exit 0 only if all DONE
|
|
remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines
|
|
remote-job.sh out NAME [N] # tail a job's output
|
|
remote-job.sh tail [NAME] # follow one job, or every job at once
|
|
remote-job.sh watch # refreshing status, for a spare pane
|
|
```
|
|
|
|
Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+
|
|
stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**).
|
|
|
|
Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or
|
|
timed out · `2` usage. The `rc=` column is the job's own code — a job that
|
|
failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and
|
|
"did it work" are different questions a supervisor must be able to ask
|
|
separately.
|
|
|
|
Written for the case where the job **restarts the endpoint the agent itself is
|
|
running on**: the ssh connection dies, the session may die, and the answer is
|
|
still sitting in `$JOBS` when a session comes back.
|
|
|
|
## Use over ssh
|
|
|
|
```bash
|
|
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
|
|
bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
|
|
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect
|
|
```
|
|
|
|
Each ssh call is short, so there is nothing there to abort, and the second call
|
|
can be made from a fresh session on a machine that rebooted underneath it.
|
|
|
|
## Failure modes
|
|
|
|
| Event | Result |
|
|
|-------|--------|
|
|
| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it |
|
|
| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE |
|
|
| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status |
|
|
| Two jobs, same NAME | the second replaces the first's state files |
|
|
| Job dir missing | created on first use |
|
|
|
|
Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel
|
|
ordering — never internals).
|