---
type: reference
title: remote-job — detached jobs that outlive the session
description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished.
tags:
  - ssh
  - fleet
  - automation
  - verification
verified: 2026-10-20
timestamp: 2026-10-20
---

# remote-job

`scripts/remote-job.sh` starts a command **detached on the host**, and reports
what actually happened from files on disk. It exists because
`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the
worktree campaign.

## Why the pipe lies

Three independent failure modes, all of them structural:

- **The broker kills long-lived ssh.** A job longer than the connection's
  patience dies mid-run; the caller sees a closed pipe, not a failure.
- **A readiness pattern can match a log header.** `sed -n /building/q` fires on
  a banner printed before the work starts — the job is at line 1 and the caller
  already moved on.
- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is
  genuinely ready-shaped, the pipeline tears down the remote command it was
  watching. Result: green report, dead job.

The only reliable signal is a **completion sentinel written by the job itself**,
read afterwards by a short-lived connection. That is what this is.

## Contract

```bash
remote-job.sh start NAME [--] CMD...   # launch detached, return at once
remote-job.sh status [NAME]            # one line per job; names literal, never globs
remote-job.sh wait NAME [SEC] [N]      # block until DONE or timeout, then tail N lines
remote-job.sh out NAME [N]             # tail a job's output
remote-job.sh tail [NAME]              # follow one job, or all; exit 1 if nothing to follow
remote-job.sh watch                    # refreshing status, for a spare pane
```

Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+
stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**).

Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or
timed out · `2` usage. The `rc=` column is the job's own code — a job that
failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and
"did it work" are different questions a supervisor must be able to ask
separately.

Written for the case where the job **restarts the endpoint the agent itself is
running on**: the ssh connection dies, the session may die, and the answer is
still sitting in `$JOBS` when a session comes back.

## Use over ssh

```bash
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
  bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20   # short-lived reconnect
```

Each ssh call is short, so there is nothing there to abort, and the second call
can be made from a fresh session on a machine that rebooted underneath it.

## Failure modes

| Event | Result |
|-------|--------|
| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it |
| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE |
| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status |
| Two jobs, same NAME | the second replaces the first's state files |
| `tail` with no job, or a name that is a glob | exit 1 saying there is nothing to follow; names are matched literally, never expanded |
| Job dir missing | created on first use |

Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel
ordering — never internals).