Frank
8393dc1090
`status` built its job list with `for n in $names`, which word-splits AND globs. The second half is the bug: a job named `x*` was expanded against the caller's working directory, so the report described unrelated files there as jobs and never mentioned the one that ran — on a harness whose whole purpose is a report you can trust after the session that started the job is gone. Splitting stays (several names in one argument is the documented call shape); expansion goes. `tail` was honest-by-accident in the wrong direction: with no jobs it handed a literal `$JOBS/*.out` to tail, which reports a missing *file* — indistinguishable from a broken tool — and a missing name produced tail's errno rather than "no job called that". Both now say what is missing and exit 1. Red first: the name test reads the report from a directory containing an unrelated file, so the old glob is visible as a phantom job; both new tests fail against the previous code and pass against this one. Refs #60
85 lines
3.6 KiB
Markdown
85 lines
3.6 KiB
Markdown
---
|
|
type: reference
|
|
title: remote-job — detached jobs that outlive the session
|
|
description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished.
|
|
tags:
|
|
- ssh
|
|
- fleet
|
|
- automation
|
|
- verification
|
|
verified: 2026-10-20
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# remote-job
|
|
|
|
`scripts/remote-job.sh` starts a command **detached on the host**, and reports
|
|
what actually happened from files on disk. It exists because
|
|
`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the
|
|
worktree campaign.
|
|
|
|
## Why the pipe lies
|
|
|
|
Three independent failure modes, all of them structural:
|
|
|
|
- **The broker kills long-lived ssh.** A job longer than the connection's
|
|
patience dies mid-run; the caller sees a closed pipe, not a failure.
|
|
- **A readiness pattern can match a log header.** `sed -n /building/q` fires on
|
|
a banner printed before the work starts — the job is at line 1 and the caller
|
|
already moved on.
|
|
- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is
|
|
genuinely ready-shaped, the pipeline tears down the remote command it was
|
|
watching. Result: green report, dead job.
|
|
|
|
The only reliable signal is a **completion sentinel written by the job itself**,
|
|
read afterwards by a short-lived connection. That is what this is.
|
|
|
|
## Contract
|
|
|
|
```bash
|
|
remote-job.sh start NAME [--] CMD... # launch detached, return at once
|
|
remote-job.sh status [NAME] # one line per job; names literal, never globs
|
|
remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines
|
|
remote-job.sh out NAME [N] # tail a job's output
|
|
remote-job.sh tail [NAME] # follow one job, or all; exit 1 if nothing to follow
|
|
remote-job.sh watch # refreshing status, for a spare pane
|
|
```
|
|
|
|
Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+
|
|
stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**).
|
|
|
|
Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or
|
|
timed out · `2` usage. The `rc=` column is the job's own code — a job that
|
|
failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and
|
|
"did it work" are different questions a supervisor must be able to ask
|
|
separately.
|
|
|
|
Written for the case where the job **restarts the endpoint the agent itself is
|
|
running on**: the ssh connection dies, the session may die, and the answer is
|
|
still sitting in `$JOBS` when a session comes back.
|
|
|
|
## Use over ssh
|
|
|
|
```bash
|
|
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
|
|
bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
|
|
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect
|
|
```
|
|
|
|
Each ssh call is short, so there is nothing there to abort, and the second call
|
|
can be made from a fresh session on a machine that rebooted underneath it.
|
|
|
|
## Failure modes
|
|
|
|
| Event | Result |
|
|
|-------|--------|
|
|
| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it |
|
|
| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE |
|
|
| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status |
|
|
| Two jobs, same NAME | the second replaces the first's state files |
|
|
| `tail` with no job, or a name that is a glob | exit 1 saying there is nothing to follow; names are matched literally, never expanded |
|
|
| Job dir missing | created on first use |
|
|
|
|
Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel
|
|
ordering — never internals).
|