From 35ceae6d7bd695c5e947b8eb3a5cef3ca7b96279 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 20 Oct 2026 19:24:15 -0700 Subject: [PATCH] Add remote-job.sh: a detached job whose result outlives the session MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that never finished, three structural ways: the broker aborts the long connection, a readiness pattern can match a log header printed before the work starts, and `sed -n /x/q` exits the instant it matches — tearing down the very command it was watching. The replacement runs the job detached on the host and writes NAME.out, NAME.rc, then NAME.done LAST, so status is a cheap read of files over a short-lived connection. That survives the connection, the restart, and the job that restarts the endpoint the agent itself is running on. Black-box tests pin the sentinel ordering (rc before done), that a finished-but- failed job is DONE rc=N with the report still exiting 0, and that a name with no state is never reported DONE. Refs #53 --- omp-docs/tools/index.md | 1 + omp-docs/tools/remote-job.md | 83 +++++++++++++++++++++ scripts/remote-job.sh | 115 +++++++++++++++++++++++++++++ tests/remote-job.bats | 135 +++++++++++++++++++++++++++++++++++ 4 files changed, 334 insertions(+) create mode 100644 omp-docs/tools/remote-job.md create mode 100755 scripts/remote-job.sh create mode 100755 tests/remote-job.bats diff --git a/omp-docs/tools/index.md b/omp-docs/tools/index.md index f990fd2..e8e0abe 100644 --- a/omp-docs/tools/index.md +++ b/omp-docs/tools/index.md @@ -20,6 +20,7 @@ Configuration guides and reference material for installed software. Each tool ha * [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation. * [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management. * [packages](packages.md) — `packages/*.pkgs` desired state and the two tools that keep a machine and its manifest agreeing: `install.sh` installs the delta, `pkg-inventory.sh` reports what the machine has outgrown. +* [remote-job](remote-job.md) — Detached job runner for `ssh` work: completion sentinels on the host replace the `tail -f | sed -n /ready/q` pattern that reported success on jobs that never finished. * [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures. ## Adding a New Tool diff --git a/omp-docs/tools/remote-job.md b/omp-docs/tools/remote-job.md new file mode 100644 index 0000000..94b526d --- /dev/null +++ b/omp-docs/tools/remote-job.md @@ -0,0 +1,83 @@ +--- +type: reference +title: remote-job — detached jobs that outlive the session +description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished. +tags: + - ssh + - fleet + - automation + - verification +verified: 2026-10-20 +timestamp: 2026-10-20 +--- + +# remote-job + +`scripts/remote-job.sh` starts a command **detached on the host**, and reports +what actually happened from files on disk. It exists because +`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the +worktree campaign. + +## Why the pipe lies + +Three independent failure modes, all of them structural: + +- **The broker kills long-lived ssh.** A job longer than the connection's + patience dies mid-run; the caller sees a closed pipe, not a failure. +- **A readiness pattern can match a log header.** `sed -n /building/q` fires on + a banner printed before the work starts — the job is at line 1 and the caller + already moved on. +- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is + genuinely ready-shaped, the pipeline tears down the remote command it was + watching. Result: green report, dead job. + +The only reliable signal is a **completion sentinel written by the job itself**, +read afterwards by a short-lived connection. That is what this is. + +## Contract + +```bash +remote-job.sh start NAME [--] CMD... # launch detached, return at once +remote-job.sh status [NAME] # one line per job; exit 0 only if all DONE +remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines +remote-job.sh out NAME [N] # tail a job's output +remote-job.sh tail [NAME] # follow one job, or every job at once +remote-job.sh watch # refreshing status, for a spare pane +``` + +Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+ +stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**). + +Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or +timed out · `2` usage. The `rc=` column is the job's own code — a job that +failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and +"did it work" are different questions a supervisor must be able to ask +separately. + +Written for the case where the job **restarts the endpoint the agent itself is +running on**: the ssh connection dies, the session may die, and the answer is +still sitting in `$JOBS` when a session comes back. + +## Use over ssh + +```bash +ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \ + bash -c 'systemctl --user restart frank-archon && ' +ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect +``` + +Each ssh call is short, so there is nothing there to abort, and the second call +can be made from a fresh session on a machine that rebooted underneath it. + +## Failure modes + +| Event | Result | +|-------|--------| +| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it | +| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE | +| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status | +| Two jobs, same NAME | the second replaces the first's state files | +| Job dir missing | created on first use | + +Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel +ordering — never internals). diff --git a/scripts/remote-job.sh b/scripts/remote-job.sh new file mode 100755 index 0000000..20082b3 --- /dev/null +++ b/scripts/remote-job.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# scripts/remote-job.sh — a detached job whose result outlives the session that started it +# +# Why this exists rather than `ssh host 'cmd | tail -f | sed -n /ready/q'`, which lied about success +# repeatedly, for three structural reasons — all of them the pipe's fault: the broker aborts +# long-lived ssh; a readiness pattern can match a log *header* before the work has begun; and +# `sed -n /x/q` exits the instant it matches, so the supervisor reports green on a job that never +# finished. +# +# Here the job runs detached on the box it is invoked on (over ssh, or locally). It writes NAME.out, +# then NAME.rc, then touches NAME.done LAST. Status is then a cheap read of a directory: no +# long-lived connection to lose, and the answer survives the session, the terminal, and the agent — +# including the case where the job restarts the very service the agent is running on. +# +# remote-job.sh start NAME [--] CMD... launch detached, return at once +# remote-job.sh status [NAME] one line per job +# remote-job.sh wait NAME [SECONDS] [N] block until DONE or timeout, then tail N lines +# remote-job.sh out NAME [LINES] print the tail of a job's output +# remote-job.sh tail [NAME] follow one job, or every job at once +# remote-job.sh watch refreshing status, for a spare pane +# +# Job directory: $REMOTE_JOBS_DIR, default $HOME/jobs. Ship this file to the host and run it there; +# nothing about it is box-specific, which is the point. +# +# Exit status: 0 when every job reported is finished · 1 when something is unfinished or a wait +# timed out · 2 usage. A job that FAILED still counts as finished for `status`: "is it done" and +# "did it work" are different questions, and the second one is the rc= column. + +set -uo pipefail + +JOBS="${REMOTE_JOBS_DIR:-$HOME/jobs}" +mkdir -p "$JOBS" + +case "${1:-}" in + start) + [[ $# -ge 2 ]] || { echo "usage: remote-job.sh start NAME [--] CMD..." >&2; exit 2; } + name="$2"; shift 2 + [[ "${1:-}" == "--" ]] && shift + [[ $# -ge 1 ]] || { echo "remote-job: start needs a command for '$name'" >&2; exit 2; } + # Clear the previous run entirely: a stale .out would make the new run's line count a lie. + rm -f "$JOBS/$name.out" "$JOBS/$name.rc" "$JOBS/$name.done" + # setsid leaves the invoking process group, so a dropped ssh or a killed agent takes nothing + # down with it. rc lands BEFORE the sentinel: a waiter must never see DONE with no exit code. + JOBS="$JOBS" setsid nohup bash -c ' + name="$1"; shift + "$@" > "$JOBS/$name.out" 2>&1 + echo $? > "$JOBS/$name.rc" + touch "$JOBS/$name.done" + ' _ "$name" "$@" < /dev/null > /dev/null 2>&1 & + printf 'started %s\n' "$name" + ;; + + status) + if [[ -n "${2:-}" ]]; then + names="$2" + else + names=$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort) + fi + pending=0 + for n in $names; do + if [[ -f "$JOBS/$n.done" ]]; then + printf ' %-20s DONE rc=%s %s lines\n' "$n" \ + "$(cat "$JOBS/$n.rc" 2>/dev/null)" "$(wc -l < "$JOBS/$n.out" 2>/dev/null | tr -d ' ')" + elif [[ -f "$JOBS/$n.out" ]]; then + printf ' %-20s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" 2>/dev/null | tr -d ' ')" + pending=1 + else + printf ' %-20s PENDING (no output yet)\n' "$n" + pending=1 + fi + done + if [[ -z "$names" ]]; then echo " (no jobs)"; exit 0; fi + exit "$pending" + ;; + + wait) + [[ $# -ge 2 ]] || { echo "usage: remote-job.sh wait NAME [SECONDS] [TAIL]" >&2; exit 2; } + name="$2"; limit="${3:-3600}"; lines="${4:-25}"; waited=0 + while [[ ! -e "$JOBS/$name.done" ]] && (( waited < limit )); do + sleep 1; waited=$((waited + 1)) + done + if [[ -e "$JOBS/$name.done" ]]; then + printf 'DONE %s rc=%s\n' "$name" "$(cat "$JOBS/$name.rc" 2>/dev/null)"; rc=0 + else + printf 'TIMEOUT %s after %ss\n' "$name" "$waited"; rc=1 + fi + tail -n "$lines" "$JOBS/$name.out" 2>/dev/null + exit "$rc" + ;; + + out) + [[ $# -ge 2 ]] || { echo "usage: remote-job.sh out NAME [LINES]" >&2; exit 2; } + tail -n "${3:-40}" "$JOBS/$2.out" 2>/dev/null + ;; + + tail) + # Multiplexed follow: no NAME means every job at once, so one pane shows the whole harness. + if [[ -n "${2:-}" ]]; then exec tail -f "$JOBS/$2.out"; fi + exec tail -f "$JOBS"/*.out + ;; + + watch) + while true; do + clear 2>/dev/null || true + date '+%H:%M:%S' + "$0" status + sleep 5 + done + ;; + + *) + echo "usage: remote-job.sh {start|status|wait|out|tail|watch} ..." >&2 + exit 2 + ;; +esac diff --git a/tests/remote-job.bats b/tests/remote-job.bats new file mode 100755 index 0000000..c980ffc --- /dev/null +++ b/tests/remote-job.bats @@ -0,0 +1,135 @@ +#!/usr/bin/env bats +# tests/remote-job.bats — black-box tests for scripts/remote-job.sh (Refs #53) +# +# Contract: a job runs detached, writes its output, then its exit code, then touches .done LAST. +# Status is then a cheap independent read that survives the session that started the job — the +# whole point is replacing `ssh host 'tail -f | sed'`, which reported success on jobs that had not +# finished. Exit status answers "is everything finished"; the rc= field answers "did it work". +# +# Jobs really run here, against a temp job dir. No fake filesystem state is pre-seeded, because a +# fixture that pre-creates .done proves nothing about the code that is supposed to create it. + +setup() { + TEST_TMP="$(mktemp -d)" + SCRIPT="$BATS_TEST_DIRNAME/../scripts/remote-job.sh" + export REMOTE_JOBS_DIR="$TEST_TMP/jobs" +} + +teardown() { + # Jobs are detached on purpose; leave nothing behind that could outlive the test run. + pkill -P 1 -f "$TEST_TMP" 2>/dev/null || true + rm -rf "$TEST_TMP" +} + +# until_done NAME SECONDS — poll instead of sleeping a fixed amount, so the suite is neither slow +# nor flaky on a loaded machine. +until_done() { + local name="$1" limit="${2:-10}" waited=0 + while [[ ! -e "$REMOTE_JOBS_DIR/$name.done" && $waited -lt $((limit * 10)) ]]; do + sleep 0.1; waited=$((waited + 1)) + done + [[ -e "$REMOTE_JOBS_DIR/$name.done" ]] +} + +@test "start returns immediately while the job is still running" { + run timeout 10 "$SCRIPT" start slow -- sleep 2 + [ "$status" -eq 0 ] + [[ "$output" == "started slow" ]] + # A 2-second job must not have blocked a start that is supposed to return at once, and while it + # is still in flight the report must say unfinished — never DONE. + run "$SCRIPT" status slow + [[ "$output" == *RUN* || "$output" == *PENDING* ]] + [ "$status" -eq 1 ] +} + +@test "a successful job is reported DONE with its output" { + "$SCRIPT" start ok -- bash -c 'echo marker-4711' + until_done ok + run "$SCRIPT" status ok + [[ "$output" == *"ok"*DONE*rc=0* ]] + [ "$status" -eq 0 ] + run "$SCRIPT" out ok + [[ "$output" == *marker-4711* ]] +} + +@test "a failing job's exit code survives into the report" { + # This is the reason the .rc file exists: a supervisor that cannot see the failure code is the + # bug the runner was written to fix. + "$SCRIPT" start boom -- bash -c 'echo it failed here >&2; exit 7' + until_done boom + run "$SCRIPT" status boom + [[ "$output" == *"boom"*DONE*rc=7* ]] + run "$SCRIPT" out boom + [[ "$output" == *"it failed here"* ]] +} + +@test "the .done sentinel is never written before the exit code" { + # If .done landed first, a waiter could read an empty rc and call the job successful. + "$SCRIPT" start order -- bash -c 'echo x' + until_done order + [ -f "$REMOTE_JOBS_DIR/order.rc" ] + [ -s "$REMOTE_JOBS_DIR/order.rc" ] + run cat "$REMOTE_JOBS_DIR/order.rc" + [[ "$output" =~ ^[0-9]+$ ]] +} + +@test "wait prints DONE with the exit code and a tail of the output" { + "$SCRIPT" start fin -- bash -c 'echo one; echo two' + run "$SCRIPT" wait fin 15 + [ "$status" -eq 0 ] + [[ "$output" == *"DONE fin rc=0"* ]] + [[ "$output" == *two* ]] +} + +@test "wait gives up on its own and says so" { + "$SCRIPT" start never -- sleep 30 + run "$SCRIPT" wait never 1 + [ "$status" -eq 1 ] + [[ "$output" == *TIMEOUT*never* ]] + "$SCRIPT" status never | grep -q . # the job is still there, unfinished +} + +@test "status distinguishes running from pending from done" { + run "$SCRIPT" status + [[ "$output" == *"(no jobs)"* ]] + [ "$status" -eq 0 ] + + "$SCRIPT" start later -- sleep 3 + run "$SCRIPT" status later + [[ "$output" == *later* ]] + [[ "$output" == *RUN* || "$output" == *PENDING* ]] + [ "$status" -eq 1 ] # unfinished: a supervisor must be able to branch on this +} + +@test "two jobs in flight keep separate state" { + "$SCRIPT" start a -- bash -c 'echo A; exit 0' + "$SCRIPT" start b -- bash -c 'echo B; exit 2' + until_done a && until_done b + run "$SCRIPT" status + [[ "$output" == *"a"*DONE*rc=0* ]] + [[ "$output" == *"b"*DONE*rc=2* ]] + run "$SCRIPT" out a + [[ "$output" == A ]] + run "$SCRIPT" out b + [[ "$output" == B ]] +} + +@test "out honours a line limit" { + "$SCRIPT" start many -- bash -c 'seq 1 40' + until_done many + run "$SCRIPT" out many 3 + [ "$(printf '%s\n' "$output" | grep -c .)" -eq 3 ] + [[ "$output" == *38* ]] # the tail, not the head +} + +@test "an unknown verb is a usage error" { + run "$SCRIPT" frobnicate + [ "$status" -eq 2 ] + [[ "$output" == *usage* ]] +} + +@test "the job directory is created on first use" { + [ ! -d "$REMOTE_JOBS_DIR" ] + "$SCRIPT" status + [ -d "$REMOTE_JOBS_DIR" ] +}