Add remote-job.sh: a detached job whose result outlives the session
`ssh host 'cmd | tail -f log | sed -n /ready/q'` reported success on jobs that never finished, three structural ways: the broker aborts the long connection, a readiness pattern can match a log header printed before the work starts, and `sed -n /x/q` exits the instant it matches — tearing down the very command it was watching. The replacement runs the job detached on the host and writes NAME.out, NAME.rc, then NAME.done LAST, so status is a cheap read of files over a short-lived connection. That survives the connection, the restart, and the job that restarts the endpoint the agent itself is running on. Black-box tests pin the sentinel ordering (rc before done), that a finished-but- failed job is DONE rc=N with the report still exiting 0, and that a name with no state is never reported DONE. Refs #53
This commit is contained in:
Frank
@@ -20,6 +20,7 @@ Configuration guides and reference material for installed software. Each tool ha
|
||||
* [niri](niri/index.md) — Scrollable-tiling Wayland compositor — configuration, keybindings, and architecture for the workstation.
|
||||
* [noctalia-shell](noctalia-shell.md) — Native desktop environment for Niri — launcher, control center, settings, lock screen, session menu, hotkey overlay, media controls, and system management.
|
||||
* [packages](packages.md) — `packages/*.pkgs` desired state and the two tools that keep a machine and its manifest agreeing: `install.sh` installs the delta, `pkg-inventory.sh` reports what the machine has outgrown.
|
||||
* [remote-job](remote-job.md) — Detached job runner for `ssh` work: completion sentinels on the host replace the `tail -f | sed -n /ready/q` pattern that reported success on jobs that never finished.
|
||||
* [tmux](tmux.md) — Terminal multiplexer configuration, extended-keys setup, keyboard input best practices, and reload/restart procedures.
|
||||
|
||||
## Adding a New Tool
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
type: reference
|
||||
title: remote-job — detached jobs that outlive the session
|
||||
description: Run a command on a remote box so its result survives the connection, the restart, and the agent that started it — the replacement for ssh piped through tail and sed, which reported success on jobs that never finished.
|
||||
tags:
|
||||
- ssh
|
||||
- fleet
|
||||
- automation
|
||||
- verification
|
||||
verified: 2026-10-20
|
||||
timestamp: 2026-10-20
|
||||
---
|
||||
|
||||
# remote-job
|
||||
|
||||
`scripts/remote-job.sh` starts a command **detached on the host**, and reports
|
||||
what actually happened from files on disk. It exists because
|
||||
`ssh host 'cmd | tail -f log | sed -n /ready/q'` lied repeatedly during the
|
||||
worktree campaign.
|
||||
|
||||
## Why the pipe lies
|
||||
|
||||
Three independent failure modes, all of them structural:
|
||||
|
||||
- **The broker kills long-lived ssh.** A job longer than the connection's
|
||||
patience dies mid-run; the caller sees a closed pipe, not a failure.
|
||||
- **A readiness pattern can match a log header.** `sed -n /building/q` fires on
|
||||
a banner printed before the work starts — the job is at line 1 and the caller
|
||||
already moved on.
|
||||
- **`sed -n /x/q` exits the instant it matches.** Even when the pattern is
|
||||
genuinely ready-shaped, the pipeline tears down the remote command it was
|
||||
watching. Result: green report, dead job.
|
||||
|
||||
The only reliable signal is a **completion sentinel written by the job itself**,
|
||||
read afterwards by a short-lived connection. That is what this is.
|
||||
|
||||
## Contract
|
||||
|
||||
```bash
|
||||
remote-job.sh start NAME [--] CMD... # launch detached, return at once
|
||||
remote-job.sh status [NAME] # one line per job; exit 0 only if all DONE
|
||||
remote-job.sh wait NAME [SEC] [N] # block until DONE or timeout, then tail N lines
|
||||
remote-job.sh out NAME [N] # tail a job's output
|
||||
remote-job.sh tail [NAME] # follow one job, or every job at once
|
||||
remote-job.sh watch # refreshing status, for a spare pane
|
||||
```
|
||||
|
||||
Job dir `$REMOTE_JOBS_DIR` (default `$HOME/jobs`). Per job: `NAME.out` (stdout+
|
||||
stderr), `NAME.rc` (exit code), `NAME.done` (sentinel, **written last**).
|
||||
|
||||
Exit status of the *report*: `0` all reported jobs finished · `1` unfinished or
|
||||
timed out · `2` usage. The `rc=` column is the job's own code — a job that
|
||||
failed is still `DONE rc=7` and `status` exits `0`, because "is it finished" and
|
||||
"did it work" are different questions a supervisor must be able to ask
|
||||
separately.
|
||||
|
||||
Written for the case where the job **restarts the endpoint the agent itself is
|
||||
running on**: the ssh connection dies, the session may die, and the answer is
|
||||
still sitting in `$JOBS` when a session comes back.
|
||||
|
||||
## Use over ssh
|
||||
|
||||
```bash
|
||||
ssh dankserver ~/.omp/scripts/remote-job.sh start swap -- \
|
||||
bash -c 'systemctl --user restart frank-archon && <the health probe that endpoint answers>'
|
||||
ssh dankserver ~/.omp/scripts/remote-job.sh wait swap 300 20 # short-lived reconnect
|
||||
```
|
||||
|
||||
Each ssh call is short, so there is nothing there to abort, and the second call
|
||||
can be made from a fresh session on a machine that rebooted underneath it.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Event | Result |
|
||||
|-------|--------|
|
||||
| Session/connection dies mid-job | job keeps running; `status` from anywhere reports it |
|
||||
| Command name is wrong | `PENDING (no output yet)`, exit 1 — never a false DONE |
|
||||
| Job exits non-zero | `DONE rc=N`; the code is data, not the report's status |
|
||||
| Two jobs, same NAME | the second replaces the first's state files |
|
||||
| Job dir missing | created on first use |
|
||||
|
||||
Pinned by `tests/remote-job.bats` (black-box: output, exit codes, sentinel
|
||||
ordering — never internals).
|
||||
Executable
+115
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env bash
|
||||
# scripts/remote-job.sh — a detached job whose result outlives the session that started it
|
||||
#
|
||||
# Why this exists rather than `ssh host 'cmd | tail -f | sed -n /ready/q'`, which lied about success
|
||||
# repeatedly, for three structural reasons — all of them the pipe's fault: the broker aborts
|
||||
# long-lived ssh; a readiness pattern can match a log *header* before the work has begun; and
|
||||
# `sed -n /x/q` exits the instant it matches, so the supervisor reports green on a job that never
|
||||
# finished.
|
||||
#
|
||||
# Here the job runs detached on the box it is invoked on (over ssh, or locally). It writes NAME.out,
|
||||
# then NAME.rc, then touches NAME.done LAST. Status is then a cheap read of a directory: no
|
||||
# long-lived connection to lose, and the answer survives the session, the terminal, and the agent —
|
||||
# including the case where the job restarts the very service the agent is running on.
|
||||
#
|
||||
# remote-job.sh start NAME [--] CMD... launch detached, return at once
|
||||
# remote-job.sh status [NAME] one line per job
|
||||
# remote-job.sh wait NAME [SECONDS] [N] block until DONE or timeout, then tail N lines
|
||||
# remote-job.sh out NAME [LINES] print the tail of a job's output
|
||||
# remote-job.sh tail [NAME] follow one job, or every job at once
|
||||
# remote-job.sh watch refreshing status, for a spare pane
|
||||
#
|
||||
# Job directory: $REMOTE_JOBS_DIR, default $HOME/jobs. Ship this file to the host and run it there;
|
||||
# nothing about it is box-specific, which is the point.
|
||||
#
|
||||
# Exit status: 0 when every job reported is finished · 1 when something is unfinished or a wait
|
||||
# timed out · 2 usage. A job that FAILED still counts as finished for `status`: "is it done" and
|
||||
# "did it work" are different questions, and the second one is the rc= column.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
JOBS="${REMOTE_JOBS_DIR:-$HOME/jobs}"
|
||||
mkdir -p "$JOBS"
|
||||
|
||||
case "${1:-}" in
|
||||
start)
|
||||
[[ $# -ge 2 ]] || { echo "usage: remote-job.sh start NAME [--] CMD..." >&2; exit 2; }
|
||||
name="$2"; shift 2
|
||||
[[ "${1:-}" == "--" ]] && shift
|
||||
[[ $# -ge 1 ]] || { echo "remote-job: start needs a command for '$name'" >&2; exit 2; }
|
||||
# Clear the previous run entirely: a stale .out would make the new run's line count a lie.
|
||||
rm -f "$JOBS/$name.out" "$JOBS/$name.rc" "$JOBS/$name.done"
|
||||
# setsid leaves the invoking process group, so a dropped ssh or a killed agent takes nothing
|
||||
# down with it. rc lands BEFORE the sentinel: a waiter must never see DONE with no exit code.
|
||||
JOBS="$JOBS" setsid nohup bash -c '
|
||||
name="$1"; shift
|
||||
"$@" > "$JOBS/$name.out" 2>&1
|
||||
echo $? > "$JOBS/$name.rc"
|
||||
touch "$JOBS/$name.done"
|
||||
' _ "$name" "$@" < /dev/null > /dev/null 2>&1 &
|
||||
printf 'started %s\n' "$name"
|
||||
;;
|
||||
|
||||
status)
|
||||
if [[ -n "${2:-}" ]]; then
|
||||
names="$2"
|
||||
else
|
||||
names=$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)
|
||||
fi
|
||||
pending=0
|
||||
for n in $names; do
|
||||
if [[ -f "$JOBS/$n.done" ]]; then
|
||||
printf ' %-20s DONE rc=%s %s lines\n' "$n" \
|
||||
"$(cat "$JOBS/$n.rc" 2>/dev/null)" "$(wc -l < "$JOBS/$n.out" 2>/dev/null | tr -d ' ')"
|
||||
elif [[ -f "$JOBS/$n.out" ]]; then
|
||||
printf ' %-20s RUN %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" 2>/dev/null | tr -d ' ')"
|
||||
pending=1
|
||||
else
|
||||
printf ' %-20s PENDING (no output yet)\n' "$n"
|
||||
pending=1
|
||||
fi
|
||||
done
|
||||
if [[ -z "$names" ]]; then echo " (no jobs)"; exit 0; fi
|
||||
exit "$pending"
|
||||
;;
|
||||
|
||||
wait)
|
||||
[[ $# -ge 2 ]] || { echo "usage: remote-job.sh wait NAME [SECONDS] [TAIL]" >&2; exit 2; }
|
||||
name="$2"; limit="${3:-3600}"; lines="${4:-25}"; waited=0
|
||||
while [[ ! -e "$JOBS/$name.done" ]] && (( waited < limit )); do
|
||||
sleep 1; waited=$((waited + 1))
|
||||
done
|
||||
if [[ -e "$JOBS/$name.done" ]]; then
|
||||
printf 'DONE %s rc=%s\n' "$name" "$(cat "$JOBS/$name.rc" 2>/dev/null)"; rc=0
|
||||
else
|
||||
printf 'TIMEOUT %s after %ss\n' "$name" "$waited"; rc=1
|
||||
fi
|
||||
tail -n "$lines" "$JOBS/$name.out" 2>/dev/null
|
||||
exit "$rc"
|
||||
;;
|
||||
|
||||
out)
|
||||
[[ $# -ge 2 ]] || { echo "usage: remote-job.sh out NAME [LINES]" >&2; exit 2; }
|
||||
tail -n "${3:-40}" "$JOBS/$2.out" 2>/dev/null
|
||||
;;
|
||||
|
||||
tail)
|
||||
# Multiplexed follow: no NAME means every job at once, so one pane shows the whole harness.
|
||||
if [[ -n "${2:-}" ]]; then exec tail -f "$JOBS/$2.out"; fi
|
||||
exec tail -f "$JOBS"/*.out
|
||||
;;
|
||||
|
||||
watch)
|
||||
while true; do
|
||||
clear 2>/dev/null || true
|
||||
date '+%H:%M:%S'
|
||||
"$0" status
|
||||
sleep 5
|
||||
done
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "usage: remote-job.sh {start|status|wait|out|tail|watch} ..." >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
Executable
+135
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env bats
|
||||
# tests/remote-job.bats — black-box tests for scripts/remote-job.sh (Refs #53)
|
||||
#
|
||||
# Contract: a job runs detached, writes its output, then its exit code, then touches .done LAST.
|
||||
# Status is then a cheap independent read that survives the session that started the job — the
|
||||
# whole point is replacing `ssh host 'tail -f | sed'`, which reported success on jobs that had not
|
||||
# finished. Exit status answers "is everything finished"; the rc= field answers "did it work".
|
||||
#
|
||||
# Jobs really run here, against a temp job dir. No fake filesystem state is pre-seeded, because a
|
||||
# fixture that pre-creates .done proves nothing about the code that is supposed to create it.
|
||||
|
||||
setup() {
|
||||
TEST_TMP="$(mktemp -d)"
|
||||
SCRIPT="$BATS_TEST_DIRNAME/../scripts/remote-job.sh"
|
||||
export REMOTE_JOBS_DIR="$TEST_TMP/jobs"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
# Jobs are detached on purpose; leave nothing behind that could outlive the test run.
|
||||
pkill -P 1 -f "$TEST_TMP" 2>/dev/null || true
|
||||
rm -rf "$TEST_TMP"
|
||||
}
|
||||
|
||||
# until_done NAME SECONDS — poll instead of sleeping a fixed amount, so the suite is neither slow
|
||||
# nor flaky on a loaded machine.
|
||||
until_done() {
|
||||
local name="$1" limit="${2:-10}" waited=0
|
||||
while [[ ! -e "$REMOTE_JOBS_DIR/$name.done" && $waited -lt $((limit * 10)) ]]; do
|
||||
sleep 0.1; waited=$((waited + 1))
|
||||
done
|
||||
[[ -e "$REMOTE_JOBS_DIR/$name.done" ]]
|
||||
}
|
||||
|
||||
@test "start returns immediately while the job is still running" {
|
||||
run timeout 10 "$SCRIPT" start slow -- sleep 2
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == "started slow" ]]
|
||||
# A 2-second job must not have blocked a start that is supposed to return at once, and while it
|
||||
# is still in flight the report must say unfinished — never DONE.
|
||||
run "$SCRIPT" status slow
|
||||
[[ "$output" == *RUN* || "$output" == *PENDING* ]]
|
||||
[ "$status" -eq 1 ]
|
||||
}
|
||||
|
||||
@test "a successful job is reported DONE with its output" {
|
||||
"$SCRIPT" start ok -- bash -c 'echo marker-4711'
|
||||
until_done ok
|
||||
run "$SCRIPT" status ok
|
||||
[[ "$output" == *"ok"*DONE*rc=0* ]]
|
||||
[ "$status" -eq 0 ]
|
||||
run "$SCRIPT" out ok
|
||||
[[ "$output" == *marker-4711* ]]
|
||||
}
|
||||
|
||||
@test "a failing job's exit code survives into the report" {
|
||||
# This is the reason the .rc file exists: a supervisor that cannot see the failure code is the
|
||||
# bug the runner was written to fix.
|
||||
"$SCRIPT" start boom -- bash -c 'echo it failed here >&2; exit 7'
|
||||
until_done boom
|
||||
run "$SCRIPT" status boom
|
||||
[[ "$output" == *"boom"*DONE*rc=7* ]]
|
||||
run "$SCRIPT" out boom
|
||||
[[ "$output" == *"it failed here"* ]]
|
||||
}
|
||||
|
||||
@test "the .done sentinel is never written before the exit code" {
|
||||
# If .done landed first, a waiter could read an empty rc and call the job successful.
|
||||
"$SCRIPT" start order -- bash -c 'echo x'
|
||||
until_done order
|
||||
[ -f "$REMOTE_JOBS_DIR/order.rc" ]
|
||||
[ -s "$REMOTE_JOBS_DIR/order.rc" ]
|
||||
run cat "$REMOTE_JOBS_DIR/order.rc"
|
||||
[[ "$output" =~ ^[0-9]+$ ]]
|
||||
}
|
||||
|
||||
@test "wait prints DONE with the exit code and a tail of the output" {
|
||||
"$SCRIPT" start fin -- bash -c 'echo one; echo two'
|
||||
run "$SCRIPT" wait fin 15
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"DONE fin rc=0"* ]]
|
||||
[[ "$output" == *two* ]]
|
||||
}
|
||||
|
||||
@test "wait gives up on its own and says so" {
|
||||
"$SCRIPT" start never -- sleep 30
|
||||
run "$SCRIPT" wait never 1
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *TIMEOUT*never* ]]
|
||||
"$SCRIPT" status never | grep -q . # the job is still there, unfinished
|
||||
}
|
||||
|
||||
@test "status distinguishes running from pending from done" {
|
||||
run "$SCRIPT" status
|
||||
[[ "$output" == *"(no jobs)"* ]]
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
"$SCRIPT" start later -- sleep 3
|
||||
run "$SCRIPT" status later
|
||||
[[ "$output" == *later* ]]
|
||||
[[ "$output" == *RUN* || "$output" == *PENDING* ]]
|
||||
[ "$status" -eq 1 ] # unfinished: a supervisor must be able to branch on this
|
||||
}
|
||||
|
||||
@test "two jobs in flight keep separate state" {
|
||||
"$SCRIPT" start a -- bash -c 'echo A; exit 0'
|
||||
"$SCRIPT" start b -- bash -c 'echo B; exit 2'
|
||||
until_done a && until_done b
|
||||
run "$SCRIPT" status
|
||||
[[ "$output" == *"a"*DONE*rc=0* ]]
|
||||
[[ "$output" == *"b"*DONE*rc=2* ]]
|
||||
run "$SCRIPT" out a
|
||||
[[ "$output" == A ]]
|
||||
run "$SCRIPT" out b
|
||||
[[ "$output" == B ]]
|
||||
}
|
||||
|
||||
@test "out honours a line limit" {
|
||||
"$SCRIPT" start many -- bash -c 'seq 1 40'
|
||||
until_done many
|
||||
run "$SCRIPT" out many 3
|
||||
[ "$(printf '%s\n' "$output" | grep -c .)" -eq 3 ]
|
||||
[[ "$output" == *38* ]] # the tail, not the head
|
||||
}
|
||||
|
||||
@test "an unknown verb is a usage error" {
|
||||
run "$SCRIPT" frobnicate
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *usage* ]]
|
||||
}
|
||||
|
||||
@test "the job directory is created on first use" {
|
||||
[ ! -d "$REMOTE_JOBS_DIR" ]
|
||||
"$SCRIPT" status
|
||||
[ -d "$REMOTE_JOBS_DIR" ]
|
||||
}
|
||||
Reference in New Issue
Block a user