serverAppConfigs/serverAppConfigs-docs/Operations/Remote Job Runner.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Remote Job Runner	Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification.	operations ssh jobs sentinel detached	2026-10-05T00:00:00-07:00

Remote Job Runner

Use for any long-running job on a remote host (benchmarks, model loads, builds, sweeps) whose result must survive the session. Also whenever you catch yourself writing ssh host 'tail -f log | sed …' as a watcher.

Why the obvious approach fails structurally

ssh host 'tail -f log | grep/sed …' as a background watcher fails in ways that all LOOK like "the job died":

• the session broker aborts long-lived ssh — the watcher vanishes and the result is lost;
• a ready regex can match the log's own header on the first line — the supervisor reports success instantly while the job is still running;
• a sed -n '/PAT/,$p' filter exits at the first match, ending the pipe;
• output goes into the ssh stream, so an aborted session loses everything.

The mechanism: detach on the HOST, poll a sentinel file

Completion is a .done sentinel touched last, not a log pattern. Status is then one fast independent command with no long-lived process to lose.

#!/usr/bin/env bash
# job.sh {start|status|wait|out|tail|watch} ...   Jobs live in $JOBS.
set -uo pipefail
JOBS=/home/dank/jobs
mkdir -p "$JOBS"

case "${1:-}" in
start)
    NAME="$2"; shift 2
    [ "${1:-}" = "--" ] && shift
    rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc"
    # setsid detaches from this session's process group: the critical bit.
    setsid nohup bash -c '
        NAME="$1"; shift
        "$@" > "'"$JOBS"'/$NAME.out" 2>&1
        echo $? > "'"$JOBS"'/$NAME.rc"
        touch "'"$JOBS"'/$NAME.done"
    ' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
    echo "started $NAME"
    ;
status)
    # Glob *.out (not *.done) or running jobs are invisible.
    names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
    for n in $names; do
        if   [ -f "$JOBS/$n.done" ]; then
            printf '  %-18s DONE   rc=%s  %s lines\n' "$n" \
                "$(cat "$JOBS/$n.rc" 2>/dev/null)" \
                "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
        elif [ -f "$JOBS/$n.out" ]; then
            printf '  %-18s RUN    %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
        else
            printf '  %-18s PENDING\n' "$n"
        fi
    done
    ;
wait)
    NAME="$2"; LIMIT="${3:-3600}"; waited=0
    while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
        sleep 5; waited=$((waited+5))
    done
    [ -f "$JOBS/$NAME.done" ] \
        && echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
        || echo "TIMEOUT $NAME after ${waited}s"
    tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
    ;
out)  NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;
esac

job.sh start NAME -- <command>   # launch detached
job.sh status [NAME]             # one-shot state
job.sh wait NAME 900 25          # block, then print the tail
job.sh out NAME 60               # retrieve any past result
job.sh tail [NAME]               # live follow
job.sh watch                     # refreshing status table

Invariants

• .done is touched LAST, after the exit code is written — its presence alone means the output file is final.
• Glob *.out for status, never *.done — globbing .done hides running jobs entirely, the exact blindness this fixes.
• Capture the exit code (rc=7 proved a self-test worked; without it a job can "complete" having failed).
• Never put the payload in a line you grep for — the sentinel is the only completion signal.
• Never pkill -f <pattern> where the pattern also matches your own ssh command line — it kills the session. Distinctive names or pkill -x.
• Write helper scripts to a file and scp them. Triple-nested ssh 'bash -c "python3 -c \"…\""' quoting broke repeatedly (unterminated quotes; grep -c returning "0\n0" because || echo 0 appended a line) every failure disappeared once the script was a file.
• A script fed over stdin owns that stdin. ssh host 'bash -s' <<'EOS' hands the script itself to stdin, so any child that reads stdin — podman exec -i, ssh -t, a bare read — swallows everything after it and the remainder vanishes without a word. It reads exactly like a hang: the commands before the child print, the ones after never existed. Drop -i, or give every child < /dev/null.

Verify before trusting it (3am law)

job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7'
job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end'
sleep 4; job.sh status      # selftest DONE rc=7; slowtest RUN

Both paths, including the failure case, and the rc round-trip — a runner that only proves the happy path is not trustworthy.

Reporting discipline

• Give the user a real terminal when the run is long: alacritty --title "bench - dankserver" -e bash -c 'ssh -o ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &
• State the MECHANISM, not vibes: the job is durable on the host; job.sh out NAME retrieves it later. Do not narrate "still loading" repeatedly — check once, act, report the resolved result.
• When the user asks for status, one fast command, not a narration.

Related

• the sweeps this runner executes
• Deploy State Triage — the two-process deploy this lesson generalizes