serverAppConfigs/serverAppConfigs-docs/Operations/Remote Job Runner.md

---
type: reference
title: Remote Job Runner
description: "Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification."
tags: [operations, ssh, jobs, sentinel, detached]
timestamp: 2026-10-05T00:00:00-07:00
---

# Remote Job Runner

Use for any long-running job on a remote host (benchmarks, model loads,
builds, sweeps) whose result must survive the session. Also whenever you catch
yourself writing `ssh host 'tail -f log | sed …'` as a watcher.

## Why the obvious approach fails structurally

`ssh host 'tail -f log | grep/sed …'` as a background watcher fails in ways
that all LOOK like "the job died":

- the session broker **aborts long-lived ssh** — the watcher vanishes and the
  result is lost;
- a `ready` regex can match the log's own **header** on the first line — the
  supervisor reports success instantly while the job is still running;
- a `sed -n '/PAT/,$p'` filter **exits at the first match**, ending the pipe;
- output goes into the ssh stream, so an aborted session loses everything.

## The mechanism: detach on the HOST, poll a sentinel file

Completion is a **`.done` sentinel touched last**, not a log pattern. Status
is then one fast independent command with no long-lived process to lose.

```bash
#!/usr/bin/env bash
# job.sh {start|status|wait|out|tail|watch} ...   Jobs live in $JOBS.
set -uo pipefail
JOBS=/home/dank/jobs
mkdir -p "$JOBS"

case "${1:-}" in
start)
    NAME="$2"; shift 2
    [ "${1:-}" = "--" ] && shift
    rm -f "$JOBS/$NAME.done" "$JOBS/$NAME.rc"
    # setsid detaches from this session's process group: the critical bit.
    setsid nohup bash -c '
        NAME="$1"; shift
        "$@" > "'"$JOBS"'/$NAME.out" 2>&1
        echo $? > "'"$JOBS"'/$NAME.rc"
        touch "'"$JOBS"'/$NAME.done"
    ' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
    echo "started $NAME"
    ;
status)
    # Glob *.out (not *.done) or running jobs are invisible.
    names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
    for n in $names; do
        if   [ -f "$JOBS/$n.done" ]; then
            printf '  %-18s DONE   rc=%s  %s lines\n' "$n" \
                "$(cat "$JOBS/$n.rc" 2>/dev/null)" \
                "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
        elif [ -f "$JOBS/$n.out" ]; then
            printf '  %-18s RUN    %s lines so far\n' "$n" "$(wc -l < "$JOBS/$n.out" | tr -d ' ')"
        else
            printf '  %-18s PENDING\n' "$n"
        fi
    done
    ;
wait)
    NAME="$2"; LIMIT="${3:-3600}"; waited=0
    while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
        sleep 5; waited=$((waited+5))
    done
    [ -f "$JOBS/$NAME.done" ] \
        && echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
        || echo "TIMEOUT $NAME after ${waited}s"
    tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
    ;
out)  NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;
esac
```

```bash
job.sh start NAME -- <command>   # launch detached
job.sh status [NAME]             # one-shot state
job.sh wait NAME 900 25          # block, then print the tail
job.sh out NAME 60               # retrieve any past result
job.sh tail [NAME]               # live follow
job.sh watch                     # refreshing status table
```

## Invariants

- **`.done` is touched LAST**, after the exit code is written — its presence
  alone means the output file is final.
- **Glob `*.out` for status**, never `*.done` — globbing `.done` hides running
  jobs entirely, the exact blindness this fixes.
- **Capture the exit code** (`rc=7` proved a self-test worked; without it a
  job can "complete" having failed).
- **Never put the payload in a line you grep for** — the sentinel is the only
  completion signal.
- **Never `pkill -f <pattern>`** where the pattern also matches your own ssh
  command line — it kills the session. Distinctive names or `pkill -x`.
- **Write helper scripts to a file and `scp` them.** Triple-nested
  `ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
  quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
  every failure disappeared once the script was a file.

## Verify before trusting it (3am law)

```bash
job.sh start selftest -- bash -c 'echo one; sleep 2; echo two; exit 7'
job.sh start slowtest -- bash -c 'echo start; sleep 30; echo end'
sleep 4; job.sh status      # selftest DONE rc=7; slowtest RUN
```

Both paths, including the failure case, and the rc round-trip — a runner that
only proves the happy path is not trustworthy.

## Reporting discipline

- Give the user a real terminal when the run is long:
  `alacritty --title "bench - dankserver" -e bash -c 'ssh -o
  ServerAliveInterval=30 -t dank@192.168.0.235 "bash /home/dank/job.sh tail"' &`
- State the MECHANISM, not vibes: the job is durable on the host;
  `job.sh out NAME` retrieves it later. Do not narrate "still loading"
  repeatedly — check once, act, report the resolved result.
- When the user asks for status, one fast command, not a narration.

## Related

-  the sweeps this runner executes
- [[Deploy State Triage]] — the two-process deploy this lesson generalizes