Frank
ab80c1366d
frank-dragoon's health gate never checked WHO answered :7770 — the false-green the fleet learned about on 2026-09-26 (a stray probe holding the port satisfies a bare curl while the unit is stopped). The other two handlers carried that check as inlined copies; all three now call assert_unit_owns_port, and dragoon gets the protection it was missing. The seam follows: the endpoints' text-grep deployer tests are deleted, because tests/llm_deploy_gates.bats EXECUTES the same handlers end to end (stub systemctl/podman/sudo/ss/curl) and asserts refusals, restarts, unit links, the pending marker, and the MainPID hardening. GATE_PREFIX is exported at each handler for the same reason the serve scripts export it.
132 lines
5.5 KiB
Bash
Executable File
132 lines
5.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# llm-gates.sh — the artifact gates every LLM endpoint runs before it starts.
|
|
#
|
|
# Why this file exists: the fleet law from the 2026-10-01 crash-loop is that
|
|
# a serve script and the deployer must refuse the SAME missing/truncated
|
|
# artifact, the SAME way. When the two gates were hand-written twins they
|
|
# drifted mid-cutover — green deploy, red unit, every time the payload moved.
|
|
# Both sides source this library, so the refusal is one text and one rule.
|
|
#
|
|
# Contract (public interface):
|
|
# require_exec PATH LABEL HINT
|
|
# require_file PATH LABEL HINT
|
|
# require_bytes PATH LABEL WANT_BYTES HINT
|
|
# require_min_mib PATH LABEL MIN_MIB HINT
|
|
# require_dir_files DIR LABEL HINT FILE...
|
|
# require_image REF HINT
|
|
#
|
|
# Each returns 0 and prints nothing on success. On failure it prints two
|
|
# lines to stderr — the first naming the endpoint ($GATE_PREFIX) and the
|
|
# artifact, the second the remedy — and returns 1. It NEVER exits: the serve
|
|
# scripts run under `set -e` (so a refusal is terminal), while the deployer
|
|
# wants `|| return 1` so a failed gate reports a red deploy for that one
|
|
# service and the dispatcher's state marker stays pending.
|
|
# Lines are labelled by ARTIFACT CLASS, and the class is the whole label
|
|
# (the path rides in the same line): the class + path is what an operator
|
|
# acts on, so no wording order and no payload rename can make the two twins
|
|
# disagree. The deployer's old "halogen vision tower" and the serve script's
|
|
# "vision-tower" were one gate in two vocabularies that no grep could join.
|
|
|
|
# The size law is equality, not a floor, for `require_bytes`: a hub fetch
|
|
# that truncates is the fleet's standing hazard (curl exits 0 on 401/403 and
|
|
# writes the error body as the artifact), and the declared byte count is a
|
|
# contract in both directions — a bigger file is a DIFFERENT artifact.
|
|
|
|
# GATE_PREFIX names the endpoint in every refusal (e.g. frank-dragoon). The
|
|
# caller exports it; shellcheck cannot follow it across the source boundary,
|
|
# so the read site carries the directive.
|
|
# shellcheck disable=SC2034
|
|
: "${GATE_PREFIX:=gate}"
|
|
|
|
# Two bash laws make the `return 1` at every call site load-bearing:
|
|
# * a function whose body ends inside an `if` returns the CONDITION's
|
|
# status (true = 0), so a gate that ends on its own refusal prints the
|
|
# refusal and reports SUCCESS;
|
|
# * `set -e` is suspended for any command in a `… || …` chain, so a gate
|
|
# that runs inside one must not be trusted to abort the caller.
|
|
# Every gate's refusal therefore carries an explicit `|| return 1`, and the
|
|
# caller consumes the status (`|| exit 1`, `|| return 1`).
|
|
_gate_fail() {
|
|
printf '%s: %s\n' "$GATE_PREFIX" "$1" >&2
|
|
printf ' hint: %s\n' "$2" >&2
|
|
return 1
|
|
}
|
|
|
|
# require_file — PATH must be a regular file. A directory (an unpacked
|
|
# staging dir wearing the payload's name) and a symlink (a HuggingFace cache
|
|
# snapshot's entry, which dangles inside the container mount) both refuse.
|
|
require_file() {
|
|
local path="$1" label="$2" hint="$3"
|
|
if [[ -L "$path" || ! -f "$path" ]]; then
|
|
_gate_fail "$label: missing at $path" "$hint" || return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# require_exec — PATH must be an executable regular file. The native runtimes
|
|
# (llama.cpp's llama-server) are binaries the operator builds; a tree that
|
|
# compiled but lost its +x, or a directory wearing the binary's name, must
|
|
# refuse the same way a missing one does.
|
|
require_exec() {
|
|
local path="$1" label="$2" hint="$3"
|
|
if [[ -L "$path" || ! -x "$path" ]]; then
|
|
_gate_fail "$label: missing at $path" "$hint" || return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# require_bytes — PATH must exist AND be exactly WANT_BYTES bytes.
|
|
require_bytes() {
|
|
local path="$1" label="$2" want="$3" hint="$4" got
|
|
require_file "$path" "$label" "$hint" || return 1
|
|
got="$(stat -c %s "$path")"
|
|
if [[ "$got" != "$want" ]]; then
|
|
_gate_fail "$label: is $got bytes, expected $want" "$hint" || return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# require_min_mib — PATH must be at least MIN_MIB mebibytes. The floor form
|
|
# is for build products (a .rad container is a range of legal sizes; a
|
|
# truncated convert is not).
|
|
require_min_mib() {
|
|
local path="$1" label="$2" min="$3" hint="$4" mib
|
|
require_file "$path" "$label" "$hint" || return 1
|
|
mib=$(( $(stat -c %s "$path") / 1048576 ))
|
|
if (( mib < min )); then
|
|
_gate_fail "$label: is $mib MiB, expected >= $min MiB" "$hint" || return 1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# require_dir_files — DIR must be a directory and every named member a
|
|
# regular file (the flat-tokenizer law: snapshot symlinks dangle in the
|
|
# container mount even when the directory itself is real. A symlinked
|
|
# DIRECTORY is refused too: a HF cache snapshot can link the directory
|
|
# itself, and a link dangles inside the container mount.
|
|
require_dir_files() {
|
|
local dir="$1" label="$2" hint="$3"
|
|
shift 3
|
|
local f
|
|
if [[ -L "$dir" || ! -d "$dir" ]]; then
|
|
_gate_fail "$label: missing at $dir" "$hint" || return 1
|
|
fi
|
|
for f in "$@"; do
|
|
if [[ -L "$dir/$f" || ! -f "$dir/$f" ]]; then
|
|
_gate_fail "$label: $dir/$f is missing or a symlink" "$hint" || return 1
|
|
fi
|
|
done
|
|
return 0
|
|
}
|
|
|
|
# require_image — the image ref must be in podman storage. A name@sha256 ref
|
|
# exists only if THAT manifest was pulled, so the gate is the ref itself, in
|
|
# one command: no pipefail/grep race (the 2026-09-30 lesson).
|
|
require_image() {
|
|
local ref="$1" hint="$2"
|
|
if ! podman image exists "$ref"; then
|
|
_gate_fail "image: $ref is not in podman storage" "$hint" || return 1
|
|
fi
|
|
return 0
|
|
}
|