command-center/agent/managed-skills/service-preflight-refusal-triage/SKILL.md

name, description

name	description
service-preflight-refusal-triage	Diagnose a systemd service crash-looping in activating (auto-restart) with a fast exit: separate a preflight refusal from an engine crash, and make a runtime sysfs/kernel knob raise survive the reboot that resets it.

Preflight refusal vs engine crash (systemd services with env gates)

Symptom that decides the branch: unit activating (auto-restart), Main process exited, code=exited, status=1/FAILURE about 50 ms after start, restart counter in the thousands. No container/app process, no app log.

Triage sequence

1. systemctl status <svc> --no-pager -n 0 → note Exec path, status=, CPU-seconds (0.0xx ms = the script never got to work).
2. journalctl -u <svc> --no-pager -n 60 → the refusal line is printed by the serve script itself and NAMES the gate. On frank-archon: ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000. journalctl -u <svc> | grep -m1 "<gate phrase>" gives the FIRST occurrence: that timestamp is the incident start (correlate with uptime -b, last reboot).
3. Check the app's own state: podman ps -a showing the container name with a zero/absurd start time = the container was never created → the gate, not the engine.
4. Read the gate's code path in the repo (scripts/serve-<svc>.sh) before touching the host: it prints exactly what it requires.

Do NOT reach for journalctl -k GPU/OOM theories, gdb, or a payload checksum until the exit latency is above ~1 s. A 50 ms exit is a shell gate.

The reboot law

Runtime kernel knobs reset at every boot. ttm.pages_limit (caps GPU-mappable pinned pages) defaults to HALF OF RAM, so any manual echo N > /sys/... raise is a countdown to the next reboot — a power failure included. Measured on dankserver 2026-10-12 23:17: raise to 52,428,800 pages (200 GiB) -> power failure -> kernel default 32,253,039 pages (125,988 MiB) -> the pinned-ledger gate (floor 150,000) refused the start 6,919 times in ~20 h.

So: never prescribe a manual sysfs write as the recovery procedure for a service that can boot before a human reaches it. Make the START raise the knob.

Making the raise privileged

The service is User=dank, the knob is root-owned 0644 → [ -w "$knob" ] is FALSE in the service process and a plain echo fails silently. A "root unit" with User= is NOT a root process.

• Canonical: ExecStartPre=+/path/to/gate.sh --raise — systemd's + prefix runs as root even under User=, before ExecStart, so Restart=always can never outrun it.
• Fallback for a human running the gate from a shell: sudo -n sh -c "echo N > '$knob'".
• Gate order: read → if below floor, raise → RE-READ → refuse by name only after a write that was refused. Reading = min over the knobs that exist (an absent knob is skipped, never read as 0).
• Belt for the boot path: the module option (/etc/modprobe.d/*.conf, options ttm pages_limit=N) — Dankclaw persists its raise that way.

Verification of a knob-raise fix

• cat /sys/module/ttm/parameters/pages_limit after the boot equals the target.
• Journal shows the hook's own pid line (ttm-preflight.sh[...]: ... raising the knob to ...) BEFORE Started <svc>.service.
• Confirm the live shape from podman inspect <ctr> --format '{{json .Config.Cmd}}' — the restart succeeding proves nothing about the flags.
• Test the gate against a temp knob dir (env-overridable sysfs path): writable below-floor knob → raised + exit 0; unwritable → cap unchanged + exit 1. Run the refusal case with a PATH containing cat+bash and NO sudo, or root writes the 444 file and the case tests nothing.