---
name: service-preflight-refusal-triage
description: "Diagnose a systemd service crash-looping in activating (auto-restart) with a fast exit: separate a preflight refusal from an engine crash, and make a runtime sysfs/kernel knob raise survive the reboot that resets it."
---

# Preflight refusal vs engine crash (systemd services with env gates)

Symptom that decides the branch: unit `activating (auto-restart)`, `Main process
exited, code=exited, status=1/FAILURE` about 50 ms after start, restart counter
in the thousands. No container/app process, no app log.

## Triage sequence
1. `systemctl status <svc> --no-pager -n 0` → note `Exec` path, `status=`, CPU-seconds
   (`0.0xx ms` = the script never got to work).
2. `journalctl -u <svc> --no-pager -n 60` → the refusal line is printed by the
   serve script itself and NAMES the gate. On frank-archon:
   `ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000`.
   `journalctl -u <svc> | grep -m1 "<gate phrase>"` gives the FIRST occurrence:
   that timestamp is the incident start (correlate with `uptime -b`, `last reboot`).
3. Check the app's own state: `podman ps -a` showing the container name with a
   zero/absurd start time = the container was never created → the gate, not the
   engine.
4. Read the gate's code path in the repo (`scripts/serve-<svc>.sh`) before
   touching the host: it prints exactly what it requires.

Do NOT reach for `journalctl -k` GPU/OOM theories, gdb, or a payload checksum
until the exit latency is above ~1 s. A 50 ms exit is a shell gate.

## The reboot law
Runtime kernel knobs reset at every boot. `ttm.pages_limit` (caps GPU-mappable
pinned pages) defaults to HALF OF RAM, so any manual `echo N > /sys/...` raise is
a countdown to the next reboot — a power failure included. Measured on dankserver
2026-10-12 23:17: raise to 52,428,800 pages (200 GiB) -> power failure -> kernel
default 32,253,039 pages (125,988 MiB) -> the pinned-ledger gate (floor 150,000)
refused the start 6,919 times in ~20 h.

So: never prescribe a manual sysfs write as the recovery procedure for a service
that can boot before a human reaches it. Make the START raise the knob.

## Making the raise privileged
The service is `User=dank`, the knob is root-owned 0644 → `[ -w "$knob" ]` is
FALSE in the service process and a plain `echo` fails silently. A "root unit"
with `User=` is NOT a root process.
- Canonical: `ExecStartPre=+/path/to/gate.sh --raise` — systemd's `+` prefix runs
  as root even under `User=`, before ExecStart, so `Restart=always` can never
  outrun it.
- Fallback for a human running the gate from a shell: `sudo -n sh -c "echo N > '$knob'"`.
- Gate order: read → if below floor, raise → RE-READ → refuse by name only after
  a write that was refused. Reading = min over the knobs that exist (an absent
  knob is skipped, never read as 0).
- Belt for the boot path: the module option (`/etc/modprobe.d/*.conf`,
  `options ttm pages_limit=N`) — Dankclaw persists its raise that way.

## Verification of a knob-raise fix
- `cat /sys/module/ttm/parameters/pages_limit` after the boot equals the target.
- Journal shows the hook's own pid line (`ttm-preflight.sh[...]: ... raising the
  knob to ...`) BEFORE `Started <svc>.service`.
- Confirm the live shape from `podman inspect <ctr> --format '{{json .Config.Cmd}}'`
  — the restart succeeding proves nothing about the flags.
- Test the gate against a temp knob dir (env-overridable sysfs path): writable
  below-floor knob → raised + exit 0; unwritable → cap unchanged + exit 1. Run the
  refusal case with a PATH containing `cat`+`bash` and NO `sudo`, or root writes
  the 444 file and the case tests nothing.