live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/service-preflight-refusal-triage/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: service-preflight-refusal-triage
+description: "Diagnose a systemd service crash-looping in activating (auto-restart) with a fast exit: separate a preflight refusal from an engine crash, and make a runtime sysfs/kernel knob raise survive the reboot that resets it."
+---
+
+# Preflight refusal vs engine crash (systemd services with env gates)
+
+Symptom that decides the branch: unit `activating (auto-restart)`, `Main process
+exited, code=exited, status=1/FAILURE` about 50 ms after start, restart counter
+in the thousands. No container/app process, no app log.
+
+## Triage sequence
+1. `systemctl status <svc> --no-pager -n 0` → note `Exec` path, `status=`, CPU-seconds
+   (`0.0xx ms` = the script never got to work).
+2. `journalctl -u <svc> --no-pager -n 60` → the refusal line is printed by the
+   serve script itself and NAMES the gate. On frank-archon:
+   `ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000`.
+   `journalctl -u <svc> | grep -m1 "<gate phrase>"` gives the FIRST occurrence:
+   that timestamp is the incident start (correlate with `uptime -b`, `last reboot`).
+3. Check the app's own state: `podman ps -a` showing the container name with a
+   zero/absurd start time = the container was never created → the gate, not the
+   engine.
+4. Read the gate's code path in the repo (`scripts/serve-<svc>.sh`) before
+   touching the host: it prints exactly what it requires.
+
+Do NOT reach for `journalctl -k` GPU/OOM theories, gdb, or a payload checksum
+until the exit latency is above ~1 s. A 50 ms exit is a shell gate.
+
+## The reboot law
+Runtime kernel knobs reset at every boot. `ttm.pages_limit` (caps GPU-mappable
+pinned pages) defaults to HALF OF RAM, so any manual `echo N > /sys/...` raise is
+a countdown to the next reboot — a power failure included. Measured on dankserver
+2026-10-12 23:17: raise to 52,428,800 pages (200 GiB) -> power failure -> kernel
+default 32,253,039 pages (125,988 MiB) -> the pinned-ledger gate (floor 150,000)
+refused the start 6,919 times in ~20 h.
+
+So: never prescribe a manual sysfs write as the recovery procedure for a service
+that can boot before a human reaches it. Make the START raise the knob.
+
+## Making the raise privileged
+The service is `User=dank`, the knob is root-owned 0644 → `[ -w "$knob" ]` is
+FALSE in the service process and a plain `echo` fails silently. A "root unit"
+with `User=` is NOT a root process.
+- Canonical: `ExecStartPre=+/path/to/gate.sh --raise` — systemd's `+` prefix runs
+  as root even under `User=`, before ExecStart, so `Restart=always` can never
+  outrun it.
+- Fallback for a human running the gate from a shell: `sudo -n sh -c "echo N > '$knob'"`.
+- Gate order: read → if below floor, raise → RE-READ → refuse by name only after
+  a write that was refused. Reading = min over the knobs that exist (an absent
+  knob is skipped, never read as 0).
+- Belt for the boot path: the module option (`/etc/modprobe.d/*.conf`,
+  `options ttm pages_limit=N`) — Dankclaw persists its raise that way.
+
+## Verification of a knob-raise fix
+- `cat /sys/module/ttm/parameters/pages_limit` after the boot equals the target.
+- Journal shows the hook's own pid line (`ttm-preflight.sh[...]: ... raising the
+  knob to ...`) BEFORE `Started <svc>.service`.
+- Confirm the live shape from `podman inspect <ctr> --format '{{json .Config.Cmd}}'`
+  — the restart succeeding proves nothing about the flags.
+- Test the gate against a temp knob dir (env-overridable sysfs path): writable
+  below-floor knob → raised + exit 0; unwritable → cap unchanged + exit 1. Run the
+  refusal case with a PATH containing `cat`+`bash` and NO `sudo`, or root writes
+  the 444 file and the case tests nothing.