live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/gluetun-dependent-boot-race-recovery/SKILL.md

@@ -0,0 +1,62 @@
+---
+name: gluetun-dependent-boot-race-recovery
+description: "Diagnose the Dankclaw pattern where gluetun is active but its netns-sharing dependents (searxng, marreta, marreta-browser) are inactive (dead) with \"Dependency failed\" after a reboot — cascade-recover them and tie it to e2e 502/SSO failures."
+---
+
+# Gluetun boot-race: netns dependents dead at boot, service is active
+
+## Signature
+After a dankclaw reboot, `gluetun.service` is ACTIVE, yet e2e fails on
+`search.franksdank.cloud` (502 on the authenticated pass), the Marreta tile
+shows a blank interstitial, and the homepage crawl reports vhost resets.
+`systemctl --user status <dep> --no-pager` shows:
+
+```
+Active: inactive (dead)
+<host> systemd[2224]: Dependency failed for <dep>.service
+<host> systemd[2224]: <dep>.service: Job ... failed with result 'dependency'
+```
+
+## Cause
+Quadlet `Network=gluetun.container` emits `Requires=`/`After=` on
+`gluetun.service`. When gluetun's start job fails once transiently at boot
+and `Restart=` re-arms it seconds later, the dependents' start jobs are
+already consumed and are NEVER re-armed — they sit `inactive (dead)`
+indefinitely while gluetun runs. `Restart=always` on the dependent does not
+help: the service never ran, so the restart policy never armed.
+
+## Diagnosis census (rootless, run as `dank`, NO sudo)
+```
+export XDG_RUNTIME_DIR=/run/user/$(id -u)
+uptime   # reboot is the trigger
+systemctl --user is-active gluetun searxng searxng-valkey marreta marreta-browser
+systemctl --user status <dep> --no-pager | head  # "Dependency failed" line
+journalctl --user --since "<boot>" | grep -iE "<dep>|dependency"
+```
+Dependents of the gluetun netns on dankclaw: `searxng`, `marreta`,
+`marreta-browser` (valkey is plain-networked and survives).
+
+## Recovery (order matters: browser before marreta)
+```
+systemctl --user start marreta-browser marreta
+systemctl --user start searxng
+```
+Verify: `podman ps` shows them; `curl http://127.0.0.1:8081/healthz` (searxng)
+and gluetun-published 8082 (marreta) answer; then rerun the Playwright gate
+(`tests/e2e/run.sh`) — the 12-crawl / 13-sso / 16-marreta specs are the trip
+wires for this fault.
+
+## Distinguish from the dead-pause-process failure
+The 2026-09-16 rootless-pause-process death breaks ALL podman commands and
+every quadlet loops; this fault leaves podman fully functional and only the
+gluetun-dependency chain is dead. The reboot repairs the former; manual
+cascade-start repairs this one.
+
+## Open gap (dankclaw, as of 2026-10-15)
+The race recurred (observed 2026-10-15 boot 01:41). Quadlet law for these
+units has no cure inside the `.container` files themselves; options are a
+systemd user-level drop-in on gluetun
+(`ExecStartPost=/usr/bin/systemctl --user restart searxng marreta`) or
+converting the dependents' dependency to `BindsTo` + `StartLimit` retry
+semantics. Fleet precedent for boot-order pain: searxng/gluetun ordering,
+observability reboot gaps (AGENTS.md).