63 lines
2.9 KiB
Markdown
63 lines
2.9 KiB
Markdown
---
|
|
name: gluetun-dependent-boot-race-recovery
|
|
description: "Diagnose the Dankclaw pattern where gluetun is active but its netns-sharing dependents (searxng, marreta, marreta-browser) are inactive (dead) with \"Dependency failed\" after a reboot — cascade-recover them and tie it to e2e 502/SSO failures."
|
|
---
|
|
|
|
# Gluetun boot-race: netns dependents dead at boot, service is active
|
|
|
|
## Signature
|
|
After a dankclaw reboot, `gluetun.service` is ACTIVE, yet e2e fails on
|
|
`search.franksdank.cloud` (502 on the authenticated pass), the Marreta tile
|
|
shows a blank interstitial, and the homepage crawl reports vhost resets.
|
|
`systemctl --user status <dep> --no-pager` shows:
|
|
|
|
```
|
|
Active: inactive (dead)
|
|
<host> systemd[2224]: Dependency failed for <dep>.service
|
|
<host> systemd[2224]: <dep>.service: Job ... failed with result 'dependency'
|
|
```
|
|
|
|
## Cause
|
|
Quadlet `Network=gluetun.container` emits `Requires=`/`After=` on
|
|
`gluetun.service`. When gluetun's start job fails once transiently at boot
|
|
and `Restart=` re-arms it seconds later, the dependents' start jobs are
|
|
already consumed and are NEVER re-armed — they sit `inactive (dead)`
|
|
indefinitely while gluetun runs. `Restart=always` on the dependent does not
|
|
help: the service never ran, so the restart policy never armed.
|
|
|
|
## Diagnosis census (rootless, run as `dank`, NO sudo)
|
|
```
|
|
export XDG_RUNTIME_DIR=/run/user/$(id -u)
|
|
uptime # reboot is the trigger
|
|
systemctl --user is-active gluetun searxng searxng-valkey marreta marreta-browser
|
|
systemctl --user status <dep> --no-pager | head # "Dependency failed" line
|
|
journalctl --user --since "<boot>" | grep -iE "<dep>|dependency"
|
|
```
|
|
Dependents of the gluetun netns on dankclaw: `searxng`, `marreta`,
|
|
`marreta-browser` (valkey is plain-networked and survives).
|
|
|
|
## Recovery (order matters: browser before marreta)
|
|
```
|
|
systemctl --user start marreta-browser marreta
|
|
systemctl --user start searxng
|
|
```
|
|
Verify: `podman ps` shows them; `curl http://127.0.0.1:8081/healthz` (searxng)
|
|
and gluetun-published 8082 (marreta) answer; then rerun the Playwright gate
|
|
(`tests/e2e/run.sh`) — the 12-crawl / 13-sso / 16-marreta specs are the trip
|
|
wires for this fault.
|
|
|
|
## Distinguish from the dead-pause-process failure
|
|
The 2026-09-16 rootless-pause-process death breaks ALL podman commands and
|
|
every quadlet loops; this fault leaves podman fully functional and only the
|
|
gluetun-dependency chain is dead. The reboot repairs the former; manual
|
|
cascade-start repairs this one.
|
|
|
|
## Open gap (dankclaw, as of 2026-10-15)
|
|
The race recurred (observed 2026-10-15 boot 01:41). Quadlet law for these
|
|
units has no cure inside the `.container` files themselves; options are a
|
|
systemd user-level drop-in on gluetun
|
|
(`ExecStartPost=/usr/bin/systemctl --user restart searxng marreta`) or
|
|
converting the dependents' dependency to `BindsTo` + `StartLimit` retry
|
|
semantics. Fleet precedent for boot-order pain: searxng/gluetun ordering,
|
|
observability reboot gaps (AGENTS.md).
|