Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
89 lines
4.0 KiB
Markdown
89 lines
4.0 KiB
Markdown
---
|
|
name: published-port-cutover-gate
|
|
description: "When replacing or restarting a service that publishes a TCP port on this rootless-podman/pasta fleet, gate on the port actually being free first — otherwise pasta's bind fails at container create and the failure hides behind Restart=on-failure."
|
|
---
|
|
|
|
# Published-port cutover gate (rootless podman / pasta)
|
|
|
|
On hosts where **rootless podman publishes ports through pasta**, the published
|
|
port is bound in userspace at **container CREATE**, not when the app binds. A
|
|
predecessor process that has just been stopped can still hold that port —
|
|
listener, accepted connection, or TIME_WAIT residue from the connections it
|
|
served — so a replacement started immediately after `systemctl stop` dies with:
|
|
|
|
```
|
|
pasta: ... bind: Address already in use
|
|
conmon ... exit status 126
|
|
```
|
|
|
|
and because the unit is usually `Restart=on-failure`/`RestartSec=10`, it
|
|
**self-heals seconds-to-minutes later**. That is the trap: the deploy looks
|
|
like it failed, then looks fine, and the real defect (no port gate) stays
|
|
invisible because the retry masks it.
|
|
|
|
Observed on Dankclaw 2026-09-16 cutting `frank-large` from a root unit
|
|
(ik_llama.cpp) to a rootless Quadlet (halogen): `systemd` logged
|
|
"Deactivated successfully", and pasta tried to bind 7770 about a second later
|
|
and reported "Address already in use". Only the quadlet's own restart policy
|
|
recovered it.
|
|
|
|
## Procedure
|
|
|
|
Before starting the replacement:
|
|
|
|
1. **Retire the old unit and wait for it to be genuinely inactive.**
|
|
`systemctl stop <old>` (root) or `systemctl --user stop <old>`.
|
|
2. **Stop the NEW unit too, if this is a redeploy.**
|
|
`systemctl --user stop <new>.service || true`
|
|
|
|
Without this the gate deadlocks against itself: the currently-running
|
|
replacement is the thing holding the port. This mistake was made and caught
|
|
in the same session — a gate placed before the stop would have failed every
|
|
subsequent deploy of an already-running service.
|
|
3. **Gate on the port being empty, covering every socket state:**
|
|
|
|
```bash
|
|
wait_port_free() {
|
|
local port="$1" limit="${2:-90}" i
|
|
for ((i = 1; i <= limit; i++)); do
|
|
if [[ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]]; then
|
|
return 0
|
|
fi
|
|
sleep 1
|
|
done
|
|
echo "FAIL: port $port is still held after ${limit}s:" >&2
|
|
ss -tan "sport = :$port" >&2 || true
|
|
return 1
|
|
}
|
|
```
|
|
|
|
`sport` matches both the listener and the server side of accepted
|
|
connections, so this also catches TIME_WAIT residue that a plain
|
|
`ss -ltn` check misses. Fail loudly with the socket table attached; a deploy
|
|
that starts a doomed container has not deployed anything.
|
|
4. **Then** install/pull and start, and wait on the service's health endpoint.
|
|
|
|
## Verifying the gate has teeth
|
|
|
|
- Re-run the same deploy: it must pass with the service already running (that
|
|
is the idempotency case the self-deadlock broke).
|
|
- Reproduce the failure once on purpose if the service is not user-facing:
|
|
stop the container and start the replacement with the gate bypassed, and
|
|
confirm `Address already in use` / exit 126.
|
|
- Assert it in the repo's seam tests: e.g. `grep -q 'wait_port_free <port>'`
|
|
and `grep -q '^wait_port_free() {'` in the deploy handler, so a later
|
|
refactor cannot quietly drop the gate.
|
|
|
|
## Related traps
|
|
|
|
- `systemctl disable` on some of these hosts **deletes the unit file** rather
|
|
than just the symlink, so a "move the unit to trash" step must tolerate a
|
|
file that is already gone — and should say so, or "no message" carries no
|
|
evidence.
|
|
- Never `rm`; move retired units into trash (`$HOME/.AGENTTRASH/<date>-<topic>/`).
|
|
- A rootless Quadlet's generated unit **cannot** be `systemctl --user enable`d;
|
|
boot persistence comes from `[Install] WantedBy=default.target` in the
|
|
`.container`. To keep a masked/failed quadlet from being regenerated, move
|
|
the `.container` out of `~/.config/containers/systemd/` and `daemon-reload` —
|
|
the generator's source is the lever, not `mask`.
|