---
name: published-port-cutover-gate
description: "When replacing or restarting a service that publishes a TCP port on this rootless-podman/pasta fleet, gate on the port actually being free first — otherwise pasta's bind fails at container create and the failure hides behind Restart=on-failure."
---

# Published-port cutover gate (rootless podman / pasta)

On hosts where **rootless podman publishes ports through pasta**, the published
port is bound in userspace at **container CREATE**, not when the app binds. A
predecessor process that has just been stopped can still hold that port —
listener, accepted connection, or TIME_WAIT residue from the connections it
served — so a replacement started immediately after `systemctl stop` dies with:

```
pasta: ... bind: Address already in use
conmon ... exit status 126
```

and because the unit is usually `Restart=on-failure`/`RestartSec=10`, it
**self-heals seconds-to-minutes later**. That is the trap: the deploy looks
like it failed, then looks fine, and the real defect (no port gate) stays
invisible because the retry masks it.

Observed on Dankclaw 2026-09-16 cutting `frank-large` from a root unit
(ik_llama.cpp) to a rootless Quadlet (halogen): `systemd` logged
"Deactivated successfully", and pasta tried to bind 7770 about a second later
and reported "Address already in use". Only the quadlet's own restart policy
recovered it.

## Procedure

Before starting the replacement:

1. **Retire the old unit and wait for it to be genuinely inactive.**
   `systemctl stop <old>` (root) or `systemctl --user stop <old>`.
2. **Stop the NEW unit too, if this is a redeploy.**
   `systemctl --user stop <new>.service || true`

   Without this the gate deadlocks against itself: the currently-running
   replacement is the thing holding the port. This mistake was made and caught
   in the same session — a gate placed before the stop would have failed every
   subsequent deploy of an already-running service.
3. **Gate on the port being empty, covering every socket state:**

   ```bash
   wait_port_free() {
       local port="$1" limit="${2:-90}" i
       for ((i = 1; i <= limit; i++)); do
           if [[ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]]; then
               return 0
           fi
           sleep 1
       done
       echo "FAIL: port $port is still held after ${limit}s:" >&2
       ss -tan "sport = :$port" >&2 || true
       return 1
   }
   ```

   `sport` matches both the listener and the server side of accepted
   connections, so this also catches TIME_WAIT residue that a plain
   `ss -ltn` check misses. Fail loudly with the socket table attached; a deploy
   that starts a doomed container has not deployed anything.
4. **Then** install/pull and start, and wait on the service's health endpoint.

## Verifying the gate has teeth

- Re-run the same deploy: it must pass with the service already running (that
  is the idempotency case the self-deadlock broke).
- Reproduce the failure once on purpose if the service is not user-facing:
  stop the container and start the replacement with the gate bypassed, and
  confirm `Address already in use` / exit 126.
- Assert it in the repo's seam tests: e.g. `grep -q 'wait_port_free <port>'`
  and `grep -q '^wait_port_free() {'` in the deploy handler, so a later
  refactor cannot quietly drop the gate.

## Related traps

- `systemctl disable` on some of these hosts **deletes the unit file** rather
  than just the symlink, so a "move the unit to trash" step must tolerate a
  file that is already gone — and should say so, or "no message" carries no
  evidence.
- Never `rm`; move retired units into trash (`$HOME/.AGENTTRASH/<date>-<topic>/`).
- A rootless Quadlet's generated unit **cannot** be `systemctl --user enable`d;
  boot persistence comes from `[Install] WantedBy=default.target` in the
  `.container`. To keep a masked/failed quadlet from being regenerated, move
  the `.container` out of `~/.config/containers/systemd/` and `daemon-reload` —
  the generator's source is the lever, not `mask`.