command-center/agent/managed-skills/published-port-cutover-gate/SKILL.md

name, description

name	description
published-port-cutover-gate	When replacing or restarting a service that publishes a TCP port on this rootless-podman/pasta fleet, gate on the port actually being free first — otherwise pasta's bind fails at container create and the failure hides behind Restart=on-failure.

Published-port cutover gate (rootless podman / pasta)

On hosts where rootless podman publishes ports through pasta, the published port is bound in userspace at container CREATE, not when the app binds. A predecessor process that has just been stopped can still hold that port — listener, accepted connection, or TIME_WAIT residue from the connections it served — so a replacement started immediately after systemctl stop dies with:

pasta: ... bind: Address already in use
conmon ... exit status 126

and because the unit is usually Restart=on-failure/RestartSec=10, it self-heals seconds-to-minutes later. That is the trap: the deploy looks like it failed, then looks fine, and the real defect (no port gate) stays invisible because the retry masks it.

Observed on Dankclaw 2026-09-16 cutting frank-large from a root unit (ik_llama.cpp) to a rootless Quadlet (halogen): systemd logged "Deactivated successfully", and pasta tried to bind 7770 about a second later and reported "Address already in use". Only the quadlet's own restart policy recovered it.

Procedure

Before starting the replacement:

1. Retire the old unit and wait for it to be genuinely inactive. systemctl stop <old> (root) or systemctl --user stop <old>.

2. Stop the NEW unit too, if this is a redeploy. systemctl --user stop <new>.service || true

   Without this the gate deadlocks against itself: the currently-running replacement is the thing holding the port. This mistake was made and caught in the same session — a gate placed before the stop would have failed every subsequent deploy of an already-running service.

3. Gate on the port being empty, covering every socket state:

   wait_port_free() {
       local port="$1" limit="${2:-90}" i
       for ((i = 1; i <= limit; i++)); do
           if [[ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]]; then
               return 0
           fi
           sleep 1
       done
       echo "FAIL: port $port is still held after ${limit}s:" >&2
       ss -tan "sport = :$port" >&2 || true
       return 1
   }
   

   sport matches both the listener and the server side of accepted connections, so this also catches TIME_WAIT residue that a plain ss -ltn check misses. Fail loudly with the socket table attached; a deploy that starts a doomed container has not deployed anything.

4. Then install/pull and start, and wait on the service's health endpoint.

Verifying the gate has teeth

• Re-run the same deploy: it must pass with the service already running (that is the idempotency case the self-deadlock broke).
• Reproduce the failure once on purpose if the service is not user-facing: stop the container and start the replacement with the gate bypassed, and confirm Address already in use / exit 126.
• Assert it in the repo's seam tests: e.g. grep -q 'wait_port_free <port>' and grep -q '^wait_port_free() {' in the deploy handler, so a later refactor cannot quietly drop the gate.

Related traps

• systemctl disable on some of these hosts deletes the unit file rather than just the symlink, so a "move the unit to trash" step must tolerate a file that is already gone — and should say so, or "no message" carries no evidence.
• Never rm; move retired units into trash ($HOME/.AGENTTRASH/<date>-<topic>/).
• A rootless Quadlet's generated unit cannot be systemctl --user enabled; boot persistence comes from [Install] WantedBy=default.target in the .container. To keep a masked/failed quadlet from being regenerated, move the .container out of ~/.config/containers/systemd/ and daemon-reload — the generator's source is the lever, not mask.