serverAppConfigs/serverAppConfigs-docs/Operations/Container Host Reachability.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Container Host Reachability	Two reachability quirks: containers cannot dial unpublished host ports, and netns membership decides reachability.	podman netavark networking rootless reachability caddy	2026-10-05T00:00:00-07:00

Container Host Reachability

Two failure families on rootless podman + netavark hosts (Dankclaw), both reading as "service down" when the service is up and the path is wrong. Fleet source of truth: serverAppConfigs-docs/Architecture/Container Network Namespaces.md (commit 2e179b0).

Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly

A container cannot reach a Network=host service at the host's LAN IP. netavark DNATs published ports only: from a bridge container, 192.168.0.143:<published> answers while 192.168.0.143:<host bind> and the gateway 10.89.1.1 are both refused. This is what 502'd hermes.franksdank.cloud until the Caddyfile upstream became host.containers.internal on 2026-09-16 — its vhost comment had claimed the LAN IP was open from that bridge; it never was.

Probe that proves it in one command:

ssh <host> "podman run --rm --network <net> --entrypoint sh \
  docker.io/library/busybox:latest -c \
  'for p in <target-port> <published-port>; do
     nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
   done'"
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'

Published OK + target BAD ⇒ quirk confirmed. Note busybox nc -z reports any failure as failure — use it for bulk sweeps; use /dev/tcp (or nc -v) when refusal-vs-timeout matters.

Fix: point the service's host-facing URL at http://host.containers.internal:<port> in compose/quadlet env, redeploy git (deploy.sh <target> --service <svc>), verify podman exec <svc> printenv <URL_VAR> and the connect log line (~40 s). Do not publish the daemon port to make the LAN IP work — publishing is exactly what makes other ports refuse.

Quirk 2 — addressability is per netns, not per host

• NetworkMode=host → the container is the host netns: dials all of 127.0.0.1, binds any free port.
• bridge/named net → own netns; only published ports are reachable, via the network's gateway IP. Unpublished host listeners refuse instantly (RST) — no firewall rule, no stale DNAT, nft/iptables -t nat empty. Do not hunt for REJECT rules; probe.

Probe sequence (read-only):

podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
sudo ss -ltnp 'sport = :<port>'        # host netns only — why 0.0.0.0 can still be unreachable
podman exec <name> cat /proc/net/route  # default route = the gateway IP to dial
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'

Interpretation without blurring:

• instant refusal = nothing listening in a reachable namespace (or rootlessport rejected an unpublished port map).
• timeout = no route to that address at all. host.containers.internal (169.254.1.2) is an alias on the default podman net only; containers on a named network time out against it.

Tool gotchas (podman 5.x, verified live):

• podman network inspect <net> --format '{{json .containers}}' errors (can't evaluate field containers in type interface {}) — list membership from the container side: podman ps -a --format '{{.Names}} {{.Networks}}'.
• An empty Networks field = host network or shared namespace (--network=container:<other>), not "no network".
• Quadlet Network=<name>.network yields the runtime name systemd-<name> — probe with podman network ls, never the .network filename.

Fix options when a bridge-net container must reach a host-netns listener: (a) move it to Network=host, or (b) publish the port — (b) makes the port reachable from every namespace and the LAN, an exposure change needing explicit operator approval, never a silent fix.

Related trap — the pasta port gate (rootless published ports)

On hosts publishing ports through pasta, the port is bound in userspace at container CREATE, not when the app binds. A predecessor stopped moments earlier can still hold listener/TIME_WAIT state, so the replacement dies pasta: ... bind: Address already in use (conmon exit 126) — then self-heals seconds later under Restart=on-failure. The failure looks like it failed, then looks fine, and the missing gate stays invisible because the retry masks it. First seen on a frank-dragoon unit cutover; which supervision shape that era was is in frank-dragoon's era table.

Gate procedure: stop the old unit AND the new one (a redeploy's running replacement is itself the port-holder — without that stop the gate deadlocks against itself), then wait for the port to be empty across all socket states:

wait_port_free() {
    local port="$1" limit="${2:-90}" i
    for ((i = 1; i <= limit; i++)); do
        if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
        sleep 1
    done
    echo "FAIL: port $port still held after ${limit}s:" >&2
    ss -tan "sport = :$port" >&2 || true
    return 1
}

sport catches accepted-connection server sockets and TIME_WAIT residue that ss -ltn misses; fail loudly with the socket table attached. Verify the gate by re-running the same deploy against a running service (idempotency) and assert it in seam tests (wait_port_free present in the deploy handler).

Related

• Quadlet Lifecycle — the unit layer these containers run under
• Container Network Namespaces — fleet network namespace source of truth
• Observability — the alerting paths that depend on reachability
• Deploy State Triage — the deploy flow that installs these configs