serverAppConfigs/serverAppConfigs-docs/Operations/Container Host Reachability.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Container Host Reachability	The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.	podman netavark networking rootless reachability caddy	2026-10-05T00:00:00-07:00

Container Host Reachability

Two failure families on rootless podman + netavark hosts (Dankclaw), both reading as "service down" when the service is up and the path is wrong. Fleet source of truth: serverAppConfigs-docs/Architecture/Container Network Namespaces.md (commit 2e179b0).

Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly

A container cannot reach a Network=host service at the host's LAN IP. netavark DNATs published ports only: from a bridge container, 192.168.0.143:<published> answers while 192.168.0.143:<host bind> and the gateway 10.89.1.1 are both refused. This is what 502'd hermes.franksdank.cloud until the Caddyfile upstream became host.containers.internal on 2026-09-16 — its vhost comment had claimed the LAN IP was open from that bridge; it never was.

Probe that proves it in one command:

ssh <host> "podman run --rm --network <net> --entrypoint sh \
  docker.io/library/busybox:latest -c \
  'for p in <target-port> <published-port>; do
     nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
   done'"
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'

Published OK + target BAD ⇒ quirk confirmed. Note busybox nc -z reports any failure as failure — use it for bulk sweeps; use /dev/tcp (or nc -v) when refusal-vs-timeout matters.

Fix: point the service's host-facing URL at http://host.containers.internal:<port> in compose/quadlet env, redeploy via git (deploy.sh <target> --service <svc>), verify podman exec <svc> printenv <URL_VAR> and the connect log line (~40 s). Do not publish the daemon port to make the LAN IP work — publishing is exactly what makes other ports refuse.

Quirk 2 — addressability is per netns, not per host

• NetworkMode=host → the container is the host netns: dials all of 127.0.0.1, binds any free port.
• bridge/named net → own netns; only published ports are reachable, via the network's gateway IP. Unpublished host listeners refuse instantly (RST) — no firewall rule, no stale DNAT, nft/iptables -t nat empty. Do not hunt for REJECT rules; probe.

Probe sequence (read-only):

podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
sudo ss -ltnp 'sport = :<port>'        # host netns only — why 0.0.0.0 can still be unreachable
podman exec <name> cat /proc/net/route  # default route = the gateway IP to dial
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'

Interpretation without blurring:

• instant refusal = nothing listening in a reachable namespace (or rootlessport rejected an unpublished port map).
• timeout = no route to that address at all. host.containers.internal (169.254.1.2) is an alias on the default podman net only; containers on a named network time out against it.

Tool gotchas (podman 5.x, verified live):

• podman network inspect <net> --format '{{json .containers}}' errors (can't evaluate field containers in type interface {}) — list membership from the container side: podman ps -a --format '{{.Names}} {{.Networks}}'.
• An empty Networks field = host network or shared namespace (--network=container:<other>), not "no network".
• Quadlet Network=<name>.network yields the runtime name systemd-<name> — probe with podman network ls, never the .network filename.

Fix options when a bridge-net container must reach a host-netns listener: (a) move it to Network=host, or (b) publish the port — (b) makes the port reachable from every namespace and the LAN, an exposure change needing explicit operator approval, never a silent fix.

Related trap — the pasta port gate (rootless published ports)

On hosts publishing ports through pasta, the port is bound in userspace at container CREATE, not when the app binds. A predecessor stopped moments earlier can still hold listener/TIME_WAIT state, so the replacement dies pasta: ... bind: Address already in use (conmon exit 126) — then self-heals seconds later under Restart=on-failure. The failure looks like it failed, then looks fine, and the missing gate stays invisible because the retry masks it. Observed 2026-09-16 cutting frank-large from a root unit to a rootless Quadlet.

Gate procedure: stop the old unit AND the new one (a redeploy's running replacement is itself the port-holder — without that stop the gate deadlocks against itself), then wait for the port to be empty across all socket states:

wait_port_free() {
    local port="$1" limit="${2:-90}" i
    for ((i = 1; i <= limit; i++)); do
        if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
        sleep 1
    done
    echo "FAIL: port $port still held after ${limit}s:" >&2
    ss -tan "sport = :$port" >&2 || true
    return 1
}

sport catches accepted-connection server sockets and TIME_WAIT residue that ss -ltn misses; fail loudly with the socket table attached. Verify the gate by re-running the same deploy against a running service (idempotency) and assert it in seam tests (wait_port_free present in the deploy handler).

Related

• Quadlet Lifecycle — the unit layer these containers run under
• Container Network Namespaces — fleet network namespace source of truth
• Observability — the alerting paths that depend on reachability
• Deploy State Triage — the deploy flow that installs these configs