serverAppConfigs/serverAppConfigs-docs/Operations/Container Host Reachability.md

---
type: reference
title: Container Host Reachability
description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate.
tags: [podman, netavark, networking, rootless, reachability, caddy]
timestamp: 2026-10-05T00:00:00-07:00
---

# Container Host Reachability

Two failure families on rootless podman + netavark hosts (Dankclaw), both
reading as "service down" when the service is up and the *path* is wrong.
Fleet source of truth: `serverAppConfigs-docs/Architecture/Container Network
Namespaces.md` (commit `2e179b0`).

## Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly

A container cannot reach a `Network=host` service at the host's LAN IP.
netavark DNATs **published** ports only: from a bridge container,
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
gateway `10.89.1.1` are both refused. This is what 502'd
`hermes.franksdank.cloud` until the Caddyfile upstream became
`host.containers.internal` on 2026-09-16 — its vhost comment had claimed the
LAN IP was open from that bridge; it never was.

Probe that proves it in one command:

```bash
ssh <host> "podman run --rm --network <net> --entrypoint sh \
  docker.io/library/busybox:latest -c \
  'for p in <target-port> <published-port>; do
     nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
   done'"
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'
```

Published OK + target BAD ⇒ quirk confirmed. Note busybox `nc -z` reports any
failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when
refusal-vs-timeout matters.

Fix: point the service's host-facing URL at
`http://host.containers.internal:<port>` in compose/quadlet env, redeploy via
git (`deploy.sh <target> --service <svc>`), verify
`podman exec <svc> printenv <URL_VAR>` and the connect log line (~40 s).
**Do not publish the daemon port** to make the LAN IP work — publishing is
exactly what makes other ports refuse.

## Quirk 2 — addressability is per netns, not per host

- `NetworkMode=host` → the container **is** the host netns: dials all of
  `127.0.0.1`, binds any free port.
- bridge/named net → own netns; only published ports are reachable, via the
  network's **gateway IP**. Unpublished host listeners refuse **instantly
  (RST)** — no firewall rule, no stale DNAT, `nft`/`iptables -t nat` empty. Do
  not hunt for REJECT rules; probe.

Probe sequence (read-only):

```bash
podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
sudo ss -ltnp 'sport = :<port>'        # host netns only — why 0.0.0.0 can still be unreachable
podman exec <name> cat /proc/net/route  # default route = the gateway IP to dial
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'
```

Interpretation without blurring:

- **instant refusal** = nothing listening in a reachable namespace (or
  rootlessport rejected an unpublished port map).
- **timeout** = no route to that address at all. `host.containers.internal`
  (`169.254.1.2`) is an alias on the **default `podman` net only**; containers
  on a named network time out against it.

Tool gotchas (podman 5.x, verified live):

- `podman network inspect <net> --format '{{json .containers}}'` errors
  (`can't evaluate field containers in type interface {}`) — list membership
  from the container side: `podman ps -a --format '{{.Names}} {{.Networks}}'`.
- An **empty** `Networks` field = host network or shared namespace
  (`--network=container:<other>`), not "no network".
- Quadlet `Network=<name>.network` yields the runtime name **`systemd-<name>`**
  — probe with `podman network ls`, never the `.network` filename.

Fix options when a bridge-net container must reach a host-netns listener:
(a) move it to `Network=host`, or (b) publish the port — (b) makes the port
reachable from every namespace **and the LAN**, an exposure change needing
explicit operator approval, never a silent fix.

## Related trap — the pasta port gate (rootless published ports)

On hosts publishing ports through pasta, the port is bound in **userspace at
container CREATE**, not when the app binds. A predecessor stopped moments
earlier can still hold listener/TIME_WAIT state, so the replacement dies
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
seconds later** under `Restart=on-failure`. The failure looks like it failed,
then looks fine, and the missing gate stays invisible because the retry masks
it. Observed 2026-09-16 cutting `frank-large` from a root unit to a rootless
Quadlet.

Gate procedure: stop the old unit AND the new one (a redeploy's running
replacement is itself the port-holder — without that stop the gate deadlocks
against itself), then wait for the port to be empty across **all socket
states**:

```bash
wait_port_free() {
    local port="$1" limit="${2:-90}" i
    for ((i = 1; i <= limit; i++)); do
        if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
        sleep 1
    done
    echo "FAIL: port $port still held after ${limit}s:" >&2
    ss -tan "sport = :$port" >&2 || true
    return 1
}
```

`sport` catches accepted-connection server sockets and TIME_WAIT residue that
`ss -ltn` misses; fail loudly with the socket table attached. Verify the gate
by re-running the same deploy against a running service (idempotency) and
assert it in seam tests (`wait_port_free` present in the deploy handler).

## Related

- [[Quadlet Lifecycle]] — the unit layer these containers run under
- [[Container Network Namespaces]] — fleet network namespace source of truth
- [[Observability]] — the alerting paths that depend on reachability
- [[Deploy State Triage]] — the deploy flow that installs these configs