Files
serverAppConfigs/serverAppConfigs-docs/Architecture/Container Network Namespaces.md
T
Frank 2e179b024d docs(hermes): document namespace addressability and fix stale claims
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.

Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
  /home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
  daemon itself is a host user unit (signal-cli-daemon.service) bound to
  0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
  compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
  GET /health from another LAN machine returned 200) — the SSH tunnel is an access
  path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
  Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
  tunnel via the shared named network searxng.network, not service:gluetun.

Docs only — deploy is a no-op for this change.
2026-09-02 16:10:57 -07:00

6.5 KiB

type, title, description, tags, timestamp
type title description tags timestamp
reference Container Network Namespaces Which local services are addressable from which Podman network namespace on Dankclaw, and why bridge-network containers cannot dial host-netns loopback listeners.
networking
podman
netavark
rootless
dankclaw
2026-09-02

Container Network Namespaces

On a rootless-Podman host, "talk to a local service" is not one question — it is one question per network namespace. A container in the host network namespace sees every host listener on 127.0.0.1; a container on a bridge network does not, and cannot be made to by name or by loopback. This page is the address map for Dankclaw (192.168.0.143) and the rule that explains it.

The rule

A container only reaches services that are reachable from inside its own network namespace:

  • Network=host → the container is in the host netns. It dials other host services on loopback (127.0.0.1:PORT) and binds any free port itself.
  • Network=<name>.network / Network=dankclaw_default (bridge, netavark) → the container has its own netns. Loopback inside it is its own loopback. The only host address it can dial is the network's gateway IP, and on a rootless host that path is served by rootlessport, which forwards published port maps only. An unpublished host listener is refused immediately (RST), not routed.
  • Container DNS names (gitea-server, authentik-postgres, valkey) resolve only inside the network that owns them, so a container on another network can never reach a peer by name — see the naming quirk below.

Live address map (Dankclaw, verified 2026-09-02)

Container NetworkMode Reachable from it
systemd-hermes (gateway) host everything on the host netns, incl. signal-cli at 127.0.0.1:8080 and the webui on :8787
hermes-webui host same; binds 0.0.0.0:8787, so it is LAN-reachable, not loopback-only
chrome-devtools-protocol host binds 127.0.0.1:9222 — deliberately loopback-only, unreachable from the LAN and from bridge nets
hermes-dashboard dankclaw_default (bridge) only published ports, via the gateway IP 10.89.1.1

Network= values are read from the live Quadlet sources in /home/dank/.config/containers/systemd/*.container; runtime network names on the host at the same date were podman, dankclaw_default, systemd-gitea, systemd-searxng, systemd-open-webui.

Two consequences that are easy to miss:

  1. The Hermes gateway can dial the Signal daemon on loopback only because it runs in the host netns. The dashboard — same image, same /data — cannot; it shares state through the bind mount instead, never over the network.
  2. Caddy sits on dankclaw_default, so every upstream it proxies must be reachable from that network by IP + published port. Cross-network traffic by container DNS name is not available to it: gitea-server lives on systemd-gitea, valkey on systemd-searxng, and neither resolves from Caddy's namespace. That is why Gitea is fronted as IP:3000 and SearXNG as :8081 rather than by service name.

Naming quirk: <name>.network becomes systemd-<name>

A Quadlet Network=gitea.network yields a runtime network named systemd-gitea, not gitea. Probe with podman network ls, never by the filename, or health checks and Network-Alias plans target a network that does not exist.

Verification recipe (read-only)

# which containers are on which network? (`podman network inspect` in this
# podman version has no `.containers` field, so list from the container side)
podman ps -a --format '{{.Names}} {{.Networks}}'   # empty Networks = host or shared netns
# what listens on the host netns, and who owns it?
sudo ss -ltnp 'sport = :8080'

# what can a bridge-net container actually dial? run the probe inside its netns
podman exec <name> cat /proc/net/route          # default route → gateway IP
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?'

Verified outcomes of that recipe on 2026-09-02: from inside hermes-dashboard, 10.89.1.1:3000 (Gitea, published), :8081 (SearXNG via Gluetun's netns, published) and :9119 (its own published port) connect; :8080 and :8787 are refused instantly even though both listeners are bound to 0.0.0.0 in the host netns. Probes to host.containers.internal (169.254.1.2) from that network time out rather than refuse — the alias is an address on the default podman network, and this container is not on it.

Operational rules

  • Health checks for a bridge-network service must target a published port (or run via podman exec inside the unit), never localhost:<port> from the host — and never the reverse either: the deployer's health probe runs on the host, so it sees host-netns listeners that the container itself cannot dial.
  • To give a bridge-net container access to a host-netns listener, the options are (a) move it to Network=host, or (b) publish the needed port. Option (b) makes that port reachable from every netns and from the LAN — an exposure change that needs operator approval before deployment, not a silent fix.
  • chrome-devtools-protocol is intentionally Network=host with a loopback-only bind so the browser tool stays unreachable from the LAN; do not "tidy" it onto a bridge network — its two consumers (gateway, webui) are host-netns and reach it on loopback.

Exposure note: signal-cli-daemon is a host process, not a container

signal-cli-daemon.service runs as a user unit for dank (systemctl --user cat signal-cli-daemon.service) with org.asamk.signal.Main --account <redacted> daemon --http 0.0.0.0:8080. The bind is all-interfaces, so the daemon's HTTP API is reachable from anywhere on the LAN; verified from a second LAN machine with an unauthenticated GET /v1/version → HTTP 404 and body <h1>404 Not Found</h1>No context found for request (Micronaut's no-route response), i.e. the socket answers to strangers. The Hermes gateway in the host netns dials it on loopback, which is the only reason a non-loopback bind is not required by any consumer here — see Hermes for the consumer side.

See Also

  • Hermes — the gateway + Signal daemon this page's address map is keyed on
  • Hermes-Dashboard — the one Hermes container that is not in the host netns
  • Hermes-WebUI — host-netns service with a 0.0.0.0 bind and password-only auth
  • Network Architecture — the Caddy/Anubis/Authentik front door this map feeds