Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

5.8 KiB

Raw Permalink Blame History

type, title, description, tags, timestamp

type

title

description

Container Host Reachability

Two failure families on rootless podman + netavark hosts (Dankclaw), both reading as "service down" when the service is up and the path is wrong. Fleet source of truth: serverAppConfigs-docs/Architecture/Container Network Namespaces.md (commit 2e179b0).

Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly

A container cannot reach a Network=host service at the host's LAN IP. netavark DNATs published ports only: from a bridge container, 192.168.0.143:<published> answers while 192.168.0.143:<host bind> and the gateway 10.89.1.1 are both refused. This is what 502'd hermes.franksdank.cloud until the Caddyfile upstream became host.containers.internal on 2026-09-16 — its vhost comment had claimed the LAN IP was open from that bridge; it never was.

Probe that proves it in one command:

ssh <host> "podman run --rm --network <net> --entrypoint sh \
  docker.io/library/busybox:latest -c \
  'for p in <target-port> <published-port>; do
     nc -w2 -z -v <host-lan-ip> \$p >/dev/null 2>&1 && echo port-\$p-OK || echo port-\$p-BAD
   done'"
podman exec <svc> bash -c 'exec 3<>/dev/tcp/host.containers.internal/<port> && echo ALIAS-OK'

Published OK + target BAD ⇒ quirk confirmed. Note busybox nc -z reports any failure as failure — use it for bulk sweeps; use /dev/tcp (or nc -v) when refusal-vs-timeout matters.

Fix: point the service's host-facing URL at http://host.containers.internal:<port> in compose/quadlet env, redeploy via git (deploy.sh <target> --service <svc>), verify podman exec <svc> printenv <URL_VAR> and the connect log line (~40 s). Do not publish the daemon port to make the LAN IP work — publishing is exactly what makes other ports refuse.

Quirk 2 — addressability is per netns, not per host

NetworkMode=host → the container is the host netns: dials all of 127.0.0.1, binds any free port.
bridge/named net → own netns; only published ports are reachable, via the network's gateway IP. Unpublished host listeners refuse instantly (RST) — no firewall rule, no stale DNAT, nft/iptables -t nat empty. Do not hunt for REJECT rules; probe.

Probe sequence (read-only):

podman inspect <name> --format '{{.HostConfig.NetworkMode}}'
sudo ss -ltnp 'sport = :<port>'        # host netns only — why 0.0.0.0 can still be unreachable
podman exec <name> cat /proc/net/route  # default route = the gateway IP to dial
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/<gw-ip>/<port>"; echo rc=$?'

Interpretation without blurring:

instant refusal = nothing listening in a reachable namespace (or rootlessport rejected an unpublished port map).
timeout = no route to that address at all. host.containers.internal (169.254.1.2) is an alias on the default podman net only; containers on a named network time out against it.

Tool gotchas (podman 5.x, verified live):

podman network inspect <net> --format '{{json .containers}}' errors (can't evaluate field containers in type interface {}) — list membership from the container side: podman ps -a --format '{{.Names}} {{.Networks}}'.
An empty Networks field = host network or shared namespace (--network=container:<other>), not "no network".
Quadlet Network=<name>.network yields the runtime name systemd-<name> — probe with podman network ls, never the .network filename.

Fix options when a bridge-net container must reach a host-netns listener: (a) move it to Network=host, or (b) publish the port — (b) makes the port reachable from every namespace and the LAN, an exposure change needing explicit operator approval, never a silent fix.

On hosts publishing ports through pasta, the port is bound in userspace at container CREATE, not when the app binds. A predecessor stopped moments earlier can still hold listener/TIME_WAIT state, so the replacement dies pasta: ... bind: Address already in use (conmon exit 126) — then self-heals seconds later under Restart=on-failure. The failure looks like it failed, then looks fine, and the missing gate stays invisible because the retry masks it. First seen on a frank-dragoon unit cutover; which supervision shape that era was is in frank-dragoon's era table.

Gate procedure: stop the old unit AND the new one (a redeploy's running replacement is itself the port-holder — without that stop the gate deadlocks against itself), then wait for the port to be empty across all socket states:

wait_port_free() {
    local port="$1" limit="${2:-90}" i
    for ((i = 1; i <= limit; i++)); do
        if [ -z "$(ss -Htan "sport = :$port" 2>/dev/null)" ]; then return 0; fi
        sleep 1
    done
    echo "FAIL: port $port still held after ${limit}s:" >&2
    ss -tan "sport = :$port" >&2 || true
    return 1
}

sport catches accepted-connection server sockets and TIME_WAIT residue that ss -ltn misses; fail loudly with the socket table attached. Verify the gate by re-running the same deploy against a running service (idempotency) and assert it in seam tests (wait_port_free present in the deploy handler).

Quadlet Lifecycle — the unit layer these containers run under
Container Network Namespaces — fleet network namespace source of truth
Observability — the alerting paths that depend on reachability
Deploy State Triage — the deploy flow that installs these configs

5.8 KiB Raw Permalink Blame History

Container Host Reachability

Quirk 1 — LAN-IP dialing of an unpublished host port refuses instantly

Quirk 2 — addressability is per netns, not per host

Related trap — the pasta port gate (rootless published ports)

Related

5.8 KiB

Raw Permalink Blame History