Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

12 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Quadlet Lifecycle

Fleet rule (AGENTS.md "Boot Persistence & Self-Healing"): every deployed, working container MUST have a self-healing, boot-persistent Quadlet unit — .container file in the repo (<host>/<svc>/quadlet/), installed by the deployer, [Install] WantedBy=default.target, Restart=always, linger on. A container that runs only from compose dies silently at reboot. This page is the operating manual for that lifecycle.

1. Ownership audit — find the containers that will not survive a reboot

for c in $(podman ps --format '{{.Names}}'); do
  printf '%-26s %s\n' "$c" "$(podman inspect $c --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}')"
done

label present → the unit owns it; persistence = [Install] + loginctl show-user <user> -p Linger = yes.
label absent, compose service exists → the classic gap: dies at reboot. Check the publish address while there — compose forms often bind the wildcard (an unauthenticated UI/API on every interface).
label present AND a compose service with the same name → two owners; a stray podman compose up can create a duplicate that fights the unit for the name and the port. Remove the compose service, leave an explanatory comment (repo convention).

2. Cutover: compose service → Quadlet unit

Checklist (precedents to copy: dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/, dankclaw/observability/quadlet/):

Unit file at <host>/<svc>/quadlet/<name>.container, mirroring compose line for line: Image= (localhost/ prefix for locally built images + Pull=never), ContainerName= (Caddy and scrape configs address containers by name), Network=, User=0:0 when the image's own user cannot write a rootless-owned bind mount, Volume= (:ro preserved), Exec= for compose command:, PublishPort=, health keys, [Service] Restart=always + RestartSec=10, [Install] WantedBy=default.target. Never systemctl --user enable the generated unit — [Install] + linger IS the persistence (is-enabled = generated is healthy; explicit enable fails with "transient or generated").
Exposure: publish on 127.0.0.1 unless something off-host genuinely needs it (Caddy reaches containers by name over the shared network; the wildcard publish is a guard-chain bypass).
Handler in scripts/deploy-remote.sh: install_quadlet <dir> <name> → systemctl --user restart <name>.service → wait_for_health; retire any legacy hand-written unit into ~/.AGENTTRASH/ (a leftover ~/.config/systemd/user/<svc>.service that shells out to podman compose up collides BY NAME with the generated unit); podman rm -f <container> so the unit's container can take the name.
Planner: map <host>/<svc>/** → service in scripts/deploy-plan.py AND add a service_is_allowed entry; keep SERVICE_ORDER dependencies ordered. Add a test-changed case above the generic */quadlet/*.container fallback.
Retire the compose entry in the SAME change — two owners is how a stale duplicate comes back.
Seam test tests/<svc>_quadlet.bats: boot persistence, restart policy, no wildcard publish, container name matches Caddy/scrape references, health command, deployer installs+restarts+waits, never rm, never enable. Migrate any older assertion that read the compose file for devices/flags.
Deploy --service <name>; prove: is-active/is-enabled = active/generated, container healthy, endpoint answers, list-dependencies default.target shows the unit, off-host probes on previously published ports now refuse, and the container no longer carries io.podman.compose.project.

Traps that cost real time:

HealthCmd = CMD-SHELL (podman runs /bin/sh -c); a distroless image has no shell → the check fails forever. --health-cmd won't take exec form either. Put HEALTHCHECK … CMD ["binary","arg"] in the image and build with --format docker — the default OCI format has no HEALTHCHECK field and drops it with only a warning.
Never point a unit's EnvironmentFile at the repo-root .env — it is a gluetun env carrying HOME=/root; systemd sets HOME for the service and the unit dies with stat /root/.config: permission denied. Use a per-service env under /home/dank/Containers/<svc>/env, seeded by the handler with a single-variable filter, validated so a missing secret fails loudly.
Network=host exporters scraped by containerized Prometheus must bind the host's LAN address (containers cannot reach host loopback) PLUS 127.0.0.1 for the homepage probe — two --web.listen-address flags, never the wildcard.
alloy 1.x refuses the ui config block; the unit must pass --server.http.listen-addr=0.0.0.0:12345 before its port can be published.
GPU services: AddDevice= + GroupAdd=keep-groups (user-unit spawns carry no supplementary groups).
When the last service leaves a compose file, the compose-validity seam fails (requires ≥1 service): keep the file as comments and relax the seam deliberately, or delete the file and fix the planner's compose-diff path — decide with the owner, the deploy tooling reads it.

3. Drift / crash-loop diagnosis (5 minutes, in order)

Symptom: crash-looping or perpetually unhealthy container whose DEFINITION looks right; or a service quietly down for days because a restart policy kept something running. Cause: the running container was created from an OLDER definition — a compose/unit edit does not touch an existing container, and the restart policy faithfully resurrects it forever.

Who started it: podman inspect <name> --format 'unit={{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}} restart={{.HostConfig.RestartPolicy.Name}} created={{.Created}}'; compare created with when the definition changed (git log -p --follow <config>). Created-before-the-edit is the signature.
What it is ACTUALLY running: {{.Config.CreateCommand}} is the truth. {{.HostConfig.Devices}} / {{.HostConfig.GroupAdd}} print empty/0 even for containers started with --device/--group-add — a false negative that sends you hunting the wrong root cause.
What the process says: podman logs <name> | grep -iE 'error|cannot|Runtime' (a capability error like No HIP GPUs are available beside a missing device flag confirms drift). Count the damage: {{.RestartCount}} — one instance reached 9,146 restart attempts on stale config.

Quickest unblock: podman compose up -d --force-recreate <svc>. Durable fix = the §2 cutover. Verify the capability, not the status: podman exec <name> python3 -c "import torch; print(torch.cuda.is_available())"; ss -ltn | grep <port> for the actual bind addresses. ComfyUI incident: 2026-09-16, stale device-less container, 9,146 restarts. History repeats: 2026-08-26 Caddy (no unit), 2026-09-11 five observability services.

4. Podman 5.x quadlet v1 generator quirks (verified live 2026-08-28)

Hard-fail keys — the whole unit silently fails to generate (systemctl --user cat <unit> shows "could not be found"): any dash-prefixed [Container] option (--shm-size=… → unsupported key … in group 'Container'); Command= is unsupported, use Exec=.
HealthCmd space-splitting: the v1 generator splits HealthCmd on spaces into the healthcheck argv; a quoted probe /bin/sh -c "pg_isready …" lands as argv token "pg_isready → command-not-found → container reads unhealthy while the DB is up. HealthCmd must contain only space-free tokens (HealthCmd=pg_isready -d authentik -U authentik). If health reads unhealthy but logs look fine, compare the baked {{.Config.Healthcheck.Test}} array against the intended command before believing the status.
Probe a new key before shipping: drop a throwaway probe.container, daemon-reload, journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key", systemctl --user cat probe.service (shows the generated ExecStart verbatim), then remove + reload.

5. Rootless Quadlet log retrieval

Step 0 — establish ownership: systemctl --user status <name>.service (works ⇒ rootless user unit); loginctl show-user <user> -p Linger. If --user knows the unit, EVERY command runs as that user with NO sudo.

Need	Command
Current instance tail/follow	`podman logs --tail 200 / -f <name>`
Across restarts/recreates	`journalctl --user -u <name>.service -n 200 --no-pager`
Time-bounded / crash-only	`--since "14:30"` / `-p warning`
Kernel-side evidence (OOM, SVM)	`sudo journalctl -k --since …`

Quadlet units commonly run --replace --rm: podman logs is destroyed on recreation — the journal is the only durable record. Traps: sudo podman logs → "no such container" (root's store does not contain rootless containers; same for sudo podman ps); a system-level journalctl -u <name> returns nothing for a user unit; during first-start weight downloads health reports unhealthy BY DESIGN (check the payload file is growing). Verbose vendor switches are a config change — through git and deploy.sh, never a live env poke.

6. Compose bring-up with ground truth (rootless hosts)

Audit reality before pressing buttons: podman ps -a runs per-user — check every namespace (sudo podman ps -a) and native units (systemctl list-units) before declaring a service absent. Compare compose mount sources against the real filesystem — compose files name phantom paths (e.g. /mnt/media/* vs the real LUKS array at /mnt/entropy). Fix compose through git first, deploy second.
Pre-create bind-mount sources (sudo mkdir -p + chown 1000:1000) or creation fails statfs …: no such file or directory.
Staged bring-up: infra first (up -d mariadb redis), poll {{.State.Health.Status}}, then the rest. Fresh containers answer 000/refused for the first seconds — poll.
Never guess image tags: ghcr.io answers 403 on the bearer-token request for nonexistent/private repos ("manifest unknown"/token len 0 = repo or tag does not exist, not auth). Query the registry API (ghcr.io/token?scope=repository:<ns>/<repo>:pull → /v2/<ns>/<repo>/tags/list).
Read the interface from inside the image before guessing flags/envs: podman exec <ctr> cat /z/cpp.sh etc. HTTP 500 with a styled error page means RUNNING-but-misconfigured, not down. Copyparty ac ground truth: entrypoint /bin/ash /z/cpp.sh -c /z/initcfg; accounts/volumes are ARGV flags (-a USER:PASS, -v SRC:DST:PERMS), not envs — and with none configured it serves cwd read/write for everyone (an unauthenticated world-writable share).
Remote-shell quoting: zsh eats bare ===/? tokens; a zero-match from a mangled pattern is not absence — re-verify the target exists.
user: "1000:1000" SIGSEGV at boot = unmapped uid in the rootless userns (host 1000 == container 0): drop the user line, run as container-root.

Deployment Flow — the deployer that installs these units
Stale Signal Audit — the audit that catches drift's victims
Container Network Namespaces — reachability between containers
Dankclaw/ComfyUI — the 9,146-restart incident
Infrastructure/Observability — same class, twice

12 KiB Raw Blame History