---
name: audit-never-run-container-deploy
description: "Audit a locally built podman/quadlet deploy path that has never actually run: prove base-image assumptions, env-file reader divergence, and image-hash coverage before the first deploy."
---

# Audit a never-run container deploy path

A deployable that has never been built or started has never been *executed*: every
fault in it is invisible to tests that stub the runtime. Audit it as static code
against the tools it calls, then prove each suspicion cheaply against the real
tool. The faults worth hunting are the ones that **report success while doing
nothing** — those are the ones that survive a green `/health` and a green e2e run.

## The three fault classes (all measured 2026-10-18, frank-tts-engine)

1. **A command the base image does not ship.** e.g. `adduser` in a `RUN` line
   against a trimmed `ubuntu:24.04` — absent, and not pulled in by
   `ca-certificates curl libgomp1`. Dies at build time (loud), but only the first
   time anyone builds it.
   Prove it in the base, never from memory:
   `podman run --rm <base> sh -c "command -v adduser || echo absent; useradd --system --uid 10000 --home-dir /app --shell /usr/sbin/nologin tts && id tts"`
   Note the accepted cosmetic warning (`uid 10000 is greater than SYS_UID_MAX 999`)
   in a comment, or a future agent "fixes" it by dropping `--system` and the uid
   stops matching the sibling container.
2. **Two readers of one env file disagree.** systemd units that pass
   `EnvironmentFile=` hand it to podman `--env-file`, which does **no shell
   parsing** — it takes the literal remainder of the line — while bash `source`
   strips one quoting layer. So `WARM_TEXT="Warm up, one two three."` reaches the
   container *with its quotes*, and a script that interpolates it into a JSON body
   produces invalid JSON: the container boots, `active`, every voice registered,
   every voice cold. Fix in the consumer: strip exactly one matched pair **only**
   when both ends are the same quote AND the inner content carries another quote or
   backslash (never mangle a value that legitimately contains quotes), then escape.
3. **A build-skip hash that under-covers the image.** `build_compose_image`-style
   helpers skip `podman build` when a label hash matches; hashing the `Containerfile`
   alone means editing a `COPY`'d app/script rebuilds **nothing** — deploy reports
   success, `/health` answers, container serves old code. Each case arm must name
   the paths its `Containerfile` copies (`extra=(…)`) and hash Containerfile+extras.
   Arms that have no extras keep their old hash, so they do not needlessly rebuild.

## Test it as behaviour, never as a grep of the handler

- Drive the **real** function: `awk '/^build_compose_image\(\) \{/,/^\}/' deploy-remote.sh > bci.sh`,
  `source` it, put a stub `podman` first on `PATH`, assert the observable question —
  *did the deploy ask podman to build, or did it skip?*
- Stub traps that cost real time: podman is invoked `podman image inspect <image> --format …`
  so the tag is **`$3`**, not `$2`; tags contain `/` so a state file named after one
  needs `${tag//\//_}` or the write silently lands in a nonexistent directory and
  every image looks permanently unbuilt; `--label cfhash=…` arrives **before** `-t <tag>`,
  so collect the label and write it after the arg loop.
- For an env-file bug, write one test that reads the file **both** ways — `source`
  and the `--env-file` way (parse `KEY=literal remainder`) — and assert the program
  receives the same value in both shapes, including a value that carries its own
  quotes and a backslash. A suite that only sources the file is how the bug shipped.
- Prove red: `git show HEAD:scripts/deploy-remote.sh > scripts/deploy-remote.sh`,
  run the new test (must fail at the exact intended step), restore. Keep each fix in
  its own commit and run the seam at each one so the branch bisects.

## Base-image and host facts to check while auditing

- `useradd` lives in `login(1)` (base carries it); `adduser` does not.
- Rootless podman **cannot raise rlimits** — `--ulimit nofile=…` above the host hard
  limit dies at container create, not at runtime.
- A referenced `Network=` `.network` file the generator cannot find means the
  container unit is **not produced at all**.
- Two containers talk by DNS name only if they attach to a **common** network;
  check both unit files rather than trusting a comment that says they share one.
- Container DNS name + the network that owns it, never `host.containers.internal`
  or a published loopback port, for service-to-service traffic.

## Do not

- Do not deploy the audited thing because the audit is finished: first build of a
  heavy image (recursive source clone ~10 min) plus a new unit on a host serving
  people is the operator's call.
- Do not add a Prometheus target for a container with no `/metrics` — say the probe
  is a decision (blackbox HTTP), do not fake a scrape.