78 lines
4.9 KiB
Markdown
78 lines
4.9 KiB
Markdown
---
|
|
name: audit-never-run-container-deploy
|
|
description: "Audit a locally built podman/quadlet deploy path that has never actually run: prove base-image assumptions, env-file reader divergence, and image-hash coverage before the first deploy."
|
|
---
|
|
|
|
# Audit a never-run container deploy path
|
|
|
|
A deployable that has never been built or started has never been *executed*: every
|
|
fault in it is invisible to tests that stub the runtime. Audit it as static code
|
|
against the tools it calls, then prove each suspicion cheaply against the real
|
|
tool. The faults worth hunting are the ones that **report success while doing
|
|
nothing** — those are the ones that survive a green `/health` and a green e2e run.
|
|
|
|
## The three fault classes (all measured 2026-10-18, frank-tts-engine)
|
|
|
|
1. **A command the base image does not ship.** e.g. `adduser` in a `RUN` line
|
|
against a trimmed `ubuntu:24.04` — absent, and not pulled in by
|
|
`ca-certificates curl libgomp1`. Dies at build time (loud), but only the first
|
|
time anyone builds it.
|
|
Prove it in the base, never from memory:
|
|
`podman run --rm <base> sh -c "command -v adduser || echo absent; useradd --system --uid 10000 --home-dir /app --shell /usr/sbin/nologin tts && id tts"`
|
|
Note the accepted cosmetic warning (`uid 10000 is greater than SYS_UID_MAX 999`)
|
|
in a comment, or a future agent "fixes" it by dropping `--system` and the uid
|
|
stops matching the sibling container.
|
|
2. **Two readers of one env file disagree.** systemd units that pass
|
|
`EnvironmentFile=` hand it to podman `--env-file`, which does **no shell
|
|
parsing** — it takes the literal remainder of the line — while bash `source`
|
|
strips one quoting layer. So `WARM_TEXT="Warm up, one two three."` reaches the
|
|
container *with its quotes*, and a script that interpolates it into a JSON body
|
|
produces invalid JSON: the container boots, `active`, every voice registered,
|
|
every voice cold. Fix in the consumer: strip exactly one matched pair **only**
|
|
when both ends are the same quote AND the inner content carries another quote or
|
|
backslash (never mangle a value that legitimately contains quotes), then escape.
|
|
3. **A build-skip hash that under-covers the image.** `build_compose_image`-style
|
|
helpers skip `podman build` when a label hash matches; hashing the `Containerfile`
|
|
alone means editing a `COPY`'d app/script rebuilds **nothing** — deploy reports
|
|
success, `/health` answers, container serves old code. Each case arm must name
|
|
the paths its `Containerfile` copies (`extra=(…)`) and hash Containerfile+extras.
|
|
Arms that have no extras keep their old hash, so they do not needlessly rebuild.
|
|
|
|
## Test it as behaviour, never as a grep of the handler
|
|
|
|
- Drive the **real** function: `awk '/^build_compose_image\(\) \{/,/^\}/' deploy-remote.sh > bci.sh`,
|
|
`source` it, put a stub `podman` first on `PATH`, assert the observable question —
|
|
*did the deploy ask podman to build, or did it skip?*
|
|
- Stub traps that cost real time: podman is invoked `podman image inspect <image> --format …`
|
|
so the tag is **`$3`**, not `$2`; tags contain `/` so a state file named after one
|
|
needs `${tag//\//_}` or the write silently lands in a nonexistent directory and
|
|
every image looks permanently unbuilt; `--label cfhash=…` arrives **before** `-t <tag>`,
|
|
so collect the label and write it after the arg loop.
|
|
- For an env-file bug, write one test that reads the file **both** ways — `source`
|
|
and the `--env-file` way (parse `KEY=literal remainder`) — and assert the program
|
|
receives the same value in both shapes, including a value that carries its own
|
|
quotes and a backslash. A suite that only sources the file is how the bug shipped.
|
|
- Prove red: `git show HEAD:scripts/deploy-remote.sh > scripts/deploy-remote.sh`,
|
|
run the new test (must fail at the exact intended step), restore. Keep each fix in
|
|
its own commit and run the seam at each one so the branch bisects.
|
|
|
|
## Base-image and host facts to check while auditing
|
|
|
|
- `useradd` lives in `login(1)` (base carries it); `adduser` does not.
|
|
- Rootless podman **cannot raise rlimits** — `--ulimit nofile=…` above the host hard
|
|
limit dies at container create, not at runtime.
|
|
- A referenced `Network=` `.network` file the generator cannot find means the
|
|
container unit is **not produced at all**.
|
|
- Two containers talk by DNS name only if they attach to a **common** network;
|
|
check both unit files rather than trusting a comment that says they share one.
|
|
- Container DNS name + the network that owns it, never `host.containers.internal`
|
|
or a published loopback port, for service-to-service traffic.
|
|
|
|
## Do not
|
|
|
|
- Do not deploy the audited thing because the audit is finished: first build of a
|
|
heavy image (recursive source clone ~10 min) plus a new unit on a host serving
|
|
people is the operator's call.
|
|
- Do not add a Prometheus target for a container with no `/metrics` — say the probe
|
|
is a decision (blackbox HTTP), do not fake a scrape.
|