Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
58 lines
3.6 KiB
Markdown
58 lines
3.6 KiB
Markdown
---
|
|
name: bats-assertion-trust
|
|
description: "When writing or reviewing bats shell tests, catch vacuously-passing assertions: grep -vq, sed range boundaries, unanchored section slices, and fixtures too small to trigger pipefail/SIGPIPE bugs."
|
|
---
|
|
|
|
Prove a bats assertion can actually fail before trusting that it passes. These four traps each made a green suite lie in real work.
|
|
|
|
## 1. `grep -vq` proves nothing
|
|
|
|
`some_list | grep -vq 'x'` succeeds when ANY line lacks `x` — nearly always true. To assert absence, capture the text and test it:
|
|
|
|
```bash
|
|
s="$(report_section)"
|
|
[[ "$s" != *libfoo* ]] # absence
|
|
[[ "$s" == *bun* ]] # presence
|
|
```
|
|
|
|
Same trap in reverse: bash `[[ $s == *A*B* ]]` — `*` crosses newlines, so a substring check spanning two lines matches the wrong lines. Slice to one line first, then assert.
|
|
|
|
## 2. Section slicing helpers must have explicit boundary semantics
|
|
|
|
A helper used to scope assertions to one report section must define, and comment, both ends. The default you want: **start inclusive** (labels carry the count/names), **terminator exclusive** (a `sed -n "/A/,/B/p"` range includes B and leaks the next section's first line):
|
|
|
|
```bash
|
|
# Call sites pass a bare anchored regex; both ends documented.
|
|
section() { printf '%s\n' "$output" | awk -v s="$1" -v e="$2" '$0 ~ e {on=0} $0 ~ s {on=1} on'; }
|
|
```
|
|
|
|
Two bugs this shape hides:
|
|
- Helper prepends `^` while call sites already pass `^name` → pattern `/^^name/` → never matches → empty slice.
|
|
- An **unanchored start address** matches the first line and the range runs to end-of-output, so every absence assertion becomes a vacuous pass. Symptom: tests that "verify X is not in section A" pass while X is printed later in the same output.
|
|
|
|
After fixing a helper like this, re-run the whole suite: tests that were passing vacuously suddenly fail, and that is the fix working.
|
|
|
|
## 3. Fixtures too small hide pipefail bugs
|
|
|
|
`set -euo pipefail` + `sort | head -1`: `head` exits, closes the pipe, `sort` dies of SIGPIPE, the command substitution returns `141`, `set -e` aborts the script. Small fixtures never overflow the 64 KiB pipe buffer, so the suite stays green and the first run against real data fails with no output.
|
|
|
|
- Take a minimum in one pass: `awk '{ if (m=="" || $1<m) m=$1 } END { print m }' file`
|
|
- `awk 'NR==1{print; exit}'` is **not** a fix — exiting early closes the pipe exactly like `head`.
|
|
- Pin it with a fixture larger than the pipe buffer: `seq 1 20000 | awk '{print "[2026-01-01T10:00:00-0700] [ALPM] installed pkgfiller (1-1)"}' > big.log` (use `seq`, not `yes | head`, or the fixture itself SIGPIPEs).
|
|
|
|
## 4. `tr` swallows the final newline and downstream tools preserve that
|
|
|
|
`tr '\n' ' ' < file` emits no trailing newline; `fold`, `sed`, and `cat` faithfully preserve the absence, so the *next* output line gets glued on. Breaks any tool that later slices output by line-start labels.
|
|
|
|
```bash
|
|
{ tr '\n' ' ' < file; echo; } | fold -s -w 88 | sed 's/^/ /'
|
|
```
|
|
|
|
Pin with: assert the following section's label does NOT appear inside the preceding section's slice.
|
|
|
|
## Debugging order when a bats assertion fails but the script output looks right
|
|
|
|
1. Reproduce the exact command outside bats (`bash -x` against the script under test) — if it works there, the bug is in the *test helper*, not the script.
|
|
2. Dump the constructed argument: `printf '%s' "$sed_script" | od -An -c` inside the helper. That is how `/^^ repo /` (double anchor) got found.
|
|
3. Keep stubs faithful to the real interface: if the stub answers `-Sqq` but the script calls `-Sl`, the test measures nothing. Make the stub exit `64` on unhandled args so a new real query fails loudly instead of silently returning empty.
|