Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
70 lines
3.1 KiB
Markdown
70 lines
3.1 KiB
Markdown
---
|
|
name: locale-c-multibyte-awk-silent-loss
|
|
description: "Debug generated output that silently loses data under LC_ALL=C — a regex quantifier on a multibyte character in awk/grep, plus the locale-dependent red test it produces."
|
|
---
|
|
|
|
# LC_ALL=C kills multibyte regex quantifiers — silently
|
|
|
|
Symptom: a generator emits a **complete, exit-0, missing-a-third-of-the-page** artifact in
|
|
cron / systemd user units / agent shells, and the same command looks perfect in the user's
|
|
terminal. Companion symptom: a repo test is red for the agent and green for the operator.
|
|
|
|
## Probe (do these in order, they are 5 seconds apart)
|
|
|
|
```bash
|
|
echo "LANG=[$LANG] LC_ALL=[$LC_ALL]" # LC_ALL overrides LANG — en_US.UTF-8 + LC_ALL=C is byte-wise
|
|
locale -a | grep -iE 'utf-?8' | head # which UTF-8 locale actually exists on this box
|
|
```
|
|
|
|
Then test the suspect pattern alone, under both locales, with the literal character:
|
|
|
|
```bash
|
|
printf '%s\n' ' // ─── Applications ───' | awk '/─+ / { print "QUANT MATCHED" }'
|
|
printf '%s\n' ' // ─── Applications ───' | awk '/─/ { print "LITERAL MATCHED" }'
|
|
printf '%s\n' ' // ─── Applications ───' | LC_ALL=C.utf8 awk '/─+ / { print "QUANT MATCHED under UTF-8" }'
|
|
```
|
|
|
|
Literal matches everywhere; the quantifier matches only under a UTF-8 locale. That asymmetry IS
|
|
the bug: gawk under `LC_ALL=C` matches bytes, so `─+` means "the last byte of U+2500, repeated" —
|
|
a byte sequence that never occurs. mawk/busybox/Go regexp have the same class of failure
|
|
(quantifier or bracket class spanning a multibyte char).
|
|
|
|
## Fix
|
|
|
|
Match and strip the multibyte literal with no quantifier:
|
|
|
|
```awk
|
|
/^[[:space:]]*\/\/[[:space:]]*─/ { # literal gate
|
|
s = $0; sub(/^[[:space:]]*\/\/[[:space:]]*/, "", s); sub(/[[:space:]]+$/, "", s)
|
|
if (s ~ /─ .*─/) { gsub(/─/, "", s); gsub(/[[:space:]]+/, " ", s); sub(/^ /, "", s); ... }
|
|
next
|
|
}
|
|
```
|
|
|
|
**Never pin `LC_ALL=en_US.UTF-8` to "fix" it** — that locale often does not exist (a CachyOS box
|
|
here shipped only `C.utf8`), so the script then breaks on another machine. Make the matcher
|
|
byte-safe instead.
|
|
|
|
## Pin it with a test
|
|
|
|
`run env LC_ALL=C "$GEN" "$IN"` + assert the lost content is present. The operator's UTF-8 terminal
|
|
can never catch this class of bug, so the explicit C-locale run is the only durable guard.
|
|
|
|
## Suite-verification law (how this got missed)
|
|
|
|
NEVER judge `bats tests/` from `tail` output — a failure at test 28 is invisible in a 25-line
|
|
window ending at test 109. Always:
|
|
|
|
```bash
|
|
out=$(bats tests/ 2>&1); echo "$out" | grep -cE '^ok'; echo "$out" | grep -E '^not ok' || echo GREEN
|
|
```
|
|
|
|
Run it under `LC_ALL=C` too: that is the environment every automated caller inherits. Same rule
|
|
for `| grep -c` on a multi-line JSON field — count what you claim to have counted.
|
|
|
|
## Related, do not confuse
|
|
|
|
- `bats-assertion-trust` — vacuously-passing assertions (green that proves nothing). This skill is
|
|
the mirror case: correct assertions that are only red in one locale.
|
|
- `gitea-pr-api-cycle` — unrelated; landed here only because the PR work is where this surfaced.
|