Frank
71601f72a8
The live-sync heartbeat failed three hourly ticks (22:25, 23:27, 00:28) because a managed skill that documents how to write a secret gate quotes the vendor's all-zeros placeholder access-key id, and the shape rule fired on the documentation. The machine tier is prose ABOUT credentials — the gate's own header says a gate that fires there gets overridden until it means nothing — and a heartbeat that cannot commit is a fleet that silently stops syncing config. - shape hits are judged on the matched token, not the line: a documented fake is inert, one live token sharing the file still blocks. The match is judged with its following context, because a fixed quantifier truncates `EXAMPLE` to `EXA` and the placeholder would vanish from what is examined. - assignment values keep the stricter rule from the previous commit: a comment on the same line cannot vouch for the token beside it. - `ALLOW_SECRETS` remains the only override; nothing here widens what is scanned, it only stops flagging text that was never a credential. Verified against the payload that is stuck right now: the skill page that the heartbeat cannot commit passes the fixed hook and was refused by the live one. Refs #60
147 lines
8.0 KiB
Bash
Executable File
147 lines
8.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Secret gate: block any commit whose staged diff contains live-secret shapes or
|
|
# credential assignments. This repo IS the deployment and every workstation
|
|
# pulls it, so a committed secret is instantly in every agent's context window
|
|
# on the fleet. Deliberate override: ALLOW_SECRETS=1 git commit ...
|
|
#
|
|
# Two rules, deliberately different in scope:
|
|
#
|
|
# 1. Token SHAPES are scanned across the ENTIRE staged diff, whatever the file
|
|
# type. The first version of this gate whitelisted extensions
|
|
# (*.md *.yml *.yaml *.json *.toml *.sh *.ts *.js), so a `postgres.env`, a
|
|
# `.container` quadlet, an `.ini`, or an extension-less dotfile — exactly
|
|
# the shapes the dotfiles tier is made of — was never read at all.
|
|
# 2. Credential ASSIGNMENTS (PASSWORD=…, secret_key: …) are scanned only in
|
|
# non-prose files. The auto-learned memory tier is prose ABOUT credentials,
|
|
# and a gate that fires there gets overridden until it means nothing; the
|
|
# 2026-10-20 audit measured 7 such prose hits on the tracked tree. Markdown
|
|
# keeps shape detection only, which is what it had before.
|
|
#
|
|
# Bcrypt hashes and Basic-auth headers joined the shape list on 2026-10-20
|
|
# (issue #40) after a live Caddy basicauth hash plus its plaintext password —
|
|
# and the Pi-hole internal password — were found in a tracked memory file.
|
|
set -uo pipefail
|
|
|
|
shapes='sk-[A-Za-z0-9]{18,}|ghp_[A-Za-z0-9]{36}|github_pat_[A-Za-z0-9_]{22,}|glpat-[A-Za-z0-9_-]{20}|xox[baprs]-[A-Za-z0-9-]{10,}|AKIA[A-Z0-9]{16}|ASIA[A-Z0-9]{16}|BEGIN [A-Z ]*PRIVATE KEY|aes_[A-Za-z0-9/+]{40}|eyJ[A-Za-z0-9_-]{20,}\.ey|\$2[aby]\$[0-9]{2}\$[./A-Za-z0-9]{20,}|Authorization:[[:space:]]*[Bb]asic[[:space:]]+[A-Za-z0-9+/=]{24,}'
|
|
|
|
# `newpass1234`, `secret_key: "rotated-value"` — an assignment with a literal
|
|
# value. `$VAR`, `${VAULT}`, <placeholders> and empty values cannot match.
|
|
assign='[A-Za-z0-9_]*(PASSW|SECRET|TOKEN|API_?KEY|ACCESS_KEY|PRIVATE_KEY|CREDENTIAL)[A-Za-z0-9_]*[[:space:]]*[:=][[:space:]]*["'"'"']?[A-Za-z0-9/+._@-]{8,}'
|
|
placeholders='changeme|change[-_]|placeholder|redacted|example|sample|dummy|nokey|no[-_]key|not[-_]needed|your[-_]|localhost|xxxx|yyyy|\$\{|<[a-z]|\*\*\*|%s|null|false|true'
|
|
|
|
# Exemptions: a variable naming a location (`…_FILE=`, `…_PATH=`) or a value
|
|
# that is a filesystem path is a pointer, not a credential. Flagging those is
|
|
# how a gate gets permanently overridden.
|
|
assign_exempt='(_FILE|_PATH|_DIR|_URL|_ENDPOINT|_SOCKET)[[:space:]]*[:=]|[:=][[:space:]]*["'"'"']?(/|\.\./)'
|
|
# Fail closed: a gate that cannot read the staged diff must not report success.
|
|
#
|
|
# `-z`, and the list lives in a file. Two traps decide this: git quotes any path it cannot print
|
|
# as UTF-8 (core.quotePath is on by default), and a quoted name handed back to
|
|
# `git diff -- <path>` matches nothing — the file is skipped, and a skipped file used to be a
|
|
# silent pass. Command substitution cannot hold the list either, because $() deletes NUL bytes,
|
|
# so the names go to a temp file and the loop reads them with a NUL delimiter.
|
|
TMPD=$(mktemp -d) || TMPD=""
|
|
if [ -z "$TMPD" ]; then
|
|
echo "SECRET GATE: cannot create a temp dir — refusing to commit." >&2
|
|
[ -n "${ALLOW_SECRETS:-}" ] && exit 0
|
|
exit 1
|
|
fi
|
|
trap 'rm -rf "$TMPD"' EXIT
|
|
if ! git diff --cached --diff-filter=ACM --name-only -z > "$TMPD/files" 2> "$TMPD/err"; then
|
|
{
|
|
echo "SECRET GATE: cannot read the staged diff — refusing to commit."
|
|
sed 's/^/ /' "$TMPD/err"
|
|
echo "Override deliberately: ALLOW_SECRETS=1 git commit ..."
|
|
} >&2
|
|
[ -n "${ALLOW_SECRETS:-}" ] && exit 0
|
|
exit 1
|
|
fi
|
|
# git's stderr stays out of the name list on purpose: a `warning:` line mistaken for a path
|
|
# becomes a file that "cannot be read", which is how a gate starts skipping the real ones.
|
|
[ -s "$TMPD/files" ] || exit 0
|
|
|
|
report=""
|
|
exempt=""
|
|
count=0
|
|
while IFS= read -r -d '' f; do
|
|
[ -z "$f" ] && continue
|
|
# Self-exemption, deliberately an exact filename list and not `tests/`: the
|
|
# gate's fixtures must contain credential-shaped strings or there is nothing to
|
|
# test against, and this hook documents the shapes it looks for. A secret parked
|
|
# in any other test file is still caught.
|
|
case "$f" in
|
|
.githooks/pre-commit|tests/secret-gate.bats) exempt="$exempt $f"; continue ;;
|
|
esac
|
|
diff=$(git diff --cached -U0 -- "$f")
|
|
[ -z "$diff" ] && continue
|
|
# Only ADDED lines can leak. Matching removals too blocks the very commit that
|
|
# deletes a credential — which is exactly what happened on 2026-10-20, when the
|
|
# gate refused the redaction of the password it exists to catch.
|
|
added=$(printf '%s\n' "$diff" | grep -E '^\+' | grep -vE '^\+\+\+' | sed 's/^+//' || true)
|
|
[ -z "$added" ] && continue
|
|
# Shape hits are judged on the MATCHED TOKEN, not on the whole line. Documentation about
|
|
# credentials quotes vendor placeholders — AWS publishes an all-zeros access-key id that ends
|
|
# in EXAMPLE for exactly this purpose — and
|
|
# a gate that blocks prose over an inert example gets worked around until it means nothing:
|
|
# the live-sync heartbeat failed three ticks on 2026-10-21 because a managed skill page quotes
|
|
# exactly that id. A line is exempted only when EVERY shape on it is inert, so one live token
|
|
# sharing the file with a documented fake still blocks.
|
|
hits=$(printf '%s\n' "$added" | while IFS= read -r line; do
|
|
live=0
|
|
while IFS= read -r m; do
|
|
# The shape regexes carry fixed quantifiers, so the match they return is TRUNCATED:
|
|
# `AKIA` + exactly 16 characters stops inside the word EXAMPLE and the placeholder
|
|
# disappears from what is judged. Judge the match plus what follows it instead.
|
|
ctx=$(printf '%s' "$line" | awk -v m="$m" '{ i = index($0, m); print (i > 0 ? substr($0, i, length(m) + 8) : m) }')
|
|
printf '%s' "$ctx" | grep -Eiq 'EXAMPLE|X{4,}|Y{4,}|Z{4,}|_{4,}|REDACTED|CHANGEME|PLACEHOLDER' || live=1
|
|
done < <(printf '%s\n' "$line" | grep -Eio "$shapes" || true)
|
|
[ "$live" = 1 ] && printf '%s\n' "$line"
|
|
done)
|
|
if [ "${f##*.}" != "md" ]; then
|
|
# The placeholder filter reads the VALUE, never the whole line. A trailing comment that
|
|
# says `# example` says nothing about the token the same line assigns; scoping the filter
|
|
# to what follows the separator is what keeps `API_TOKEN=<real> # placeholder` a block.
|
|
assigns=$(printf '%s\n' "$added" | grep -Ei "$assign" | grep -Ev "$assign_exempt" \
|
|
| while IFS= read -r line; do
|
|
# The judged value is the assigned token alone: it ends at whitespace or `#`, so a
|
|
# trailing comment cannot vouch for it. Truncating a quoted value early only ever errs
|
|
# toward blocking, which is the safe direction for a gate that can be overridden.
|
|
value=${line#*[:=]}
|
|
value=${value#"${value%%[![:space:]]*}"}
|
|
value=${value%%[[:space:]]*}
|
|
value=${value%%#*}
|
|
printf '%s\n' "$value" | grep -Eiq "$placeholders" && continue
|
|
printf '%s\n' "$line"
|
|
done)
|
|
hits=$(printf '%s\n%s\n' "$hits" "$assigns" | sed '/^[[:space:]]*$/d')
|
|
fi
|
|
if [ -n "$hits" ]; then
|
|
n=$(printf '%s\n' "$hits" | wc -l)
|
|
count=$((count + n))
|
|
report+=" $f"$'\n'"$(printf '%s\n' "$hits" | cut -c1-100 | sed 's/^/ /')"$'\n'
|
|
fi
|
|
done < "$TMPD/files"
|
|
|
|
# Say so out loud: an exemption that prints nothing is how a carve-out becomes
|
|
# invisible, and invisible is the property a secret wants.
|
|
if [ -n "$exempt" ]; then
|
|
printf 'secret-gate: self-test fixtures exempt:%s\n' "$exempt" >&2
|
|
fi
|
|
|
|
if [ -n "${ALLOW_SECRETS:-}" ]; then
|
|
[ -n "$report" ] && printf 'secret-gate: ALLOW_SECRETS set, allowing %s secret-shaped string(s) through.\n' "$count"
|
|
exit 0
|
|
fi
|
|
|
|
if [ -n "$report" ]; then
|
|
{
|
|
echo "SECRET GATE: staged diff contains $count secret-shaped string(s):"
|
|
printf '%s\n' "$report"
|
|
echo "Fix it, or override deliberately: ALLOW_SECRETS=1 git commit ..."
|
|
echo "A credential that reaches a tracked file is already in history on every"
|
|
echo "machine that pulls this repo — deleting it later does not un-leak it."
|
|
} >&2
|
|
exit 1
|
|
fi
|
|
exit 0
|