live: heartbeat sync 2026-10-21T02:32:51-07:00 (Refs #11)
This commit is contained in:
Frank committed
1 parent
22d2f3bcd8
commit
336201e814
22 files changed
+451
-322
No files matched your search
@@ -1,60 +1,62 @@
|
||||
---
|
||||
name: harden-precommit-secret-gate
|
||||
description: "Fix or build a pre-commit secret-scan hook that actually blocks leaks: scope hole, fail-closed, quarantine list, and the self-trip that kills an automated committer."
|
||||
description: "Fix or build a pre-commit secret-scan hook that actually blocks: the -z path hole, line-wide placeholder exemptions, inert doc placeholders, glued report lines, self-test fixture paths — each with a red-first bats case."
|
||||
---
|
||||
|
||||
Fix or build a pre-commit secret-scan hook that actually blocks leaks: scope hole, fail-closed, quarantine list, and the self-trip that kills an automated committer. Use when a secret gate is wrong (misses a leak) or too right (rejects an inert example and bricks the heartbeat).
|
||||
A pre-commit secret gate that fails open is worse than no gate: it teaches the operator that scanning happened. Four defect classes, each with a black-box test that goes red against the broken version (serverAppConfigs PR #70, command-center #59 — 4 holes closed same night, all red first).
|
||||
|
||||
## The two failure directions
|
||||
## 1. Non-ASCII paths fail open
|
||||
|
||||
A gate can be **too weak** (misses a real key) or **too strong** (rejects its own documentation and takes down an automated committer). Both are outages; the second is silent.
|
||||
|
||||
## Audit recipe for an existing gate
|
||||
|
||||
1. Read the hook and list what it actually inspects. The classic scope hole: `git diff --cached -- '*.env' '*.pem'` — an extension whitelist. A credential in `agent/config.yml`, a quadlet unit, or a SKILL.md is then invisible to the gate. The rule that survives: scan **the entire staged diff**, every path, and let only binary content out.
|
||||
2. Test fail-closed behaviour: can the hook be made to print nothing and still exit 0? Common escapes — running outside a git repo, unreadable staged diff, a hook that pipes through `grep` and inherits `grep`'s exit code from the wrong stage, or `set -e` swallowing the verdict. Verdict must be: unknown ⇒ **block**.
|
||||
3. Quarantine list, never tracked: `.env`, `auth.json`, `trust.json`, `*.pem`, `*.key`, `identity*` — and the `.gitignore` policy must quarantine env files **at any depth under any name** (`**/.env*`, not `.env`), or `config/prod.env` slips past.
|
||||
4. Escape hatch must be explicit, per-commit, and loud: `ALLOW_SECRETS=1 git commit …`, and the hook says so when it honours it.
|
||||
5. Gate the gate: a test that commits a fixture repo through the hook and asserts it blocks. `tests/secret_gate.bats` here covers: shape in an unexpected extension, bcrypt hash, JWT, placeholder/indirection allowed, prose allowed, binary ignored, unreadable diff refused, deletion of a credential never blocked, `ALLOW_SECRETS` override.
|
||||
|
||||
## Self-trip: the gate rejects its own example (live 2026-10-21)
|
||||
|
||||
Symptom: a systemd timer that commits machine-written files fails **every hour**, and the journal shows the gate's own refusal:
|
||||
|
||||
```
|
||||
SECRET GATE: staged diff contains 1 secret-shaped string(s):
|
||||
agent/managed-skills/git-secret-gate-hook/SKILL.md
|
||||
Fake secrets must be inert (RFC 2606 domains, all-zero ids, `AKIA0000000000000EXAMPLE`) AND must not
|
||||
```
|
||||
|
||||
The offender is the AWS **documentation** example key — entirely inert — and `AKIA[0-9A-Z]{16}` cannot tell it from a leak. So the gate's own advice about writing an inert fake is uncommittable by the gate. Auto-learned skills describe credential shapes for a living, so this recurs by construction, and an automated committer cannot use `ALLOW_SECRETS=1`: that would blind the gate for the *other* 64 staged files.
|
||||
|
||||
Diagnose, in this order:
|
||||
`git diff --cached --name-only` quotes non-ASCII paths (`"docs/\303\251.md"`), so `git diff` on that name returns **empty**, and a loop that `continue`s on an empty diff skips the file silently. Fail closed with NUL delimiting:
|
||||
|
||||
```bash
|
||||
journalctl --user -u live-sync.service -b --no-pager | tail -25 # which file, which line
|
||||
git -C ~/.omp status --short | head # how much is queued behind it
|
||||
# then scan the WHOLE staged queue, not just the reported file:
|
||||
git diff --cached -U0 | awk '/^diff --git/{f=$3;sub("^a/","",f)} /^\+/ && !/^\+\+\+/ {
|
||||
l=substr($0,2)
|
||||
if (l ~ /sk-[A-Za-z0-9]{16,}|ghp_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,}|glpat-[A-Za-z0-9_-]{16,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,}|eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,}\.|-----BEGIN [A-Z ]*PRIVATE KEY-----/)
|
||||
print f": "substr(l,1,90) }'
|
||||
git diff --cached -z --name-only | while IFS= read -r -d '' f; do ...
|
||||
```
|
||||
|
||||
The second count is the decision input: one hit ⇒ fix the line; many ⇒ the gate needs an exemption channel before anything lands.
|
||||
Test: a tracked file whose name contains a non-ASCII character, holding a fake key on an added line, must be blocked.
|
||||
|
||||
Fixes, in order of preference:
|
||||
## 2. Placeholder filters must be value-scoped
|
||||
|
||||
1. **Describe the shape instead of printing a matching literal** — `AKIA` followed by zeroes and `EXAMPLE`, spelled out in prose, cannot match. One line, the gate stays strict, the next timer run drains the queue.
|
||||
2. **Per-line allow marker** honoured by the hook (e.g. a trailing `secret-gate: allow` on the flagged line) — durable, keeps documentation honest, still fail-closed everywhere else. Needs a test both ways: a marked line passes, an unmarked one still blocks. Note the risk honestly: an attacker-adjacent string can self-authorise, so the marker belongs in review-able prose tiers, not in a diff that also touches secrets config.
|
||||
3. Never widen the regex to skip markdown, never run the automation with the override env var set.
|
||||
`if echo "$line" | grep -q "$placeholders"` exempts the whole line, so a comment word vouches for the credential beside it:
|
||||
|
||||
Inert-fake rule for anything that must *show* a credential shape: RFC 2606 domains, all-zero ids, the word `EXAMPLE` — and if the shape itself trips the gate, describe it in prose rather than print it.
|
||||
```python
|
||||
token = "sk-…" # example config for the docs
|
||||
```
|
||||
|
||||
## Prose must not be collaterally blocked
|
||||
Filter only the captured value (the `sk-…` group), never the line. Two directions to test: `key = "example"` stays allowed; `token = "sk-<real>" # example` is blocked.
|
||||
|
||||
Markdown explaining credential handling is the bulk of what a knowledge repo commits. `xox` inside a sentence about Slack token prefixes, or a `sk-` prefix discussed abstractly, must pass; the shapes are anchored long-suffix patterns for exactly that reason. When a gate fires on prose, that is a bug in the gate — but if the prose contains a *literal* matching token, the fix is the prose (option 1 above), not the pattern.
|
||||
## 3. Inert placeholders in documentation
|
||||
|
||||
## Verifying a gate change
|
||||
A skill/README that *documents* a secret gate quotes the vendor's placeholder ids (`AKIA0000000000000EXAMPLE`, all-zero JWTs). Exempt them by value shape (all-zero / `EXAMPLE` suffix / RFC 2606 domain), or the machine tier stops committing — and the refusal usually lands on a heartbeat commit, where it looks like infrastructure failure, not policy. Also note: an unlanded fix means the checkout runs the **old** hook, so a landing can be blocked by the very bug the PR corrects; audit the staged diff and use the documented override (`ALLOW_SECRETS=1`, never `--no-verify`), recording the audit in the commit message.
|
||||
|
||||
Write the fixture repo in a temp dir, commit through the hook, assert block/allow for each case. Do not test the regex by grepping the hook's source for the pattern — a copy of the pattern in a comment is not a gate that runs.
|
||||
## 4. Report-line assembly
|
||||
|
||||
`report+=(" $file: $line")` inside a loop that keeps only the last hit per file glues the next filename onto the previous line when a second hit exists — the report attributes the finding to the wrong file. Test a two-file staged set where both match, and assert each report line names exactly one existing path.
|
||||
|
||||
## 5. Self-test fixtures in the hook itself
|
||||
|
||||
The hook's own fixtures contain fake keys by necessity, so the gate would refuse any commit touching the hook or its tests (or a docs page quoting a key as documentation). Pin the bypass to specific paths and to added lines only, and print what was exempted:
|
||||
|
||||
```bash
|
||||
case "$f" in .githooks/pre-commit|tests/secret-gate.bats)
|
||||
echo "secret-gate: self-test fixtures exempt: $f" >&2; continue;; esac
|
||||
```
|
||||
|
||||
## The scan rule
|
||||
|
||||
Scan **added content only** — `git diff --cached -U0 | grep '^+[^+]'` — or every commit touching a file with an old key in context gets blocked, and the operator's response to a gate that blocks everything is to override it always.
|
||||
|
||||
## Test harness (fixture repo, never the real checkout)
|
||||
|
||||
`git init` a temp repo, `git config core.hooksPath .githooks`, copy the hook in, `git add` a payload, then run it with the env overrides the hook honours (point its cache/notify paths at `$TMP` so tests never touch the user's config). Pin, in order:
|
||||
|
||||
| Case | Expect |
|
||||
|---|---|
|
||||
| fake key on an added line | exit 1, message names the file |
|
||||
| `key = "example"` | pass |
|
||||
| real key + `# example` comment | exit 1 |
|
||||
| non-ASCII filename with a key | exit 1 |
|
||||
| two files with hits | report names both |
|
||||
| self-test fixtures | pass, with the exempt line printed |
|
||||
|
||||
`bats -r` after each edit. `--no-verify` is forbidden on Frank's boxes: the hook IS the control.
|
||||
Reference in new issue
Block a user