50 lines
3.1 KiB
Markdown
50 lines
3.1 KiB
Markdown
---
|
|
name: html-report-ground-truth-audit
|
|
description: "Audit a self-generated HTML report: ground every numeric/code/diagram claim against the repo, fix wrong values, delete filler that restates git, verify the served artifact byte-identical, and flag unverifiable content to the operator."
|
|
---
|
|
|
|
Ground-truth audit of an HTML report (generated by an agent, for a human). Three defects hide in a plausible report: **wrong numbers, unverifiable claims, and filler that restates git for a reader who has git.**
|
|
|
|
## Law
|
|
|
|
Every numeric claim must be re-measured against the repo, not the narrative that produced it.
|
|
|
|
## Procedure
|
|
|
|
1. **Dump claims with file:line anchors.**
|
|
`sed 's/<[^>]*>//g' report.html | grep -vE '^\s*$' | grep -nE 'commit|file|line|fix|refactor' > /tmp/claims.txt`
|
|
Then enumerate the ground truth: `git log --oneline origin/main..HEAD` for commits; `git diff --stat origin/main..HEAD` for the per-file census — **sort it**: `--sort=-deletions` and `--sort=-creations` give the top movers, so the "largest single change" and "largest net code change" are computed facts.
|
|
|
|
2. **Check code blocks compile.** Write every snippet to a temp file and `bash -n` each. Common breakages from HTML generation: `"`/`&` entities leaking into code, and `<br>` line breaks that split a line mid-pipe.
|
|
|
|
3. **Check the diagrams carry information.** A mermaid `flowchart LR` whose nodes merely restate the bullet list above it is decoration — delete it.
|
|
|
|
4. **Check counts.** Commit and file counts must match `git rev-list --count` and `--name-only | sort -u | wc -l`. A report whose sections were written by different subagents drifts: each counts its own slice ("4 commits / 14 files" vs the true 11 / 25).
|
|
|
|
5. **Flag what cannot be verified.** Test pass claims, external tool output, runtime behaviour — if the repo can't confirm it, say so to the operator instead of asserting it.
|
|
|
|
## Filler test (delete on failure)
|
|
|
|
- A `file → line-count → description` table restating `git diff --stat`? Delete — the reader has git.
|
|
- A "what this means" / "conclusion" / "key takeaways" section? Delete — it is the same facts in softer prose. (Same rule as vault pages: point of use or gone.)
|
|
- Keep only what a reader cannot rebuild from `git log` + `git show`.
|
|
|
|
## Serve and verify the artifact
|
|
|
|
**Verify the file that is served, not the source.** If a `public/` copy is byte-identical, edit both or the served artifact stays wrong. Then re-fetch:
|
|
|
|
```bash
|
|
base=http://host:port/reports/x.html
|
|
for i in $(seq 1 30); do curl -fsS -o /tmp/got.html "$base" && break; sleep 1; done # service may still be activating
|
|
for i in 1 2; do curl -fsS "$base" | md5sum; done # stable across refetches
|
|
diff <(curl -fsS "$base") /path/to/source.html && echo "served == source"
|
|
```
|
|
|
|
## Voice drift
|
|
|
|
If the operator asked for a tone ("no filler", "terse"), the audit must enforce it too — marketing phrasing ("ships the", "This means:") is a style defect, fix it in the same pass, not a later one.
|
|
|
|
## Report
|
|
|
|
Fixes grouped by defect class (wrong number / unverifiable / filler / style), then what stays unverifiable and why, with the measured ground truth quoted.
|