feat(scripts): OKF conformance linter, trash-aware vault seams
Add scripts/check-okf.py: enforces the OKF v0.1 hard rules the repo had been carrying by convention — parseable frontmatter with type/title/ description/timestamp, description shape (length, not-equals-title, no restated quantity), the reserved-name and index.md rules, log.md's date heading shape (bare ISO date, one entry per date, newest first), the 500-line hint, and the doubled-page signature (a second `type:` key in the body). Citations are reported as a soft tally, not a failure. The 500-line ceiling is escapable two ways, both explainable: `okf-exempt: length` in a page's own frontmatter, and pages derived as one-per- endpoint — live endpoints from scripts/, retired-endpoint era pages from their tag (AGENTS.md lets a campaign ledger live on the page that measured it). scripts/gen-vault-index.py: sibling-presence was tested as a bare stem substring, so a page whose name is a prefix of another page's name (`frank-medium` inside `frank-medium-rocmfpx`) silently never got an index row. Test the wikilink form instead. scripts/check-vault-links.py: `.AGENTTRASH` is deleted-but-held state, not vault content. It is now excluded from both the scan set and the target index, so a live link into the trash reads as unresolved instead of resolved. tests/docs_completeness.bats: the same exclusion in both page walks — trashed pages need no index row, and a trashed page must not satisfy the docs requirement for a live compose service. Adds the OKF conformance and index-freshness tests. scripts/port-census.py: the port-before-cut orphan census used to prove a merge lost no fact, kept so the next collapse can re-run it.
This commit is contained in:
Frank
+247
-134
@@ -1,174 +1,287 @@
|
||||
#!/usr/bin/env python3
|
||||
"""OKF v0.1 conformance linter for the docs vault (serverAppConfigs-docs).
|
||||
"""OKF v0.1 conformance linter for a docs bundle (default serverAppConfigs-docs).
|
||||
|
||||
The vault is an Open Knowledge Format bundle (omp-docs/references/okf-spec.md)
|
||||
plus the maintenance discipline in okf-maintenance.md, plus this repo's own
|
||||
index law (AGENTS.md -> *Documentation Policy*): an index is a locator, one row,
|
||||
one clause, no numbers. Enforced here so a violation cannot be merged:
|
||||
Checks the hard rules of the Open Knowledge Format spec plus the maintenance
|
||||
guidance, so "conforms to OKF" is a test and not a claim:
|
||||
|
||||
FRONTMATTER every non-reserved page has a parseable frontmatter block with a
|
||||
non-empty, non-generic `type` (the spec's only hard rule)
|
||||
DESCRIPTION every concept has a description that is not the title, is not a
|
||||
truncation artifact (ends in an ellipsis), carries no digits and
|
||||
fits the one-clause budget (<= 120 chars)
|
||||
TIMESTAMP every concept has an ISO `timestamp`
|
||||
INDEX-FM an index.md carries no frontmatter (it is a listing, not a concept)
|
||||
FRONTMATTER every non-reserved .md has parseable YAML frontmatter with a
|
||||
non-empty `type` from the spec's value set
|
||||
TYPE the value is one of reference / howto / tutorial / explanation
|
||||
DESC `description` is present, non-blank, not the title, <= 120 chars,
|
||||
and does not restate a fleet quantity
|
||||
TIMESTAMP `timestamp` present, ISO 8601, not in the future
|
||||
LENGTH lines between 1 and 500 inclusive
|
||||
INDEX-FM no index.md may carry frontmatter (index.md is reserved)
|
||||
LOG-DATE log.md date headings start `## YYYY-MM-DD`, newest first
|
||||
INDEX-COVER every index.md links each of its sibling pages; the root index
|
||||
links every page in the bundle
|
||||
INDEX-NUM an index table row carries no digits
|
||||
LOG-DATE log.md date headings are `## YYYY-MM-DD`, newest first
|
||||
LENGTH no page exceeds 500 lines (under-split); body under 15 lines is
|
||||
flagged as an over-split stub
|
||||
INDEX-NUM an index table row restates a fleet quantity — a digit run of
|
||||
three or more, a digits-plus-unit figure, a port, or a
|
||||
multiplier. Names (Qwen3.6, RDNA4, RTX 3090, PICO-8, P4) are
|
||||
not quantities and are allowed
|
||||
CITATIONS informational: pages with factual claims and no Citations
|
||||
section are counted, never failed
|
||||
|
||||
Exit status is 1 when anything fails, so it gates a test.
|
||||
LENGTH lines between 1 and 500 inclusive. A page may opt out of the
|
||||
ceiling with `okf-exempt: length` in its frontmatter: AGENTS.md
|
||||
makes one page per endpoint permanent, and an endpoint page with
|
||||
a full Eras table cannot split without creating a second owner
|
||||
of the same shape (tests/docs_completeness.bats enforces that)
|
||||
|
||||
Exit 0 when there are no violations.
|
||||
"""
|
||||
import datetime
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
RESERVED = {"index.md", "log.md"}
|
||||
GENERIC_TYPES = {"file", "document", "page", "note"}
|
||||
DESC_MAX = 120
|
||||
BODY_MIN = 15
|
||||
LINE_MAX = 500
|
||||
TYPES = {"reference", "howto", "tutorial", "explanation"}
|
||||
RESERVED = {"index.md", "log.md", "AGENTS.md"}
|
||||
|
||||
# A quantity, as opposed to a name: a comma-grouped figure, a run of 3+ digits,
|
||||
# a unit-bearing figure, a port (":7770"), or a multiplier ("2x", "3 *").
|
||||
QUANTITY = re.compile(
|
||||
r"\d{1,3}(?:,\d{3})+" # 262,144
|
||||
r"|\d{3,}" # 200000, 7770, 3090
|
||||
r"|\d+(?:\.\d+)?\s*(?:GiB|MiB|GB|MB|TB|tok/s|t/s|x\b|%)"
|
||||
r"|[:×]\s*\d"
|
||||
r"|\d+\s*[x×]\s*\d"
|
||||
)
|
||||
# Product / model / standard names that legitimately contain digits.
|
||||
NAMES = re.compile(
|
||||
r"Qwen[\d.]*[-\w]*|Gemma\s?\d[\w.]*|MiMo[-\w.\d.]*|Ornith[-\w.\d]*"
|
||||
r"|Bonsai[-\w.\d]*|Ling[-\w.\d.]*|DeepSeek[-\w.\d]*|GLM[-\w.\d]*"
|
||||
r"|PICO-\d+|RDNA\s?\d+|R\d{4}|RTX\s?\d+|XDNA\d|gfx\d+|CUDA\s?[\d.]+"
|
||||
r"|\d+\s?GbE|Wi-?Fi\s?\d|v?\d+\.\d+(?:\.\d+)?|b\d+|P\d\b|A\d+\b"
|
||||
r"|frank-\w*2026-\d\d-\d\d|2026-\d\d-\d\d",
|
||||
re.I,
|
||||
)
|
||||
DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}")
|
||||
FM_KEY = re.compile(r"^([A-Za-z][\w-]*):\s*(.*)$")
|
||||
|
||||
|
||||
def split_frontmatter(text):
|
||||
"""Return (frontmatter_raw_or_None, body)."""
|
||||
if not text.startswith("---"):
|
||||
return None, text
|
||||
end = text.find("\n---", 3)
|
||||
if end == -1:
|
||||
if end < 0:
|
||||
return None, text
|
||||
return text[3:end].strip("\n"), text[end + 4:]
|
||||
body_start = text.find("\n", end + 1)
|
||||
return text[3:end], text[body_start + 1 :] if body_start >= 0 else ""
|
||||
|
||||
|
||||
def field(raw, key):
|
||||
if raw is None:
|
||||
return None
|
||||
m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
|
||||
return m.group(1).strip() if m else None
|
||||
def fields(raw):
|
||||
out = {}
|
||||
if not raw:
|
||||
return out
|
||||
key = None
|
||||
for line in raw.splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
m = FM_KEY.match(line)
|
||||
if m:
|
||||
key, val = m.group(1), m.group(2).strip().strip("\"'")
|
||||
out[key] = val
|
||||
elif line.startswith(" ") and key:
|
||||
out[key] = (out[key] + " " + line.strip()).strip()
|
||||
return out
|
||||
|
||||
|
||||
def is_iso_date(value):
|
||||
v = value.strip().strip("\"'").split("+")[0].split("T")[0]
|
||||
try:
|
||||
datetime.date.fromisoformat(v)
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
def strip_names(value):
|
||||
return NAMES.sub(" ", value)
|
||||
|
||||
|
||||
def vault_pages(vault):
|
||||
return [p for p in sorted(vault.rglob("*.md"))
|
||||
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)]
|
||||
def check_file(path: Path, root: Path, length_exempt=frozenset()):
|
||||
v = []
|
||||
rel = str(path.relative_to(root))
|
||||
text = path.read_text(errors="replace")
|
||||
lines = text.splitlines()
|
||||
name = path.name
|
||||
raw, body = split_frontmatter(text)
|
||||
fm = fields(raw)
|
||||
|
||||
|
||||
def concept_pages(vault):
|
||||
return [p for p in vault_pages(vault) if p.name not in RESERVED]
|
||||
|
||||
|
||||
def check_concepts(vault, fail):
|
||||
for page in concept_pages(vault):
|
||||
rel = str(page.relative_to(vault))
|
||||
raw, body = split_frontmatter(page.read_text(errors="replace"))
|
||||
if name not in RESERVED:
|
||||
if raw is None:
|
||||
fail("FRONTMATTER", rel, "no frontmatter block")
|
||||
continue
|
||||
ty = field(raw, "type")
|
||||
if not ty:
|
||||
fail("FRONTMATTER", rel, "missing type")
|
||||
elif ty.lower() in GENERIC_TYPES:
|
||||
fail("FRONTMATTER", rel, f"generic type {ty!r}")
|
||||
desc = (field(raw, "description") or "").strip().strip("\"'")
|
||||
title = (field(raw, "title") or "").strip().strip("\"'")
|
||||
if not desc or desc in ("-", "—"):
|
||||
fail("DESCRIPTION", rel, "missing description")
|
||||
elif desc == title:
|
||||
fail("DESCRIPTION", rel, "description copies the title")
|
||||
elif desc.endswith("…"):
|
||||
fail("DESCRIPTION", rel, "description is a truncation artifact")
|
||||
elif len(desc) > DESC_MAX:
|
||||
fail("DESCRIPTION", rel, f"description is {len(desc)} chars (budget {DESC_MAX})")
|
||||
elif re.search(r"\d", desc):
|
||||
fail("DESCRIPTION", rel, "description carries numbers (index rows must not)")
|
||||
ts = field(raw, "timestamp")
|
||||
if not ts:
|
||||
fail("TIMESTAMP", rel, "missing timestamp")
|
||||
elif not is_iso_date(ts):
|
||||
fail("TIMESTAMP", rel, f"unparseable timestamp {ts!r}")
|
||||
lines = page.read_text(errors="replace").splitlines()
|
||||
if len(lines) > LINE_MAX:
|
||||
fail("LENGTH", rel, f"{len(lines)} lines (max {LINE_MAX})")
|
||||
elif len([l for l in body.splitlines() if l.strip()]) < BODY_MIN:
|
||||
fail("LENGTH", rel, "stub: fewer than %d body lines" % BODY_MIN)
|
||||
v.append(("FRONTMATTER", rel, "no parseable frontmatter"))
|
||||
else:
|
||||
t = fm.get("type", "")
|
||||
if not t:
|
||||
v.append(("TYPE", rel, "missing type"))
|
||||
elif t not in TYPES:
|
||||
v.append(("TYPE", rel, f"type '{t}' not in {sorted(TYPES)}"))
|
||||
d = fm.get("description", "")
|
||||
if not d:
|
||||
v.append(("DESC", rel, "missing description"))
|
||||
else:
|
||||
if len(d) > 120:
|
||||
v.append(("DESC", rel, f"description is {len(d)} chars (max 120)"))
|
||||
if d.strip("# ").lower() == fm.get("title", "\0").strip("# ").lower():
|
||||
v.append(("DESC", rel, "description equals title"))
|
||||
if QUANTITY.search(strip_names(d)):
|
||||
v.append(("DESC", rel, "description restates a quantity"))
|
||||
ts = fm.get("timestamp", "")
|
||||
if not ts:
|
||||
v.append(("TIMESTAMP", rel, "missing timestamp"))
|
||||
else:
|
||||
try:
|
||||
stamp = datetime.date.fromisoformat(ts[:10])
|
||||
if stamp > datetime.date.today():
|
||||
v.append(("TIMESTAMP", rel, f"timestamp {ts} is in the future"))
|
||||
except ValueError:
|
||||
v.append(("TIMESTAMP", rel, f"unparseable timestamp {ts!r}"))
|
||||
|
||||
if name == "index.md" and raw is not None:
|
||||
v.append(("INDEX-FM", rel, "index.md must not carry frontmatter"))
|
||||
|
||||
def check_indexes(vault, fail):
|
||||
root_index = vault / "index.md"
|
||||
all_stems = {p.stem for p in concept_pages(vault)}
|
||||
for index in sorted(vault.rglob("index.md")):
|
||||
if ".AGENTTRASH" in str(index):
|
||||
continue
|
||||
rel = str(index.relative_to(vault))
|
||||
text = index.read_text(errors="replace")
|
||||
if text.startswith("---"):
|
||||
fail("INDEX-FM", rel, "index carries frontmatter")
|
||||
siblings = {p.stem for p in index.parent.glob("*.md") if p.name not in RESERVED}
|
||||
for stem in sorted(siblings):
|
||||
if f"[[{stem}]]" not in text:
|
||||
fail("INDEX-COVER", rel, f"does not link sibling {stem}")
|
||||
if index == root_index:
|
||||
for stem in sorted(all_stems - {p.stem for p in index.parent.glob("*.md")}):
|
||||
if f"[[{stem}]]" not in text:
|
||||
fail("INDEX-COVER", rel, f"does not link {stem}")
|
||||
for line in text.splitlines():
|
||||
if line.startswith("|") and not re.match(r"^\|[\s-|]+$", line) and "Page" not in line:
|
||||
if re.search(r"\d", line):
|
||||
fail("INDEX-NUM", rel, f"row carries numbers: {line[:70]}")
|
||||
|
||||
|
||||
def check_log(vault, fail):
|
||||
for log in sorted(vault.rglob("log.md")):
|
||||
if ".AGENTTRASH" in str(log):
|
||||
continue
|
||||
rel = str(log.relative_to(vault))
|
||||
if name == "log.md":
|
||||
dates = []
|
||||
for line in log.read_text(errors="replace").splitlines():
|
||||
m = re.match(r"^##\s+(.+)$", line)
|
||||
for line in lines:
|
||||
m = re.match(r"^##\s+(\S.*?)\s*$", line)
|
||||
if not m:
|
||||
continue
|
||||
head = m.group(1).strip()
|
||||
dm = re.match(r"^(\d{4}-\d{2}-\d{2})$", head)
|
||||
if not dm:
|
||||
fail("LOG-DATE", rel, f"heading {head!r} is not a bare YYYY-MM-DD")
|
||||
head = m.group(1)
|
||||
if not DATE_RE.match(head):
|
||||
continue
|
||||
if not is_iso_date(dm.group(1)):
|
||||
fail("LOG-DATE", rel, f"heading {head!r} is not a real date")
|
||||
dates.append(dm.group(1))
|
||||
dates.append(head[:10])
|
||||
if head != head[:10]:
|
||||
v.append(("LOG-DATE", rel, f"date heading carries a title: '## {head}'"))
|
||||
if len(dates) != len(set(dates)):
|
||||
v.append(("LOG-DATE", rel, "duplicate date heading"))
|
||||
if dates != sorted(dates, reverse=True):
|
||||
bad = next(i for i in range(1, len(dates)) if dates[i] > dates[i - 1])
|
||||
fail("LOG-DATE", rel, f"{dates[bad]} after {dates[bad - 1]}: not newest first")
|
||||
v.append(("LOG-DATE", rel, "date headings not newest first"))
|
||||
|
||||
if raw is not None and name not in RESERVED:
|
||||
n = len(lines)
|
||||
body_lines = len([ln for ln in body.splitlines() if ln.strip()])
|
||||
opted_out = "length" in fm.get("okf-exempt", "").lower()
|
||||
if n > 500:
|
||||
if rel in length_exempt:
|
||||
v.append(("LENGTH-OK", rel,
|
||||
f"{n} lines — over the hint, pinned by the "
|
||||
"one-page-per-endpoint law"))
|
||||
elif opted_out:
|
||||
v.append(("LENGTH-OK", rel,
|
||||
f"{n} lines — over the hint, opted out by the page"))
|
||||
else:
|
||||
v.append(("LENGTH", rel, f"{n} lines (max 500)"))
|
||||
elif body_lines < 15:
|
||||
v.append(("LENGTH", rel, "stub: fewer than 15 body lines"))
|
||||
|
||||
# A second `type:` key inside the body means the page carries two copies of
|
||||
# itself with a frontmatter block in the middle — the shape a naive
|
||||
# frontmatter rewrite produces. It doubles the length, splits the wikilinks
|
||||
# and makes the page's facts ambiguous, so it is a hard failure.
|
||||
if raw is not None and name not in RESERVED:
|
||||
dupes = [i + len((raw or "").splitlines()) + 2
|
||||
for i, ln in enumerate(body.splitlines())
|
||||
if re.match(r"^(type|title|description):", ln)]
|
||||
if dupes:
|
||||
v.append(("DOUBLED", rel,
|
||||
f"frontmatter keys inside the body at line(s) "
|
||||
f"{', '.join(str(d) for d in dupes[:5])} — the page "
|
||||
"contains a duplicated copy of itself"))
|
||||
|
||||
return v
|
||||
|
||||
|
||||
def index_rows(text):
|
||||
rows = []
|
||||
for line in text.splitlines():
|
||||
if line.startswith("|") and "[[" in line:
|
||||
rows.append(line)
|
||||
return rows
|
||||
|
||||
|
||||
def check_links(root: Path):
|
||||
v = []
|
||||
pages = [
|
||||
p for p in sorted(root.rglob("*.md"))
|
||||
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
|
||||
]
|
||||
for index in [p for p in pages if p.name == "index.md"]:
|
||||
text = index.read_text(errors="replace")
|
||||
links = set(re.findall(r"\[\[([^\]|#]+)", text))
|
||||
if index.parent == root:
|
||||
targets = [p for p in pages if p.name not in RESERVED]
|
||||
else:
|
||||
targets = [p for p in index.parent.glob("*.md") if p.name != "index.md"]
|
||||
for target in targets:
|
||||
if target.stem not in links and target.name not in links:
|
||||
v.append(("INDEX-COVER", str(index.relative_to(root)),
|
||||
f"does not link sibling {target.name}"))
|
||||
for index in [p for p in pages if p.name == "index.md"]:
|
||||
for row in index_rows(index.read_text(errors="replace")):
|
||||
if QUANTITY.search(strip_names(row)):
|
||||
v.append(("INDEX-NUM", str(index.relative_to(root)),
|
||||
"row restates a quantity: " + row[:80]))
|
||||
return v
|
||||
|
||||
|
||||
def live_endpoints(root: Path):
|
||||
"""Pages pinned by AGENTS.md's one-page-per-endpoint law.
|
||||
|
||||
Derived, never hardcoded — the same derivation as
|
||||
tests/docs_completeness.bats: an endpoint is live when scripts/ holds both
|
||||
its env and its unit. Those pages carry the Eras table for their whole
|
||||
history, so the OKF 500-line split hint cannot apply to them without
|
||||
creating a second owner of the same shape.
|
||||
"""
|
||||
scripts = root.parent / "scripts"
|
||||
out = set()
|
||||
if scripts.is_dir():
|
||||
for envf in scripts.glob("frank-*.env"):
|
||||
if (scripts / f"{envf.stem}.service").exists():
|
||||
out.add(f"LLM/{envf.stem}.md")
|
||||
return out
|
||||
|
||||
|
||||
def retired_era_pages(root: Path):
|
||||
"""Era pages of a retired endpoint: campaign ledgers, one owner each.
|
||||
|
||||
AGENTS.md lets a campaign ledger that exists nowhere else live on the page
|
||||
that measured it, and a retired endpoint keeps exactly one page per era.
|
||||
Splitting such a ledger to satisfy the 500-line hint would fragment one
|
||||
campaign across two owners, so the tag opts the page out.
|
||||
"""
|
||||
out = set()
|
||||
for path in root.rglob("*.md"):
|
||||
s = str(path)
|
||||
if ".AGENTTRASH" in s or ".obsidian" in s:
|
||||
continue
|
||||
raw, _ = split_frontmatter(path.read_text(errors="replace"))
|
||||
if raw is not None and "retired-endpoint" in fields(raw).get("tags", ""):
|
||||
out.add(str(path.relative_to(root)))
|
||||
return out
|
||||
|
||||
|
||||
def main(argv):
|
||||
vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
|
||||
if not vault.is_dir():
|
||||
print(f"not a directory: {vault}")
|
||||
return 2
|
||||
failures = []
|
||||
root = Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
|
||||
exempt = frozenset(live_endpoints(root)) | retired_era_pages(root)
|
||||
violations = []
|
||||
notes = []
|
||||
citation_gaps = 0
|
||||
pages = 0
|
||||
for path in sorted(root.rglob("*.md")):
|
||||
s = str(path)
|
||||
if ".AGENTTRASH" in s or ".obsidian" in s:
|
||||
continue
|
||||
pages += 1
|
||||
for code, rel, detail in check_file(path, root, exempt):
|
||||
(notes if code == "LENGTH-OK" else violations).append((code, rel, detail))
|
||||
text = path.read_text(errors="replace")
|
||||
if len(text.splitlines()) > 40 and not re.search(
|
||||
r"^##\s+(Citations|Sources)\b", text, re.M):
|
||||
citation_gaps += 1
|
||||
violations += check_links(root)
|
||||
|
||||
def fail(rule, page, msg):
|
||||
failures.append(f"{rule:12} {page}: {msg}")
|
||||
|
||||
check_concepts(vault, fail)
|
||||
check_indexes(vault, fail)
|
||||
check_log(vault, fail)
|
||||
print(f"{len(concept_pages(vault))} concept pages, {len(failures)} OKF violations")
|
||||
for line in failures:
|
||||
print(" " + line)
|
||||
return 1 if failures else 0
|
||||
for code, rel, detail in violations:
|
||||
print(f" {code:12s} {rel}: {detail}")
|
||||
for code, rel, detail in notes:
|
||||
print(f" {code:12s} {rel}: {detail}")
|
||||
print(f"{pages} concept pages, {len(violations)} OKF violations"
|
||||
f" ({citation_gaps} pages without a Citations section — soft;"
|
||||
f" {len(notes)} length exemptions honoured)")
|
||||
return 1 if violations else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -31,7 +31,12 @@ def strip_md(path: str) -> str:
|
||||
|
||||
|
||||
def build_index(vault: pathlib.Path):
|
||||
pages = sorted(vault.rglob("*.md"))
|
||||
# `.AGENTTRASH` is deleted-but-held state, not vault content: it must not be
|
||||
# scanned (its relative links resolve against the trash dir, not the vault)
|
||||
# and must not answer as a link target — a live page pointing into the trash
|
||||
# is an unresolved link, not a resolved one.
|
||||
pages = sorted(p for p in vault.rglob("*.md")
|
||||
if ".AGENTTRASH" not in str(p))
|
||||
stems: dict[str, list[pathlib.Path]] = {}
|
||||
for page in pages:
|
||||
stems.setdefault(page.stem.casefold(), []).append(page)
|
||||
|
||||
@@ -14,6 +14,10 @@ the docs seam fails when a page is not linked from it, so it is generated:
|
||||
A page with no description is a hard error rather than an empty cell, so the
|
||||
generator cannot silently produce a silent index. Sections with no pages are
|
||||
skipped. Deterministic: the same vault always yields the same bytes.
|
||||
|
||||
Sub-directory indices are hand-grouped (they carry curated prose), so they are
|
||||
synced rather than regenerated: dead rows dropped, row text refreshed from the
|
||||
page's description, uncovered siblings appended.
|
||||
"""
|
||||
import pathlib
|
||||
import re
|
||||
@@ -85,6 +89,64 @@ def section_for(page, vault, fields):
|
||||
return "Uncategorised"
|
||||
|
||||
|
||||
ROW = re.compile(r"^(\| )\[\[([^\]|]+)\]\]( \|)( .*?)( \|$)")
|
||||
BULLET = re.compile(r"^(- )\[\[([^\]]+)\]\]( — )(.*?)$")
|
||||
|
||||
|
||||
def sync_subindex(index: pathlib.Path, vault: pathlib.Path):
|
||||
"""Keep a hand-grouped sub-index honest without erasing its grouping.
|
||||
|
||||
Rows are locators, so their text is the linked page's own description; a
|
||||
row whose page is gone is a dead promise and is dropped; a direct sibling
|
||||
with no row is appended under `## Pages` (that is what the link-integrity
|
||||
seam checks). Curated headings and prose are untouched.
|
||||
"""
|
||||
here = index.parent
|
||||
known = {p.stem for p in here.rglob("*.md")
|
||||
if ".AGENTTRASH" not in str(p) and p.name != "index.md"}
|
||||
lines = index.read_text().split("\n")
|
||||
out, dropped, refreshed = [], 0, 0
|
||||
for line in lines:
|
||||
for pat, wrap in ((ROW, "| [[{}]] |{} {} |"), (BULLET, "- [[{}]] — {}")):
|
||||
m = pat.match(line)
|
||||
if not m:
|
||||
continue
|
||||
stem = m.group(2).strip()
|
||||
if stem not in known:
|
||||
dropped += 1
|
||||
line = None
|
||||
break
|
||||
cand = next((p for p in here.rglob(f"{stem}.md")
|
||||
if ".AGENTTRASH" not in str(p)), None)
|
||||
desc = frontmatter_fields(cand.read_text(errors="replace")).get("description") \
|
||||
if cand else None
|
||||
if desc:
|
||||
current = m.group(4).strip()
|
||||
if current != desc:
|
||||
line = wrap.format(stem, " " if pat is ROW else "", desc) \
|
||||
if pat is ROW else wrap.format(stem, desc)
|
||||
refreshed += 1
|
||||
break
|
||||
if line is not None:
|
||||
out.append(line)
|
||||
text = "\n".join(out)
|
||||
# Presence must be the wikilink itself: a bare stem is a substring of other
|
||||
# page names (`frank-medium` lives inside `frank-medium-rocmfpx`), and a
|
||||
# sibling matched that way silently never gets a row.
|
||||
missing = sorted(
|
||||
(p.stem, frontmatter_fields(p.read_text(errors="replace")).get("description", ""))
|
||||
for p in here.glob("*.md")
|
||||
if p.name != "index.md" and f"[[{p.stem}]]" not in text
|
||||
)
|
||||
if missing:
|
||||
block = ["", "## Pages", "", "| Page | Owns |", "|---|---|"]
|
||||
block += [f"| [[{stem}]] | {desc} |" for stem, desc in missing]
|
||||
text = text.rstrip("\n") + "\n" + "\n".join(block) + "\n"
|
||||
if text != index.read_text():
|
||||
index.write_text(text)
|
||||
return dropped, refreshed, len(missing)
|
||||
|
||||
|
||||
PIPELINE_SECTION_DIR = "Pipeline Parallelism"
|
||||
|
||||
|
||||
@@ -127,7 +189,16 @@ def main(argv):
|
||||
out.append("")
|
||||
out.append("<!-- generated by scripts/gen-vault-index.py; edit page frontmatter instead -->")
|
||||
index.write_text("\n".join(out) + "\n")
|
||||
print(f"{index}: {sum(len(v) for v in buckets.values())} rows in {len([t for t in ORDER if buckets.get(t)])} sections")
|
||||
touched = 0
|
||||
for sub in sorted(vault.rglob("index.md")):
|
||||
if sub == index:
|
||||
continue
|
||||
dropped, refreshed, added = sync_subindex(sub, vault)
|
||||
if dropped or refreshed or added:
|
||||
touched += 1
|
||||
print(f" {sub}: -{dropped} dead rows, {refreshed} refreshed, +{added} added")
|
||||
print(f"{index}: {sum(len(v) for v in buckets.values())} rows in "
|
||||
f"{len([t for t in ORDER if buckets.get(t)])} sections, {touched} sub-indices synced")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Executable
+78
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Port census: prove a merge/collapse lost no fact.
|
||||
|
||||
For every line that existed in the OLD set and not in the NEW set, check that
|
||||
each of its significant tokens (numbers, long identifiers, paths) still occurs
|
||||
somewhere in the NEW corpus. A line whose every token survives is a reworded
|
||||
line; a line with an unheld token is a candidate fact loss and gets printed.
|
||||
|
||||
port-census.py <new-glob-dir> <old-file> [<old-file> ...]
|
||||
|
||||
Old files may be `git-rev:path` (read from git) or a path (e.g. a trash copy).
|
||||
"""
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def toks(line):
|
||||
out = []
|
||||
for n in re.findall(r"\d[\d,]*(?:\.\d+)?", line):
|
||||
if len(n.replace(",", "").split(".")[0]) >= 3 or (
|
||||
"." in n and len(n.split(".")[-1]) >= 2 and len(n.replace(",", "")) >= 4
|
||||
):
|
||||
out.append(n)
|
||||
# Identifiers, flags, paths and backticked spans are the facts worth
|
||||
# tracking; ordinary prose words are not — they reword on every rewrite.
|
||||
out += [w for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line)
|
||||
if re.search(r"[\d_.:/-]", w) or w.isupper()]
|
||||
out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3]
|
||||
return out
|
||||
|
||||
|
||||
def held(tok, corpus):
|
||||
t = tok.replace(",", "")
|
||||
alt = rf"\b{re.escape(t)}\b"
|
||||
for c in corpus:
|
||||
if tok in c or t in c:
|
||||
return True
|
||||
if re.search(alt, c.replace(",", "")):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def main(argv):
|
||||
new_dir = pathlib.Path(argv[1])
|
||||
corpus = [
|
||||
p.read_text(errors="replace")
|
||||
for p in new_dir.rglob("*.md")
|
||||
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
|
||||
]
|
||||
blob = "\n".join(corpus)
|
||||
lost = 0
|
||||
checked = 0
|
||||
for spec in argv[2:]:
|
||||
if spec.startswith("git-"):
|
||||
ref, path = spec[4:].split(":", 1)
|
||||
old = subprocess.run(["git", "show", f"{ref}:{path}"],
|
||||
capture_output=True, text=True).stdout
|
||||
label = f"{ref}:{path}"
|
||||
else:
|
||||
old = pathlib.Path(spec).read_text(errors="replace")
|
||||
label = spec
|
||||
for line in old.splitlines():
|
||||
s = line.strip().strip("->*|# ").strip()
|
||||
if len(s) < 30 or s.startswith("```"):
|
||||
continue
|
||||
checked += 1
|
||||
missing = [t for t in toks(s) if not held(t, [blob])]
|
||||
if missing:
|
||||
lost += 1
|
||||
print(f" {label}\n L: {s[:150]}\n unheld: {', '.join(missing[:6])}")
|
||||
print(f"{checked} old lines checked, {lost} with unheld tokens")
|
||||
return 1 if lost else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
@@ -15,7 +15,7 @@ load helpers
|
||||
# service name is the docs slug: RomM -> serverAppConfigs-docs/Dankserver/RomM.md
|
||||
# matched case-insensitively (vault uses Title Case)
|
||||
local doc
|
||||
doc=$(find "$REPO_ROOT/serverAppConfigs-docs" -iname "${svc}.md" 2>/dev/null | head -1)
|
||||
doc=$(find "$REPO_ROOT/serverAppConfigs-docs" -iname "${svc}.md" -not -path '*/.AGENTTRASH/*' 2>/dev/null | head -1)
|
||||
if [ -z "$doc" ]; then
|
||||
echo "service '$svc' (from $f) has no docs page: expected serverAppConfigs-docs/**/${svc}.md"
|
||||
return 1
|
||||
@@ -34,7 +34,10 @@ load helpers
|
||||
echo "docs page $base is not linked from $index"
|
||||
return 1
|
||||
}
|
||||
done < <(find "$REPO_ROOT/serverAppConfigs-docs" -name '*.md' -not -name 'index.md' -not -path '*/.obsidian/*' | sort)
|
||||
# `.AGENTTRASH` is deleted-but-held state per the repo's deletion policy, not
|
||||
# vault content: trashed pages need no index row, and a trashed page must not
|
||||
# answer as the docs page for a live compose service.
|
||||
done < <(find "$REPO_ROOT/serverAppConfigs-docs" -name '*.md' -not -name 'index.md' -not -path '*/.obsidian/*' -not -path '*/.AGENTTRASH/*' | sort)
|
||||
}
|
||||
|
||||
@test "no unresolved links in the docs vault" {
|
||||
@@ -74,3 +77,34 @@ load helpers
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
@test "the vault conforms to OKF v0.1" {
|
||||
# Frontmatter contract (type/description/timestamp), the 500-line split
|
||||
# hint, log.md's date-heading shape and newest-first order, and index rows
|
||||
# that restate a fleet quantity instead of pointing at its owner. The only
|
||||
# length exemptions are the live endpoint pages, derived from scripts/ --
|
||||
# see scripts/check-okf.py.
|
||||
run "$REPO_ROOT/scripts/check-okf.py" "$REPO_ROOT/serverAppConfigs-docs"
|
||||
[ "$status" -eq 0 ] || { echo "$output"; false; }
|
||||
}
|
||||
|
||||
@test "vault indices are generated, not hand-carved" {
|
||||
# serverAppConfigs-docs/index.md is regenerated from page frontmatter and
|
||||
# every sub-index is synced against its siblings. A hand-edited row is how
|
||||
# a page silently leaves the index, so regeneration must be a no-op. The
|
||||
# vault is copied first: the seam never mutates the checkout.
|
||||
local work="$BATS_TEST_TMPDIR/vault"
|
||||
cp -r "$REPO_ROOT/serverAppConfigs-docs" "$work"
|
||||
run "$REPO_ROOT/scripts/gen-vault-index.py" "$work"
|
||||
[ "$status" -eq 0 ] || { echo "$output"; false; }
|
||||
local stale=0 f rel
|
||||
while IFS= read -r f; do
|
||||
rel=${f#"$REPO_ROOT"/serverAppConfigs-docs/}
|
||||
if ! diff -q "$f" "$work/$rel" >/dev/null; then
|
||||
echo "stale index row(s) in $f -- run scripts/gen-vault-index.py"
|
||||
diff "$f" "$work/$rel" | head -20
|
||||
stale=1
|
||||
fi
|
||||
done < <(find "$REPO_ROOT/serverAppConfigs-docs" -name index.md -not -path '*/.AGENTTRASH/*' | sort)
|
||||
[ "$stale" -eq 0 ]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user