fix(scripts): port-census must not call prose words facts

A hyphen or a trailing period inside a word was enough to make it a tracked
token, so `long-standing`, `deployed:` and `cleaned.` were reported as lost
facts while real identifiers hid behind them. Require a digit, dot, slash,
colon, `--`, two hyphens (`frank-tts-engine`) or ALL-CAPS in the stripped
word. The census of the log rewrite went from 979 noisy lines to a set whose
unheld tokens are all fact-shaped.
This commit is contained in:
Frank
2026-10-19 05:08:22 -07:00
parent a5fe045b58
commit 483fd843b7
+8 -2
View File
@@ -25,8 +25,14 @@ def toks(line):
out.append(n)
# Identifiers, flags, paths and backticked spans are the facts worth
# tracking; ordinary prose words are not — they reword on every rewrite.
out += [w for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line)
if re.search(r"[\d_.:/-]", w) or w.isupper()]
# A hyphen alone is not evidence of an identifier (`long-standing`), so a
# dashed word counts only with two hyphens (`frank-tts-engine`) or with a
# digit, dot, slash, colon or `--` inside it.
for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line):
core = w.strip(".,;:!?\"')]")
if (core.startswith("--") or core.isupper()
or re.search(r"[\d_/:.]", core) or core.count("-") >= 2):
out.append(core)
out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3]
return out