fix(scripts): port-census must not call prose words facts
A hyphen or a trailing period inside a word was enough to make it a tracked token, so `long-standing`, `deployed:` and `cleaned.` were reported as lost facts while real identifiers hid behind them. Require a digit, dot, slash, colon, `--`, two hyphens (`frank-tts-engine`) or ALL-CAPS in the stripped word. The census of the log rewrite went from 979 noisy lines to a set whose unheld tokens are all fact-shaped.
This commit is contained in:
@@ -25,8 +25,14 @@ def toks(line):
|
|||||||
out.append(n)
|
out.append(n)
|
||||||
# Identifiers, flags, paths and backticked spans are the facts worth
|
# Identifiers, flags, paths and backticked spans are the facts worth
|
||||||
# tracking; ordinary prose words are not — they reword on every rewrite.
|
# tracking; ordinary prose words are not — they reword on every rewrite.
|
||||||
out += [w for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line)
|
# A hyphen alone is not evidence of an identifier (`long-standing`), so a
|
||||||
if re.search(r"[\d_.:/-]", w) or w.isupper()]
|
# dashed word counts only with two hyphens (`frank-tts-engine`) or with a
|
||||||
|
# digit, dot, slash, colon or `--` inside it.
|
||||||
|
for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line):
|
||||||
|
core = w.strip(".,;:!?\"')]")
|
||||||
|
if (core.startswith("--") or core.isupper()
|
||||||
|
or re.search(r"[\d_/:.]", core) or core.count("-") >= 2):
|
||||||
|
out.append(core)
|
||||||
out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3]
|
out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3]
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user