docs(vault): documentation exists for point of use — destroy the ledgers (−13.7k lines) #31

Merged
dank merged 13 commits from docs/vault-destruction-20261019 into main 2026-10-19 10:59:51 -07:00
Owner

Documentation exists for point of use: a page earns its bytes only if it changes what the reader does next. It is not a lab notebook, and "this is the only copy of that fact" is not a reason to keep it. A throughput figure for a payload nobody runs is worse than nothing — AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed.

This PR applies that test and destroys what fails it. Supersedes #28, which you rejected.

Destroyed

What Pages Lines
frank-medium endpoint + five era pages, frank-npu, FastFlowLM, the ikllama / kingjones / dealign / frank-redo eras, the ling / swift15 / freetoken / ubatch campaign pages, qwen3.8 spec-matrix + strix-optimizations, both "Running Qwen3.6 …" pages 22 ~5,900
Pipeline Parallelism/ (no live script passes a pipeline flag) + the multi-device / x4-pair / hybrid / offload / Dual-R9700 experiment pages 11 ~2,800
Every vLLM page — no vLLM endpoint exists on this fleet 8 ~1,830
External surveys: SGLang, Lemonade, llama.cpp forks + code diff, self-hosted app census, the TTS survey 6 ~1,260
LLM/deployment.md (the Architecture/Deployment Flow.md page is the test-referenced owner), bench-window methodology 2 ~230
log.md — the change diary 1 2,380

Corpses and the manifest (KILL-LIST.txt) sit in .AGENTTRASH/20261019-destruction/ pending your OK to purge.

Result, measured against b9d21ae

Area Before After Δ
serverAppConfigs-docs/ 31,210 17,542 −13,668
AGENTS.md 997 820 −177
scripts/ 19,520 20,129 +609 (four destruction/repair tools)
Repo total 74,223 ~61,000 ≈ −13,200

155 → 101 pages. ./tests/run.sh: ALL TESTS PASSED (451). OKF linter: 0 violations. Vault link check: 0 unresolved.

What survived, and why

  • Live shapes, ports, context ceilings, KV budgets, env numbers — the deployer and models.yml consume them.
  • Laws and causal stories (PCIe atomics, mmap, GTT aperture, ttm preflight, --metrics, the leftover-drop-in hijack), with the throughput figures cut out of them.
  • Service pages for running software, the bench procedure for when you order a bench, security posture.
  • Eras tables keep window / payload / runtime / reason / rollback pointer; their rate columns are blanked to —, and older eras fold into one row that says the numbers were destroyed and that rollback is a commit.

Two self-inflicted wounds, both recorded in code

  • The orphan census (port-census.py) built its corpus from the directory containing the page under test, so every page proved itself held and ten era pages reported "0 sole-record lines" — which read as "trash them". Fixed to exclude the file under test; the lesson is a Key Lessons bullet now.
  • The first version of repair-damage.py matched every wrapped prose line ending in a comma and deleted 660 of them. restore-punctuation.py undid them from HEAD (325 lines byte-identical apart from trailing punctuation); the rule is now … see . only, with the incident in its docstring.

Also

The same purpose test is now in the global agent settings (~/.omp/agent/APPEND_SYSTEM.md → THE PURPOSE TEST), verified live with a quote probe from a fresh session.

Open question

Purge .AGENTTRASH/ now (the 54 pages + diary are still on disk, ~14k lines), or keep them recoverable a while?


cwd: /home/blank/.omp/wt/wt-20261019-021028-6d1fc1f · session: 01a12303-23f0-75e3-b3cd-546771efacc7

Documentation exists for **point of use**: a page earns its bytes only if it changes what the reader does next. It is not a lab notebook, and "this is the only copy of that fact" is not a reason to keep it. A throughput figure for a payload nobody runs is worse than nothing — AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. This PR applies that test and destroys what fails it. Supersedes #28, which you rejected. ## Destroyed |What|Pages|Lines| |---|---|---| |`frank-medium` endpoint + five era pages, `frank-npu`, FastFlowLM, the ikllama / kingjones / dealign / frank-redo eras, the ling / swift15 / freetoken / ubatch campaign pages, qwen3.8 spec-matrix + strix-optimizations, both "Running Qwen3.6 …" pages|22|~5,900| |`Pipeline Parallelism/` (no live script passes a pipeline flag) + the multi-device / x4-pair / hybrid / offload / Dual-R9700 experiment pages|11|~2,800| |Every vLLM page — no vLLM endpoint exists on this fleet|8|~1,830| |External surveys: SGLang, Lemonade, llama.cpp forks + code diff, self-hosted app census, the TTS survey|6|~1,260| |`LLM/deployment.md` (the `Architecture/Deployment Flow.md` page is the test-referenced owner), bench-window methodology|2|~230| |`log.md` — the change diary|1|2,380| Corpses and the manifest (`KILL-LIST.txt`) sit in `.AGENTTRASH/20261019-destruction/` pending your OK to purge. ## Result, measured against `b9d21ae` |Area|Before|After|Δ| |---|---|---|---| |`serverAppConfigs-docs/`|31,210|17,542|**−13,668**| |`AGENTS.md`|997|820|**−177**| |`scripts/`|19,520|20,129|+609 (four destruction/repair tools)| |**Repo total**|74,223|~61,000|**≈ −13,200**| 155 → **101 pages**. `./tests/run.sh`: **ALL TESTS PASSED** (451). OKF linter: **0 violations**. Vault link check: **0 unresolved**. ## What survived, and why - Live shapes, ports, context ceilings, KV budgets, env numbers — the deployer and `models.yml` consume them. - Laws and causal stories (PCIe atomics, mmap, GTT aperture, ttm preflight, `--metrics`, the leftover-drop-in hijack), with the throughput figures **cut out of** them. - Service pages for running software, the bench procedure for when you order a bench, security posture. - Eras tables keep window / payload / runtime / reason / rollback pointer; their rate columns are blanked to `—`, and older eras fold into one row that says the numbers were destroyed and that rollback is a commit. ## Two self-inflicted wounds, both recorded in code - The orphan census (`port-census.py`) built its corpus from the directory containing the page under test, so every page proved itself held and ten era pages reported "0 sole-record lines" — which read as "trash them". Fixed to exclude the file under test; the lesson is a *Key Lessons* bullet now. - The first version of `repair-damage.py` matched every wrapped prose line ending in a comma and deleted 660 of them. `restore-punctuation.py` undid them from `HEAD` (325 lines byte-identical apart from trailing punctuation); the rule is now `… see .` only, with the incident in its docstring. ## Also The same purpose test is now in the **global** agent settings (`~/.omp/agent/APPEND_SYSTEM.md` → *THE PURPOSE TEST*), verified live with a quote probe from a fresh session. ## Open question Purge `.AGENTTRASH/` now (the 54 pages + diary are still on disk, ~14k lines), or keep them recoverable a while? --- cwd: /home/blank/.omp/wt/wt-20261019-021028-6d1fc1f · session: 01a12303-23f0-75e3-b3cd-546771efacc7
dank added 13 commits 2026-10-19 10:57:16 -07:00
Key Lessons Learned 483 -> 340 lines. Every law survives; only the essays
went. Port-before-cut census over the diff: 454 deleted lines, 0 orphaned
facts (every number still has a holder in the vault).

- Each lesson is now one bullet with the page that owns the detail.
- De-duplicated the page's own internal repeats: the layer-split law was
  told twice (PCIe and ROCm sections), the FA-quants law twice more.
- CORRECTION, verified on dankclaw 2026-10-19: `GGML_CUDA_FA_ALL_QUANTS` is
  deprecated upstream (master's ggml/cmake/common.cmake tells you to use
  GGML_CUDA_FA_QUANTS=all), the successor's default pair list
  `q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16` compiles the quantized-KV kernels
  with the old flag OFF (master build-cuda cache: FA_ALL_QUANTS=OFF), and
  neither llama tree references the old macro under ggml/src/ggml-cuda/.
  The "required or prefill collapses ~12x" law is obsolete; the new law is
  to read the CMakeCache pair list.
- frank-zealot: ported the three facts AGENTS.md alone carried, and verified
  them against the host first — the 2026-10-18 13:16 cutover boot really did
  load PTQ1_0 at `new slot, n_ctx = 133888` x3 (journal), the mmproj fit
  check really printed 873.10 MiB against a 21,439 MiB projection, and the
  warmup spread is 2.6 -> 1,258 t/s prefill.
- New tooling: scripts/check-okf.py (OKF v0.1 conformance linter) and
  scripts/gen-vault-index.py (root index regenerated from frontmatter).

Co-Authored-By: omp <omq@openclaw.sh>
Add scripts/check-okf.py: enforces the OKF v0.1 hard rules the repo had
been carrying by convention — parseable frontmatter with type/title/
description/timestamp, description shape (length, not-equals-title, no
restated quantity), the reserved-name and index.md rules, log.md's date
heading shape (bare ISO date, one entry per date, newest first), the
500-line hint, and the doubled-page signature (a second `type:` key in
the body). Citations are reported as a soft tally, not a failure. The
500-line ceiling is escapable two ways, both explainable: `okf-exempt:
length` in a page's own frontmatter, and pages derived as one-per-
endpoint — live endpoints from scripts/, retired-endpoint era pages from
their tag (AGENTS.md lets a campaign ledger live on the page that
measured it).

scripts/gen-vault-index.py: sibling-presence was tested as a bare stem
substring, so a page whose name is a prefix of another page's name
(`frank-medium` inside `frank-medium-rocmfpx`) silently never got an
index row. Test the wikilink form instead.

scripts/check-vault-links.py: `.AGENTTRASH` is deleted-but-held state,
not vault content. It is now excluded from both the scan set and the
target index, so a live link into the trash reads as unresolved instead
of resolved.

tests/docs_completeness.bats: the same exclusion in both page walks —
trashed pages need no index row, and a trashed page must not satisfy the
docs requirement for a live compose service. Adds the OKF conformance
and index-freshness tests.

scripts/port-census.py: the port-before-cut orphan census used to prove
a merge lost no fact, kept so the next collapse can re-run it.
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
A hyphen or a trailing period inside a word was enough to make it a tracked
token, so `long-standing`, `deployed:` and `cleaned.` were reported as lost
facts while real identifiers hid behind them. Require a digit, dot, slash,
colon, `--`, two hyphens (`frank-tts-engine`) or ALL-CAPS in the stripped
word. The census of the log rewrite went from 979 noisy lines to a set whose
unheld tokens are all fact-shaped.
Four bullets in Key Lessons Learned carried narrative a vault page owns: the
2026-09-20 board swap ([[Dual-R9700-Cooperation]]), the `--override-tensor`
gate ([[Llama Cpp Multi-Device Splits]]), the x4 link budget
([[x4-pair-experiment]]), and the silent-RAM incident plus the Prometheus
`--metrics` law ([[frank-archon]], [[frank-zealot]]). Each keeps its law, its
numbers and its error strings; the causal narration that duplicated a page is
what went. The RAM bullet linked to [[log]] — the incident is owned by
[[frank-archon]], so the link now names the owner.
- frank-archon KV dtype law: `bf16|fp8` is a **radiance** restriction. Stock
  vLLM advertises `auto|fp8|fp8_e5m2|fp8_e4m3` (docs.vllm.ai engine args), and
  even there the advertised set is not the usable set — several accepted
  values crash engine init at the attention layer's own dtype check
  (vllm-project/vllm#42587). The page stated the restriction as if upstream
  had it.
- Open Notebook: re-censused 2026-10-19 — no unit file, no installed quadlet,
  no container. The page already said NOT deployed; it now says what was
  checked, so the claim is re-checkable instead of remembered.
The ROCmFPX row said "single card, pinned at 36e9acd": `36e9acd` is the
SHA that `scripts/build-rocmfpx.sh` pins for the *build*, and the era page
records the deployed commit as `c49ebdb` running a 2× R9700 `-sm layer`
split over `ROCm0,ROCm1` at 131,072 × 1 slot with draft-MTP pinned to
`ROCm1`. The 2026-09-20 row skipped a payload generation: the same day the
endpoint swapped MXFP4 → FP8 E4M3 128×128 ([[frank-medium-model-swap]])
before MXFP6 arrived on 09-21. Both cells now quote the era pages.
main moved under this campaign — PR #25/#26 resliced the RTX 3090 endpoint.
The frontmatter conflict is resolved in favour of the live shape, confirmed
by re-probe the same day: `--ctx-size 602112 --parallel 3` (3 × 200,704),
`--cache-type-k f16 --cache-type-v q8_0`, `--spec-type draft-mtp`, `--jinja
--reasoning auto`, `--metrics`. The 262,144 figures that remain on the page
are era rows and the checkpoint's trained window, not the live shape.

log.md's 2026-10-19 fleet-state bullet now records both probes and the law
this taught: a docs pass that reports live shapes must re-probe before
writing the number, because main moving mid-campaign makes the first probe
stale.
`Multi-Node Runtimes.md` (+257) restated launch material that
[[Qwen3.8 PP Deployment]] (build target, two-process RPC table, cache-prune
cron), [[Sample Script Set]] (runnable scripts), [[Ethernet Bandwidth Math]]
and [[Expected Performance]] already own, and mixed-vendor numbers owned by
[[x4-pair-experiment]]. Consolidation means the facts end up in ONE place, not
that a new page is written beside the old owners: the four stubs it replaced
were frontmatter-only (9 lines each) and needed no successor. Page trashed,
three inbound links repointed at the real owners.
`scripts/port-census.py` built its corpus from the new directory, so each page
under test was held by itself: every era page reported 0 orphan lines. With the
corpus excluding the file under test, each candidate holds 14-63 sole-record
lines, so no era page is padding and none was trashed.

Deletions of restatement only:
- Multi-Node Runtimes.md (-257): restated launch material owned by
  Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed.
- log.md: 66 headings folded into 35 dated entries (one entry per date, newest
  first), titled headings kept as `date — Title` per the spec's wording, and 78
  lines of transient deploy chatter dropped where every fact token in the line
  is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/).
- Authentic: the 5-page split reverted (-93); the parent carries
  `okf-exempt: length` and the three stub pages stay deleted.

check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file
under test and gained a docstring note about the trap. One-off compaction tool
moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations,
155 pages, 0 unresolved links.
Point-of-use enforcement needs hands, not advice:

- `destroy-vault.py` — moves the pages on a kill list to `.AGENTTRASH/` and then
  deletes every reference to them from the survivors, including the whole line
  when the dead link was the line's only content. Stems are seeded from the trash
  so the pass is re-runnable; `pathlib.Path.stem` is NOT used to compare link
  targets, because it strips the real suffix off names like `sd.cpp`.
- `axe-measurements.py` — deletes bench/throughput ledger sections and rate rows,
  blanks any table cell that reports a rate, and folds Eras older than the current
  one into a single row that says the numbers were destroyed and that rollback is
  a commit. Never touches a shape number or a Law/Wire/Rollback section.
- `repair-damage.py` — fixes the two artifacts those passes can leave (a list
  marker that lost its space, a table separator left without its header).
- `restore-punctuation.py` — restores a line whose only difference from HEAD is
  stripped trailing punctuation. Written after `repair-damage.py`'s first version
  matched every wrapped prose line ending in a comma and deleted 660 of them: the
  rule is now `… see .` only, and the incident is recorded in its docstring.
- `gen-vault-index.py` — never walks into `.AGENTTRASH/`, and sweeps rows whose
  page cell is empty (a description left nameless by a destruction).

Full suite: 451 tests pass.
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
*Documentation Policy* rewritten: a page earns its bytes only if it changes what
the reader does next. Measurements are disposable — a rate for anything not
running and not being compared is destroyed, not filed in an Eras row or a
technique page. "It is the only copy" is explicitly NOT a reason to keep a fact.
`log.md` retired from the policy; the Runbook Index no longer promises a page per
retired endpoint (rollback is a commit, not a document).

*Key Lessons Learned* purged of the dead: the dual-R9700 board-swap narrative and
the layer-split/x4-pair experiment laws (no live endpoint uses either), the QSA
patch note and the retired Kairic/FastMTP traps, and the throughput figures inside
the laws that survive — the law stays, the number goes. Two lessons added, both
from this campaign's own failures: a verdict is a lead, the scout's and the tool's
alike; and never let a census decide what to keep, because a tool can say where a
fact lives but not whether anything needs it.

997 → 820 lines.
dank merged commit d09d2745cf into main 2026-10-19 10:59:51 -07:00
dank deleted branch docs/vault-destruction-20261019 2026-10-19 10:59:52 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#31