docs(agents): documentation exists for point of use

*Documentation Policy* rewritten: a page earns its bytes only if it changes what
the reader does next. Measurements are disposable — a rate for anything not
running and not being compared is destroyed, not filed in an Eras row or a
technique page. "It is the only copy" is explicitly NOT a reason to keep a fact.
`log.md` retired from the policy; the Runbook Index no longer promises a page per
retired endpoint (rollback is a commit, not a document).

*Key Lessons Learned* purged of the dead: the dual-R9700 board-swap narrative and
the layer-split/x4-pair experiment laws (no live endpoint uses either), the QSA
patch note and the retired Kairic/FastMTP traps, and the throughput figures inside
the laws that survive — the law stays, the number goes. Two lessons added, both
from this campaign's own failures: a verdict is a lead, the scout's and the tool's
alike; and never let a census decide what to keep, because a tool can say where a
fact lives but not whether anything needs it.

997 → 820 lines.
This commit is contained in:
Frank
2026-10-19 10:56:37 -07:00
parent 327e78b950
commit a3bce7d123
+67 -96
View File
@@ -47,7 +47,7 @@ per attempt and no separate swap-history file.
The managed skill library is distilled there one page per skill-cluster:
`LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
`Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
pages. There is no `log.md`: the change diary was destroyed 2026-10-19
- `tests/` — bats suite + Playwright e2e
## Git Workflow — trunk-based (fleet law, aligned 2026-10-18)
@@ -318,35 +318,36 @@ the next), so never report request #1.
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.
## Documentation Policy (fleet law, 2026-10-18)
## Documentation Policy (fleet law, 2026-10-19)
**One fact, one owner.** A page owns exactly one thing — an endpoint, a
service, or a technique — and every other file links to it with `[[Page]]`.
Restating a shape, a digest, a port list, or a measured number in a second
file is a defect: it is the next thing to go stale.
**Documentation exists for point of use.** A page earns its bytes only if it
helps whoever performs the next task — the shape to deploy, the flag that must
be passed, the failure signature to recognise, the rollback to run. It is not a
lab notebook and not a ledger. **Losing information is an acceptable, usually
correct outcome.** Never let an orphan census ("this is the only copy of that
fact") decide what to keep: sole custody of a useless fact is still useless.
- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
— current shape, an **Eras** table (`window | payload | runtime | shape |
replaced because | rollback`), the **Laws** the endpoint taught, the wire
contract, and the files that own it. A payload swap edits the shape block
and appends one Era row. It never creates a page, and the retired era's
page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
table.
- **`log.md` records incidents and laws** — anything with a causal story
(why the host went down, why a knob resets at reboot, why the RAM was the
fault). A model swap is not an incident: it is an Era row and nothing else.
- **`AGENTS.md` carries operator-critical numbers only** — port, context
ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
- **The purpose test — before you write, and again before you keep:** would the
reader act differently because of this line? No → delete it.
- **Measurements are disposable.** A rate or throughput figure (t/s, tok/s, ms,
acceptance %, a bench ladder, a measured MiB) for a payload, runtime, or
layout that is not running now and not being compared now is destroyed, not
filed — not in an Eras row, not on a technique page, not "preserved for
history". AI moves faster than this vault: the number is wrong before anyone
reads it, and a wrong number in a runbook gets believed. Keep the **law** the
measurement taught; drop the measurement.
- **One fact, one owner**, every other file links with `[[Page]]`. The env file
beside the serve script is the authority for every live number; the endpoint
page is the authority for the shape and the wire contract.
- **One page per endpoint:** current shape, **Eras** (`window | payload |
runtime | replaced because | rollback` — the rollback is a commit, not a
number), the **Laws**, the wire contract, the files that own it. A swap edits
the shape block and appends one row; the retired era's own page is
**destroyed** to `.AGENTTRASH/`, never merged into anything.
- **`log.md` is retired** (2026-10-19: 2,380 lines destroyed). Git holds the
chronology. A causal law belongs to the page that owns the thing it governs,
or to *Key Lessons Learned* below if it is cross-cutting — never to a diary.
- **Index pages are locators:** one row, one clause, no numbers.
- **A measured number is written once:** in the env-file header beside the
serve script that consumes it, and in the endpoint page. Every other page
cites it by link.
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
technique pages: they stop carrying a per-endpoint throughput table or a live
shape, but a campaign ledger that exists nowhere else is owned by the page that
measured it. The defect is two pages carrying one figure, not a number living on a
technique page — move a figure only when an endpoint page or an env header is its
home; otherwise leave it in place and name the era it measured.
## Model Config (harness)
@@ -487,36 +488,10 @@ linked runbooks.
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
failing HIP call with its return code. [[PCIe Link and Atomics]]
- **Layer split is the no-collective path, on one vendor's cards and across two — and
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
`--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
(`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
[[x4-pair-experiment]]
**ROCm / GPU**
@@ -530,7 +505,7 @@ linked runbooks.
the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]]
and pinning the PLE tensor with `-ot` was slower than letting it page. [[Llama Cpp Serving Sizing]]
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
`1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
@@ -545,9 +520,6 @@ linked runbooks.
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
`blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the
kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]]
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
`/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
@@ -602,13 +574,11 @@ linked runbooks.
`LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
before adding flags. [[frank-dragoon]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a
loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103
(`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a
preference but the only door. Counting tensor types out of the GGUF KV section takes one python
read and settles a runtime argument in minutes. Build identity matters as much as the fork: the
2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a
file to a loader.** A non-standard tensor type in the KV section makes stock loaders refuse
the file at load, so a fork can be the only door rather than a preference. Counting tensor
types takes one python read and settles a runtime argument in minutes; build identity matters
as much as the fork, so pin the commit by SHA in any build script. [[GGUF Compatibility]]
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
(the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
@@ -617,12 +587,6 @@ linked runbooks.
exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
back before believing a number — and kill by pid, never `pkill -f llama-server`.
[[LLM Bench Windows and Measurement]]
- Retired-runtime traps that can come back: **Kairic Edge** needs
`third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean`
removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so
re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS
patch, since stock binaries fail the draft load with a tensor-shape mismatch.
**llama.cpp serving**
@@ -635,10 +599,10 @@ linked runbooks.
`--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
`ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
`draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash`
legal and it wins on prompt-echo traffic, but `draft-mtp,draft-dflash`
dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs
hybrid loses, because adding the ngram layer logs
`recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
[[frank-dragoon]]
@@ -647,7 +611,7 @@ linked runbooks.
declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
[[Llama Cpp Multi-Device Splits]]
[[Llama Cpp Serving Sizing]]
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
@@ -664,21 +628,20 @@ linked runbooks.
re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at
512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by
restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and
[[frank-zealot]].
control that proves the knob, not the traffic, moved the number. The optimum is a property of
the backend, so re-find it after any device or payload change. When a bench IS ordered, A/B it
by restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
port: one card holds one server.
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
[[Llama Cpp Multi-Device Splits]]
[[Llama Cpp Serving Sizing]]
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
too short and it proposes on ordinary prose fragments, costing verification without acceptance.
[[Llama Cpp Multi-Device Splits]]
[[Llama Cpp Serving Sizing]]
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
@@ -793,17 +756,24 @@ linked runbooks.
`curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three;
ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account
accepts the gate. [[Model Swap and Rollback]]
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six
read-only scouts worked the same pages: one classified era pages as stale *because* they
document retired payloads (that is an era page's content), one reported files that do not exist,
one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held
by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and
one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every
claimed duplicate, every claimed staleness and every quoted holder against the file and today's
fleet state before editing; a scout yield buys no evidence.
declared size, and check the LFS sha256 — the live `scripts/download-frank-*.sh` scripts do
all three; ad-hoc fetches tend not to. Gated repos 403 even with the host's HF token until
that account accepts the gate. [[Model Swap and Rollback]]
- **A verdict is a lead — the scout's and the tool's alike.** In the 2026-10-19
documentation campaign six scouts worked the same pages and produced invented
bullet IDs, files that do not exist, "duplications" grep disproved, line
numbers past the end of the file they judged, and Eras tables with invented
payloads. The mechanical tool was worse: `scripts/port-census.py` built its
corpus from the directory containing the page under test, so each page was held
by *itself* and ten era pages reported "0 sole-record lines" — which reads as
"pure duplication, trash it". Nothing was cut on a verdict; every holder was
re-grepped and every tool re-measured before it was believed.
- **Never let a census decide what to keep.** That same campaign then concluded
the era pages were irreplaceable because each held 14–63 sole-record lines —
wrong in the opposite direction: those were throughput figures for payloads that
no longer run. Point of use is the only ownership test, the operator applies it,
and 53 pages plus the 2,380-line change diary were destroyed as a result. A tool
can say where a fact lives; it cannot say whether anything needs it.
## Runbook Index
@@ -818,12 +788,13 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.
There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself
dies, its page is destroyed. Rollback is a commit, not a document.
`serverAppConfigs-docs/index.md` is the locator for every page that survives;
retired scripts live in `old/` and `.AGENTTRASH/`.
## Why