From a3bce7d123cd968c1a75bca65c97896281571fca Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 10:56:37 -0700 Subject: [PATCH] docs(agents): documentation exists for point of use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit *Documentation Policy* rewritten: a page earns its bytes only if it changes what the reader does next. Measurements are disposable — a rate for anything not running and not being compared is destroyed, not filed in an Eras row or a technique page. "It is the only copy" is explicitly NOT a reason to keep a fact. `log.md` retired from the policy; the Runbook Index no longer promises a page per retired endpoint (rollback is a commit, not a document). *Key Lessons Learned* purged of the dead: the dual-R9700 board-swap narrative and the layer-split/x4-pair experiment laws (no live endpoint uses either), the QSA patch note and the retired Kairic/FastMTP traps, and the throughput figures inside the laws that survive — the law stays, the number goes. Two lessons added, both from this campaign's own failures: a verdict is a lead, the scout's and the tool's alike; and never let a census decide what to keep, because a tool can say where a fact lives but not whether anything needs it. 997 → 820 lines. --- AGENTS.md | 163 ++++++++++++++++++++++-------------------------------- 1 file changed, 67 insertions(+), 96 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 29ec942..1db3f7a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ per attempt and no separate swap-history file. The managed skill library is distilled there one page per skill-cluster: `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`, `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service - pages (`serverAppConfigs-docs/log.md` carries the wave plan and results) + pages. There is no `log.md`: the change diary was destroyed 2026-10-19 - `tests/` — bats suite + Playwright e2e ## Git Workflow — trunk-based (fleet law, aligned 2026-10-18) @@ -318,35 +318,36 @@ the next), so never report request #1. deletes the unit file from `/etc/systemd/system/`, not just the symlink. `mask` symlinks to `/dev/null` and is safe/reversible. -## Documentation Policy (fleet law, 2026-10-18) +## Documentation Policy (fleet law, 2026-10-19) -**One fact, one owner.** A page owns exactly one thing — an endpoint, a -service, or a technique — and every other file links to it with `[[Page]]`. -Restating a shape, a digest, a port list, or a measured number in a second -file is a defect: it is the next thing to go stale. +**Documentation exists for point of use.** A page earns its bytes only if it +helps whoever performs the next task — the shape to deploy, the flag that must +be passed, the failure signature to recognise, the rollback to run. It is not a +lab notebook and not a ledger. **Losing information is an acceptable, usually +correct outcome.** Never let an orphan census ("this is the only copy of that +fact") decide what to keep: sole custody of a useless fact is still useless. -- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/.md` - — current shape, an **Eras** table (`window | payload | runtime | shape | - replaced because | rollback`), the **Laws** the endpoint taught, the wire - contract, and the files that own it. A payload swap edits the shape block - and appends one Era row. It never creates a page, and the retired era's - page goes to `.AGENTTRASH/` once its rollback path and its laws are in the - table. -- **`log.md` records incidents and laws** — anything with a causal story - (why the host went down, why a knob resets at reboot, why the RAM was the - fault). A model swap is not an incident: it is an Era row and nothing else. -- **`AGENTS.md` carries operator-critical numbers only** — port, context - ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws. +- **The purpose test — before you write, and again before you keep:** would the + reader act differently because of this line? No → delete it. +- **Measurements are disposable.** A rate or throughput figure (t/s, tok/s, ms, + acceptance %, a bench ladder, a measured MiB) for a payload, runtime, or + layout that is not running now and not being compared now is destroyed, not + filed — not in an Eras row, not on a technique page, not "preserved for + history". AI moves faster than this vault: the number is wrong before anyone + reads it, and a wrong number in a runbook gets believed. Keep the **law** the + measurement taught; drop the measurement. +- **One fact, one owner**, every other file links with `[[Page]]`. The env file + beside the serve script is the authority for every live number; the endpoint + page is the authority for the shape and the wire contract. +- **One page per endpoint:** current shape, **Eras** (`window | payload | + runtime | replaced because | rollback` — the rollback is a commit, not a + number), the **Laws**, the wire contract, the files that own it. A swap edits + the shape block and appends one row; the retired era's own page is + **destroyed** to `.AGENTTRASH/`, never merged into anything. +- **`log.md` is retired** (2026-10-19: 2,380 lines destroyed). Git holds the + chronology. A causal law belongs to the page that owns the thing it governs, + or to *Key Lessons Learned* below if it is cross-cutting — never to a diary. - **Index pages are locators:** one row, one clause, no numbers. -- **A measured number is written once:** in the env-file header beside the - serve script that consumes it, and in the endpoint page. Every other page - cites it by link. -- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay - technique pages: they stop carrying a per-endpoint throughput table or a live - shape, but a campaign ledger that exists nowhere else is owned by the page that - measured it. The defect is two pages carrying one figure, not a number living on a - technique page — move a figure only when an endpoint page or an env header is its - home; otherwise leave it in place and name the era it measured. ## Model Config (harness) @@ -487,36 +488,10 @@ linked runbooks. 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]] -- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board - whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH - endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB - per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack - gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce - here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]]. - **When a GPU collective dies, read the runtime, not the framework.** The message that identified the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the failing HIP call with its return code. [[PCIe Link and Atomics]] -- **Layer split is the no-collective path, on one vendor's cards and across two — and - `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary - activation crosses PCIe once per forward pass, so it works wherever a collective cannot be - dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce), - and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But - `--override-tensor =` sets `has_tensor_overrides()`, a REQUIRED term of the - scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a - pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with - the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is - last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix. - Pin the spec-decode drafter to one device; on a split its home is the card holding the target's - last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] -- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.** - Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned — - the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in - the adapter or cable (card and port are both wider). For any engine that STREAMS weights across - it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per - token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM - (`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care. - [[x4-pair-experiment]] **ROCm / GPU** @@ -530,7 +505,7 @@ linked runbooks. the destination, then copy the contents. - **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache, - and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]] + and pinning the PLE tensor with `-ot` was slower than letting it page. [[Llama Cpp Serving Sizing]] - **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a @@ -545,9 +520,6 @@ linked runbooks. - **Flash-Next MTP heads do not load on any current loader** — they need full per-layer `blk..hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new loader support. -- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs - `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the - kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]] - **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in `/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale. @@ -602,13 +574,11 @@ linked runbooks. `LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines before adding flags. [[frank-dragoon]] -- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a - loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103 - (`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a - preference but the only door. Counting tensor types out of the GGUF KV section takes one python - read and settles a runtime argument in minutes. Build identity matters as much as the fork: the - 2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so - `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]] +- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a + file to a loader.** A non-standard tensor type in the KV section makes stock loaders refuse + the file at load, so a fork can be the only door rather than a preference. Counting tensor + types takes one python read and settles a runtime argument in minutes; build identity matters + as much as the fork, so pin the commit by SHA in any build script. [[GGUF Compatibility]] - **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type` (the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it @@ -617,12 +587,6 @@ linked runbooks. exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came back before believing a number — and kill by pid, never `pkill -f llama-server`. - [[LLM Bench Windows and Measurement]] -- Retired-runtime traps that can come back: **Kairic Edge** needs - `third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean` - removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so - re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS - patch, since stock binaries fail the draft load with a tensor-shape mismatch. **llama.cpp serving** @@ -635,10 +599,10 @@ linked runbooks. `--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is `ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`, `draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is - legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash` + legal and it wins on prompt-echo traffic, but `draft-mtp,draft-dflash` dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 - hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs + hybrid loses, because adding the ngram layer logs `recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and MTP verification depends on that rollback. Measure chains per architecture, never by analogy. [[frank-dragoon]] @@ -647,7 +611,7 @@ linked runbooks. declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut. - [[Llama Cpp Multi-Device Splits]] + [[Llama Cpp Serving Sizing]] - **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter wins. The MTP question is never "does the file carry the head" but what draft KV + buffers @@ -664,21 +628,20 @@ linked runbooks. re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the - control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at - 512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by - restarting the SERVICE through a transient `FRANK__ENV_FILE` drop-in, never a scratch - port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and - [[frank-zealot]]. + control that proves the knob, not the traffic, moved the number. The optimum is a property of + the backend, so re-find it after any device or payload change. When a bench IS ordered, A/B it + by restarting the SERVICE through a transient `FRANK__ENV_FILE` drop-in, never a scratch + port: one card holds one server. - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance by measurement. An affine fit reproduces the deployed point and overstates the gaps in between. - [[Llama Cpp Multi-Device Splits]] + [[Llama Cpp Serving Sizing]] - **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0 only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see: too short and it proposes on ordinary prose fragments, costing verification without acceptance. - [[Llama Cpp Multi-Device Splits]] + [[Llama Cpp Serving Sizing]] - **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml` @@ -793,17 +756,24 @@ linked runbooks. `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's - declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three; - ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account - accepts the gate. [[Model Swap and Rollback]] -- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six - read-only scouts worked the same pages: one classified era pages as stale *because* they - document retired payloads (that is an era page's content), one reported files that do not exist, - one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held - by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and - one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every - claimed duplicate, every claimed staleness and every quoted holder against the file and today's - fleet state before editing; a scout yield buys no evidence. + declared size, and check the LFS sha256 — the live `scripts/download-frank-*.sh` scripts do + all three; ad-hoc fetches tend not to. Gated repos 403 even with the host's HF token until + that account accepts the gate. [[Model Swap and Rollback]] +- **A verdict is a lead — the scout's and the tool's alike.** In the 2026-10-19 + documentation campaign six scouts worked the same pages and produced invented + bullet IDs, files that do not exist, "duplications" grep disproved, line + numbers past the end of the file they judged, and Eras tables with invented + payloads. The mechanical tool was worse: `scripts/port-census.py` built its + corpus from the directory containing the page under test, so each page was held + by *itself* and ten era pages reported "0 sole-record lines" — which reads as + "pure duplication, trash it". Nothing was cut on a verdict; every holder was + re-grepped and every tool re-measured before it was believed. +- **Never let a census decide what to keep.** That same campaign then concluded + the era pages were irreplaceable because each held 14–63 sole-record lines — + wrong in the opposite direction: those were throughput figures for payloads that + no longer run. Point of use is the only ownership test, the operator applies it, + and 53 pages plus the 2,380-line change diary were destroyed as a result. A tool + can say where a fact lives; it cannot say whether anything needs it. ## Runbook Index @@ -818,12 +788,13 @@ per-attempt page and no per-swap ledger anywhere else. | `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` | | `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` | | Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | -| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | +| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` | -Retired **endpoints** (not payloads) keep their own pages — `frank-medium`, -`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns -them; `serverAppConfigs-docs/index.md` is the locator for every page in the -vault. Retired scripts live in `old/` and `.AGENTTRASH/`. +There is no page for a retired endpoint and no change diary. A payload swap +appends one row to the live endpoint's **Eras** table; when the endpoint itself +dies, its page is destroyed. Rollback is a commit, not a document. +`serverAppConfigs-docs/index.md` is the locator for every page that survives; +retired scripts live in `old/` and `.AGENTTRASH/`. ## Why