docs(vault): documentation exists for point of use — destroy the ledgers (−13.7k lines) #31
@@ -47,7 +47,7 @@ per attempt and no separate swap-history file.
|
||||
The managed skill library is distilled there one page per skill-cluster:
|
||||
`LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
|
||||
`Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
|
||||
pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
|
||||
pages. There is no `log.md`: the change diary was destroyed 2026-10-19
|
||||
- `tests/` — bats suite + Playwright e2e
|
||||
|
||||
## Git Workflow — trunk-based (fleet law, aligned 2026-10-18)
|
||||
@@ -318,35 +318,36 @@ the next), so never report request #1.
|
||||
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
|
||||
`mask` symlinks to `/dev/null` and is safe/reversible.
|
||||
|
||||
## Documentation Policy (fleet law, 2026-10-18)
|
||||
## Documentation Policy (fleet law, 2026-10-19)
|
||||
|
||||
**One fact, one owner.** A page owns exactly one thing — an endpoint, a
|
||||
service, or a technique — and every other file links to it with `[[Page]]`.
|
||||
Restating a shape, a digest, a port list, or a measured number in a second
|
||||
file is a defect: it is the next thing to go stale.
|
||||
**Documentation exists for point of use.** A page earns its bytes only if it
|
||||
helps whoever performs the next task — the shape to deploy, the flag that must
|
||||
be passed, the failure signature to recognise, the rollback to run. It is not a
|
||||
lab notebook and not a ledger. **Losing information is an acceptable, usually
|
||||
correct outcome.** Never let an orphan census ("this is the only copy of that
|
||||
fact") decide what to keep: sole custody of a useless fact is still useless.
|
||||
|
||||
- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
|
||||
— current shape, an **Eras** table (`window | payload | runtime | shape |
|
||||
replaced because | rollback`), the **Laws** the endpoint taught, the wire
|
||||
contract, and the files that own it. A payload swap edits the shape block
|
||||
and appends one Era row. It never creates a page, and the retired era's
|
||||
page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
|
||||
table.
|
||||
- **`log.md` records incidents and laws** — anything with a causal story
|
||||
(why the host went down, why a knob resets at reboot, why the RAM was the
|
||||
fault). A model swap is not an incident: it is an Era row and nothing else.
|
||||
- **`AGENTS.md` carries operator-critical numbers only** — port, context
|
||||
ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
|
||||
- **The purpose test — before you write, and again before you keep:** would the
|
||||
reader act differently because of this line? No → delete it.
|
||||
- **Measurements are disposable.** A rate or throughput figure (t/s, tok/s, ms,
|
||||
acceptance %, a bench ladder, a measured MiB) for a payload, runtime, or
|
||||
layout that is not running now and not being compared now is destroyed, not
|
||||
filed — not in an Eras row, not on a technique page, not "preserved for
|
||||
history". AI moves faster than this vault: the number is wrong before anyone
|
||||
reads it, and a wrong number in a runbook gets believed. Keep the **law** the
|
||||
measurement taught; drop the measurement.
|
||||
- **One fact, one owner**, every other file links with `[[Page]]`. The env file
|
||||
beside the serve script is the authority for every live number; the endpoint
|
||||
page is the authority for the shape and the wire contract.
|
||||
- **One page per endpoint:** current shape, **Eras** (`window | payload |
|
||||
runtime | replaced because | rollback` — the rollback is a commit, not a
|
||||
number), the **Laws**, the wire contract, the files that own it. A swap edits
|
||||
the shape block and appends one row; the retired era's own page is
|
||||
**destroyed** to `.AGENTTRASH/`, never merged into anything.
|
||||
- **`log.md` is retired** (2026-10-19: 2,380 lines destroyed). Git holds the
|
||||
chronology. A causal law belongs to the page that owns the thing it governs,
|
||||
or to *Key Lessons Learned* below if it is cross-cutting — never to a diary.
|
||||
- **Index pages are locators:** one row, one clause, no numbers.
|
||||
- **A measured number is written once:** in the env-file header beside the
|
||||
serve script that consumes it, and in the endpoint page. Every other page
|
||||
cites it by link.
|
||||
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
|
||||
technique pages: they stop carrying a per-endpoint throughput table or a live
|
||||
shape, but a campaign ledger that exists nowhere else is owned by the page that
|
||||
measured it. The defect is two pages carrying one figure, not a number living on a
|
||||
technique page — move a figure only when an endpoint page or an env header is its
|
||||
home; otherwise leave it in place and name the era it measured.
|
||||
|
||||
## Model Config (harness)
|
||||
|
||||
@@ -487,36 +488,10 @@ linked runbooks.
|
||||
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
|
||||
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
|
||||
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
|
||||
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
|
||||
whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
|
||||
endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
|
||||
per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
|
||||
gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
|
||||
here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
|
||||
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
|
||||
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
|
||||
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
|
||||
failing HIP call with its return code. [[PCIe Link and Atomics]]
|
||||
- **Layer split is the no-collective path, on one vendor's cards and across two — and
|
||||
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
|
||||
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
|
||||
dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
|
||||
and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
|
||||
`--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
|
||||
scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
|
||||
pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
|
||||
the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
|
||||
last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
|
||||
Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
|
||||
last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
|
||||
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
|
||||
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
|
||||
the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
|
||||
the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
|
||||
it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
|
||||
token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
|
||||
(`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
|
||||
[[x4-pair-experiment]]
|
||||
|
||||
|
||||
**ROCm / GPU**
|
||||
@@ -530,7 +505,7 @@ linked runbooks.
|
||||
the destination, then copy the contents.
|
||||
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
|
||||
anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
|
||||
and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]]
|
||||
and pinning the PLE tensor with `-ot` was slower than letting it page. [[Llama Cpp Serving Sizing]]
|
||||
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
|
||||
with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
|
||||
`1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
|
||||
@@ -545,9 +520,6 @@ linked runbooks.
|
||||
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
|
||||
`blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
|
||||
loader support.
|
||||
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs
|
||||
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the
|
||||
kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]]
|
||||
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
|
||||
Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
|
||||
`/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
|
||||
@@ -602,13 +574,11 @@ linked runbooks.
|
||||
`LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
|
||||
any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
|
||||
before adding flags. [[frank-dragoon]]
|
||||
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a
|
||||
loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103
|
||||
(`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a
|
||||
preference but the only door. Counting tensor types out of the GGUF KV section takes one python
|
||||
read and settles a runtime argument in minutes. Build identity matters as much as the fork: the
|
||||
2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so
|
||||
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]]
|
||||
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a
|
||||
file to a loader.** A non-standard tensor type in the KV section makes stock loaders refuse
|
||||
the file at load, so a fork can be the only door rather than a preference. Counting tensor
|
||||
types takes one python read and settles a runtime argument in minutes; build identity matters
|
||||
as much as the fork, so pin the commit by SHA in any build script. [[GGUF Compatibility]]
|
||||
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
|
||||
(the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
|
||||
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
|
||||
@@ -617,12 +587,6 @@ linked runbooks.
|
||||
exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
|
||||
every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
|
||||
back before believing a number — and kill by pid, never `pkill -f llama-server`.
|
||||
[[LLM Bench Windows and Measurement]]
|
||||
- Retired-runtime traps that can come back: **Kairic Edge** needs
|
||||
`third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean`
|
||||
removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so
|
||||
re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS
|
||||
patch, since stock binaries fail the draft load with a tensor-shape mismatch.
|
||||
|
||||
**llama.cpp serving**
|
||||
|
||||
@@ -635,10 +599,10 @@ linked runbooks.
|
||||
`--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
|
||||
`ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
|
||||
`draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
|
||||
legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash`
|
||||
legal and it wins on prompt-echo traffic, but `draft-mtp,draft-dflash`
|
||||
dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
|
||||
chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
|
||||
hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs
|
||||
hybrid loses, because adding the ngram layer logs
|
||||
`recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
|
||||
MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
|
||||
[[frank-dragoon]]
|
||||
@@ -647,7 +611,7 @@ linked runbooks.
|
||||
declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
|
||||
first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
|
||||
drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
|
||||
[[Llama Cpp Multi-Device Splits]]
|
||||
[[Llama Cpp Serving Sizing]]
|
||||
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
|
||||
compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
|
||||
wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
|
||||
@@ -664,21 +628,20 @@ linked runbooks.
|
||||
re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
|
||||
large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
|
||||
model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
|
||||
control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at
|
||||
512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by
|
||||
restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
|
||||
port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and
|
||||
[[frank-zealot]].
|
||||
control that proves the knob, not the traffic, moved the number. The optimum is a property of
|
||||
the backend, so re-find it after any device or payload change. When a bench IS ordered, A/B it
|
||||
by restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
|
||||
port: one card holds one server.
|
||||
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
|
||||
boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
|
||||
by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
|
||||
[[Llama Cpp Multi-Device Splits]]
|
||||
[[Llama Cpp Serving Sizing]]
|
||||
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
|
||||
matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
|
||||
only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
|
||||
costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
|
||||
too short and it proposes on ordinary prose fragments, costing verification without acceptance.
|
||||
[[Llama Cpp Multi-Device Splits]]
|
||||
[[Llama Cpp Serving Sizing]]
|
||||
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
|
||||
the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
|
||||
prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
|
||||
@@ -793,17 +756,24 @@ linked runbooks.
|
||||
`curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
|
||||
repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
|
||||
seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
|
||||
declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three;
|
||||
ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account
|
||||
accepts the gate. [[Model Swap and Rollback]]
|
||||
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six
|
||||
read-only scouts worked the same pages: one classified era pages as stale *because* they
|
||||
document retired payloads (that is an era page's content), one reported files that do not exist,
|
||||
one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held
|
||||
by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and
|
||||
one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every
|
||||
claimed duplicate, every claimed staleness and every quoted holder against the file and today's
|
||||
fleet state before editing; a scout yield buys no evidence.
|
||||
declared size, and check the LFS sha256 — the live `scripts/download-frank-*.sh` scripts do
|
||||
all three; ad-hoc fetches tend not to. Gated repos 403 even with the host's HF token until
|
||||
that account accepts the gate. [[Model Swap and Rollback]]
|
||||
- **A verdict is a lead — the scout's and the tool's alike.** In the 2026-10-19
|
||||
documentation campaign six scouts worked the same pages and produced invented
|
||||
bullet IDs, files that do not exist, "duplications" grep disproved, line
|
||||
numbers past the end of the file they judged, and Eras tables with invented
|
||||
payloads. The mechanical tool was worse: `scripts/port-census.py` built its
|
||||
corpus from the directory containing the page under test, so each page was held
|
||||
by *itself* and ten era pages reported "0 sole-record lines" — which reads as
|
||||
"pure duplication, trash it". Nothing was cut on a verdict; every holder was
|
||||
re-grepped and every tool re-measured before it was believed.
|
||||
- **Never let a census decide what to keep.** That same campaign then concluded
|
||||
the era pages were irreplaceable because each held 14–63 sole-record lines —
|
||||
wrong in the opposite direction: those were throughput figures for payloads that
|
||||
no longer run. Point of use is the only ownership test, the operator applies it,
|
||||
and 53 pages plus the 2,380-line change diary were destroyed as a result. A tool
|
||||
can say where a fact lives; it cannot say whether anything needs it.
|
||||
|
||||
## Runbook Index
|
||||
|
||||
@@ -818,12 +788,13 @@ per-attempt page and no per-swap ledger anywhere else.
|
||||
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
|
||||
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
|
||||
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
|
||||
|
||||
Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
|
||||
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
|
||||
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
|
||||
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.
|
||||
There is no page for a retired endpoint and no change diary. A payload swap
|
||||
appends one row to the live endpoint's **Eras** table; when the endpoint itself
|
||||
dies, its page is destroyed. Rollback is a commit, not a document.
|
||||
`serverAppConfigs-docs/index.md` is the locator for every page that survives;
|
||||
retired scripts live in `old/` and `.AGENTTRASH/`.
|
||||
|
||||
## Why
|
||||
|
||||
|
||||
Reference in New Issue
Block a user