From 293c25c50bb139fb67a9597e13bb9026f781d140 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 05:08:29 -0700 Subject: [PATCH] docs(agents): compress the load-bearing lessons, repoint their owner links MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four bullets in Key Lessons Learned carried narrative a vault page owns: the 2026-09-20 board swap ([[Dual-R9700-Cooperation]]), the `--override-tensor` gate ([[Llama Cpp Multi-Device Splits]]), the x4 link budget ([[x4-pair-experiment]]), and the silent-RAM incident plus the Prometheus `--metrics` law ([[frank-archon]], [[frank-zealot]]). Each keeps its law, its numbers and its error strings; the causal narration that duplicated a page is what went. The RAM bullet linked to [[log]] — the incident is owned by [[frank-archon]], so the link now names the owner. --- AGENTS.md | 69 ++++++++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 37 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b41f980..6fa63a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -485,13 +485,12 @@ linked runbooks. 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]] -- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of - a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+` - and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never - the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8). - The collective choice matters: TheRock's RCCL builds lack gfx1201 device code - (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all. - Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]]. +- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board + whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH + endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB + per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack + gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce + here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]]. - **When a GPU collective dies, read the runtime, not the framework.** The message that identified the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the @@ -499,26 +498,23 @@ linked runbooks. - **Layer split is the no-collective path, on one vendor's cards and across two — and `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary activation crosses PCIe once per forward pass, so it works wherever a collective cannot be - dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly - allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real - two-device graph. But `--override-tensor =` sets `has_tensor_overrides()`, a - REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets - `n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a - non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's - own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is - last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix. - Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build, - and on a split its home is the card holding the target's last layers. - [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] + dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce), + and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But + `--override-tensor =` sets `has_tensor_overrides()`, a REQUIRED term of the + scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a + pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with + the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is + last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix. + Pin the spec-decode drafter to one device; on a split its home is the card holding the target's + last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] - **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.** - Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and - pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual - "pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4, - so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence - for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model - needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same - model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations - only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]] + Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned — + the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in + the adapter or cable (card and port are both wider). For any engine that STREAMS weights across + it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per + token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM + (`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care. + [[x4-pair-experiment]] **ROCm / GPU** @@ -588,12 +584,12 @@ linked runbooks. can boot before a human reaches it. [[frank-archon]] - **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600. - Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check - (60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said - "ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin - guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does - not move it, price in the RAM before the runtime — and remember a revert tested under the same - marginal hardware is itself confounded. [[log]] + Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over + 8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was + the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is + data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM + before the runtime; a revert tested under the same marginal hardware is itself confounded. + [[frank-archon]] **Runtime-specific guards** @@ -724,11 +720,10 @@ linked runbooks. `CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]] - **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health` - answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot` - sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry - reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the - alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert - that has fired for weeks. [[Observability]] + answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics` + natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from + `/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot + re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]] - **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The sanctioned bench-window method (a transient `Environment=FRANK__ENV_FILE=/tmp/.env` drop-in sourcing the repo env) outlives the bench unless retired: a leftover