docs(agents): compress the load-bearing lessons, repoint their owner links

Four bullets in Key Lessons Learned carried narrative a vault page owns: the
2026-09-20 board swap ([[Dual-R9700-Cooperation]]), the `--override-tensor`
gate ([[Llama Cpp Multi-Device Splits]]), the x4 link budget
([[x4-pair-experiment]]), and the silent-RAM incident plus the Prometheus
`--metrics` law ([[frank-archon]], [[frank-zealot]]). Each keeps its law, its
numbers and its error strings; the causal narration that duplicated a page is
what went. The RAM bullet linked to [[log]] — the incident is owned by
[[frank-archon]], so the link now names the owner.
This commit is contained in:
Frank committed 2026-10-19 05:08:29 -07:00
1 parent 483fd843b7
commit 293c25c50b
1 file changed
+32 -37
+32 -37
View File
@@ -485,13 +485,12 @@ linked runbooks.
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]] nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of - **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+` whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8). per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
The collective choice matters: TheRock's RCCL builds lack gfx1201 device code gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
(`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all. here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified - **When a GPU collective dies, read the runtime, not the framework.** The message that identified
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
@@ -499,26 +498,23 @@ linked runbooks.
- **Layer split is the no-collective path, on one vendor's cards and across two — and - **Layer split is the no-collective path, on one vendor's cards and across two — and
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
two-device graph. But `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
`n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix. Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build, last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
and on a split its home is the card holding the target's last layers.
[[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.** - **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
"pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4, the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same (`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations [[x4-pair-experiment]]
only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]]
**ROCm / GPU** **ROCm / GPU**
@@ -588,12 +584,12 @@ linked runbooks.
can boot before a human reaches it. [[frank-archon]] can boot before a human reaches it. [[frank-archon]]
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation - **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600. repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over
(60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said 8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was
"ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is
guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM
not move it, price in the RAM before the runtime — and remember a revert tested under the same before the runtime; a revert tested under the same marginal hardware is itself confounded.
marginal hardware is itself confounded. [[log]] [[frank-archon]]
**Runtime-specific guards** **Runtime-specific guards**
@@ -724,11 +720,10 @@ linked runbooks.
`CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]] `CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it - **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health` the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot` answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics`
sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from
reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the `/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot
alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]]
that has fired for weeks. [[Observability]]
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The - **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env` sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
drop-in sourcing the repo env) outlives the bench unless retired: a leftover drop-in sourcing the repo env) outlives the bench unless retired: a leftover