docs(agents): compress the load-bearing lessons, repoint their owner links

Four bullets in Key Lessons Learned carried narrative a vault page owns: the
2026-09-20 board swap ([[Dual-R9700-Cooperation]]), the `--override-tensor`
gate ([[Llama Cpp Multi-Device Splits]]), the x4 link budget
([[x4-pair-experiment]]), and the silent-RAM incident plus the Prometheus
`--metrics` law ([[frank-archon]], [[frank-zealot]]). Each keeps its law, its
numbers and its error strings; the causal narration that duplicated a page is
what went. The RAM bullet linked to [[log]] — the incident is owned by
[[frank-archon]], so the link now names the owner.
This commit is contained in:
Frank
2026-10-19 05:08:29 -07:00
parent 483fd843b7
commit 293c25c50b
+32 -37
View File
@@ -485,13 +485,12 @@ linked runbooks.
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of
a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+`
and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never
the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8).
The collective choice matters: TheRock's RCCL builds lack gfx1201 device code
(`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all.
Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]].
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
@@ -499,26 +498,23 @@ linked runbooks.
- **Layer split is the no-collective path, on one vendor's cards and across two — and
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly
allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real
two-device graph. But `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a
REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets
`n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's
own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is
last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix.
Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build,
and on a split its home is the card holding the target's last layers.
[[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
`--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and
pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual
"pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4,
so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence
for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model
needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same
model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations
only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]]
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
(`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
[[x4-pair-experiment]]
**ROCm / GPU**
@@ -588,12 +584,12 @@ linked runbooks.
can boot before a human reaches it. [[frank-archon]]
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check
(60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said
"ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin
guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does
not move it, price in the RAM before the runtime — and remember a revert tested under the same
marginal hardware is itself confounded. [[log]]
Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over
8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was
the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is
data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM
before the runtime; a revert tested under the same marginal hardware is itself confounded.
[[frank-archon]]
**Runtime-specific guards**
@@ -724,11 +720,10 @@ linked runbooks.
`CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot`
sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry
reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the
alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert
that has fired for weeks. [[Observability]]
answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics`
natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from
`/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot
re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]]
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
drop-in sourcing the repo env) outlives the bench unless retired: a leftover