docs(agents): compress the load-bearing lessons, repoint their owner links
Four bullets in Key Lessons Learned carried narrative a vault page owns: the 2026-09-20 board swap ([[Dual-R9700-Cooperation]]), the `--override-tensor` gate ([[Llama Cpp Multi-Device Splits]]), the x4 link budget ([[x4-pair-experiment]]), and the silent-RAM incident plus the Prometheus `--metrics` law ([[frank-archon]], [[frank-zealot]]). Each keeps its law, its numbers and its error strings; the causal narration that duplicated a page is what went. The RAM bullet linked to [[log]] — the incident is owned by [[frank-archon]], so the link now names the owner.
This commit is contained in:
@@ -485,13 +485,12 @@ linked runbooks.
|
||||
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
|
||||
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
|
||||
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
|
||||
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of
|
||||
a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+`
|
||||
and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never
|
||||
the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8).
|
||||
The collective choice matters: TheRock's RCCL builds lack gfx1201 device code
|
||||
(`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all.
|
||||
Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]].
|
||||
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
|
||||
whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
|
||||
endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
|
||||
per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
|
||||
gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
|
||||
here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
|
||||
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
|
||||
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
|
||||
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
|
||||
@@ -499,26 +498,23 @@ linked runbooks.
|
||||
- **Layer split is the no-collective path, on one vendor's cards and across two — and
|
||||
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
|
||||
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
|
||||
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly
|
||||
allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real
|
||||
two-device graph. But `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a
|
||||
REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets
|
||||
`n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a
|
||||
non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's
|
||||
own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is
|
||||
last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix.
|
||||
Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build,
|
||||
and on a split its home is the card holding the target's last layers.
|
||||
[[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
|
||||
dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
|
||||
and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
|
||||
`--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
|
||||
scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
|
||||
pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
|
||||
the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
|
||||
last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
|
||||
Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
|
||||
last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
|
||||
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
|
||||
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and
|
||||
pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual
|
||||
"pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4,
|
||||
so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence
|
||||
for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model
|
||||
needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same
|
||||
model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations
|
||||
only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]]
|
||||
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
|
||||
the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
|
||||
the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
|
||||
it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
|
||||
token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
|
||||
(`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
|
||||
[[x4-pair-experiment]]
|
||||
|
||||
|
||||
**ROCm / GPU**
|
||||
@@ -588,12 +584,12 @@ linked runbooks.
|
||||
can boot before a human reaches it. [[frank-archon]]
|
||||
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
|
||||
repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
|
||||
Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check
|
||||
(60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said
|
||||
"ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin
|
||||
guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does
|
||||
not move it, price in the RAM before the runtime — and remember a revert tested under the same
|
||||
marginal hardware is itself confounded. [[log]]
|
||||
Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over
|
||||
8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was
|
||||
the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is
|
||||
data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM
|
||||
before the runtime; a revert tested under the same marginal hardware is itself confounded.
|
||||
[[frank-archon]]
|
||||
|
||||
**Runtime-specific guards**
|
||||
|
||||
@@ -724,11 +720,10 @@ linked runbooks.
|
||||
`CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
|
||||
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
|
||||
the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
|
||||
answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot`
|
||||
sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry
|
||||
reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the
|
||||
alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert
|
||||
that has fired for weeks. [[Observability]]
|
||||
answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics`
|
||||
natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from
|
||||
`/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot
|
||||
re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]]
|
||||
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
|
||||
sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
|
||||
drop-in sourcing the repo env) outlives the bench unless retired: a leftover
|
||||
|
||||
Reference in New Issue
Block a user