Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
174 lines
8.2 KiB
Markdown
174 lines
8.2 KiB
Markdown
---
|
||
type: reference
|
||
title: Heterogeneous Split Triage
|
||
description: "Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys."
|
||
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# Heterogeneous Split Triage
|
||
|
||
Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
|
||
the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
|
||
across device A and device B" with arithmetic instead of enthusiasm.
|
||
|
||
## 1. Characterise the link (bytes vs latency)
|
||
|
||
```bash
|
||
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:" # capability vs actual
|
||
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
|
||
```
|
||
|
||
Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
|
||
upshifts under load. Measure while the device is busy, or you size the design
|
||
against the wrong number. Per-hop cost against the hidden state
|
||
(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
|
||
Gen4 x4.
|
||
|
||
- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
|
||
for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
|
||
never the constraint.**
|
||
- The real budget line is **hop COUNT**: every device boundary is a
|
||
host-side sync. A runtime that splits per *op* hits ~100 splits/token and
|
||
pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
|
||
assume.
|
||
- **Design rule: partition by where the weights live, never by where the
|
||
activations flow.** Resident weights, activations crossing.
|
||
|
||
## 2. Get the workload's real bottleneck (KV is usually it)
|
||
|
||
```
|
||
KV/token = kv_bytes_per_position × context
|
||
```
|
||
|
||
Attention reads the whole cached context every token, so KV/token scales
|
||
with context — compare it against expert/weight traffic per token and total
|
||
token time. At long context on a hybrid arch (few attention layers, many
|
||
positions), **KV dominates**. Consequences:
|
||
|
||
- The highest-leverage placement is **KV cache on the fastest-memory device,
|
||
together with its attention compute** — never KV alone: separating them
|
||
puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
|
||
hundreds of ms/token).
|
||
- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
|
||
attention_trunk ≤ VRAM`.
|
||
|
||
## 3. What placement never buys
|
||
|
||
Prefill is compute-bound and expert weights are read once per batch, so
|
||
moving them to faster memory buys nothing while adding a serial stage.
|
||
Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
|
||
(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
|
||
GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
|
||
`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
|
||
decode gain to within ~2 points — usable for decode projections, and it
|
||
assumes sequential stages (true at batch 1; no pipelining).
|
||
|
||
## 4. Is the second device even a separate pool?
|
||
|
||
On a UMA APU, check before assuming additivity:
|
||
|
||
```bash
|
||
cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small)
|
||
cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
|
||
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
|
||
```
|
||
|
||
Read on Dankclaw 2026-10-18: `mem_info_vram_total` = 2147483648 (2 GiB of
|
||
dedicated carve), `mem_info_gtt_total` = 122402963456 (114 GiB) — the aperture is
|
||
the whole budget, the carve is noise, and the OS sees 121 GiB of it
|
||
([[Dankclaw System Specs]]).
|
||
|
||
A large GTT window means the "GPU memory" is the *same* LPDDR under one
|
||
memory controller: adding a stage there adds a compute engine, not capacity
|
||
or bandwidth. Only the discrete card's VRAM is additive. Fencing more
|
||
(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
|
||
|
||
## 5. Speculative decoding as the "cheap" split
|
||
|
||
A drafter on the *other* device is bandwidth-free (token ids per round, tens
|
||
of bytes), so it is tempting. Before proposing it:
|
||
|
||
- the drafter needs the target's **exact tokenizer** for a plain draft model
|
||
(compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
|
||
must be a head **trained for that target** (EAGLE/DFlash/MTP are
|
||
conditioned on the target's hidden states and are not portable between
|
||
models);
|
||
- a plain drafter costs **K sequential forwards per round**, so it must be
|
||
10–50× faster than the target. Break-even with `r = t_draft/t_target`:
|
||
`rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
|
||
(a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
|
||
- compare against the **incumbent**, not the no-draft baseline: a target
|
||
already running its own MTP head is a high bar;
|
||
- check the runtime accepts an external drafter at all — closed engines may
|
||
enumerate a fixed drafter set and 400 anything else (capability gates
|
||
keyed on the loaded checkpoint). That is a design boundary, not a config
|
||
gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
|
||
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
|
||
|
||
The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device`
|
||
(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268),
|
||
`--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v`
|
||
(`-ctkd`/`-ctvd`, 4111/4124) all exist in
|
||
`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp),
|
||
read 2026-10-18).
|
||
|
||
## 6. Expressing a split (llama.cpp, the usual vehicle)
|
||
|
||
```bash
|
||
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest)
|
||
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (KV co-location)
|
||
```
|
||
|
||
- buffer types match by name at load (`ggml_backend_buft_name`), so device
|
||
names work as `-ot` values;
|
||
- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`,
|
||
`common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`;
|
||
- several backends in one binary is fine (CUDA + Vulkan is the practical
|
||
pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs
|
||
do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the
|
||
existing build's CMakeCache tells you what needs adding;
|
||
- contiguous ranges are the WRONG split when KV bandwidth dominates: they
|
||
drag half the attention layers (and their KV) back to the slow pool.
|
||
|
||
## 7. Hazards checklist
|
||
|
||
- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
|
||
are read from; moving them is pure loss.
|
||
- Keep `mmap` on for models with huge embedding tables (anonymous memory
|
||
gets the process cgroup-killed).
|
||
- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
|
||
— check before assuming optimisations apply.
|
||
- A brand-new backend on the second device may not implement the arch's ops
|
||
at all: make the first measurement a **baseline of the runtime on that
|
||
device alone** — a split's arithmetic is worthless if the baseline is
|
||
worse than the incumbent engine.
|
||
- A split consumes the fast device entirely — check what else it was
|
||
serving before trading it away.
|
||
- Benchmark discipline on production hosts: stop and mask first
|
||
([[Fleet Ops Playbook]]).
|
||
|
||
## 8. Verdict shape
|
||
|
||
Rank and report; prefer the boring answer when it wins:
|
||
|
||
1. **Two independent endpoints** (zero shared traffic) — often the real
|
||
answer to "what's a useful combination", especially over a narrow link.
|
||
2. **One process, weights-resident, KV co-located with attention on the
|
||
fast pool** — promising and quantifiable (+% decode at long context);
|
||
needs a dual-backend build and a baseline measurement.
|
||
3. **Hot-expert cache / per-token weight streaming** — affordable only if
|
||
the miss set per token is small (a few experts; check
|
||
`misses × bytes × tok/s` against link bandwidth) and overlapped; injects
|
||
PCIe latency into the critical path; no mainstream runtime implements a
|
||
dynamic expert cache.
|
||
4. List explicitly what does NOT help: TP/collectives across unequal
|
||
devices, KV on the far device, model loading over the link,
|
||
prefill-motivated placements.
|
||
|
||
## Related
|
||
|
||
- [[PCIe Link and Atomics]] — the link and collective walls underneath
|
||
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
|
||
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
|