Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

3.8 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Expected Performance

This page estimates the speed of a 25-40B model split across two machines over ethernet. It uses the bandwidth math in Ethernet Bandwidth Math and the mixed-vendor measurements in Multi-Node Runtimes.

The Bandwidth Ceiling Is Not the Limit

A pipeline boundary moves only the hidden state. For a 27B model (hidden 5120), that is 10240 bytes per token per boundary in fp16. Even at 64 tokens per second with batch 64, the traffic is about 262 Mbit/s. 1 GbE can carry it. The real limits are latency, RPC overhead, and the slowest node.

The Petals paper measured this directly. Dropping from 1 Gbit/s to 100 Mbit/s changed throughput by 3 percent. Raising round-trip latency from 5 ms to 100 ms dropped throughput by 28 percent. Latency matters more than bandwidth for pipeline inference.

Real Benchmark Anchors

Experiment	Model	Network	Decode (t/s)
Mac Studio + DGX Spark (Metal+CUDA)	Qwen2.5-72B Q4	10 GbE	6
Mac Studio + DGX Spark (Metal+CUDA)	Qwen2.5-7B Q4	10 GbE	53
RTX 4070 + MI50 + 3050 (CUDA+ROCm)	Qwen3-VL-30B Q5	1 GbE	51.5 clean, 17.8 at 32k ctx
Two Strix Halos (ROCm RPC)	MiniMax-M2.5-139B Q8	USB4	15.35
AMD four-node Strix Halo (5 GbE)	Kimi K2.5 (1T)	5 GbE	8.30
Tesla V100 + R9700x2 (Vulkan, one box)	Qwen3.6-35B-A3B Q8	PCIe	62

Pattern: RPC decode runs about 2-5 times slower than the best single node. Prefill benefits from more compute across nodes. The 30B-class model over 1 GbE with warm cache reached 51.5 t/s on a short context and 17.8 t/s at 32k context.

Estimate for Qwen3.6-27B Q8, Two Nodes

Scenario: RTX 3090 (24 GB VRAM, NVIDIA desktop) plus Strix Halo NUC (Radeon 8060S, 30 GB physical RAM). The model is the dense 27B at Q8 (29-32 GB). Each node holds part of the weights and KV cache.

Link	Decode (t/s)	Prefill (t/s)	Notes
1 GbE	8-15	200-500	Latency-bound; per-token RPC round trip
2.5 GbE	10-18	250-600	Some headroom; latency still dominant
10 GbE	12-20	300-800	Closest to the published 10 GbE runs

The dense 27B decodes slower per token than the MoE 30B A3B class. Expect the lower half of the ranges at full context.

With Speculative Decoding

Speculative decoding (MTP, DFlash, or ngram-mod) raises effective decode. Tightwad measured 1.86x on a slow RPC pool. Estimate the effective rate at 1.4-1.8x the raw RPC decode:

Link	Raw decode (t/s)	With spec (t/s)
1 GbE	8-15	12-24
2.5 GbE	10-18	15-28
10 GbE	12-20	18-32

Context That Fits

The two real machines give about 51 GiB of usable memory combined (24 GiB VRAM plus about 27 GiB on the Strix Halo, which has 30 GiB physical RAM; the 96 GiB driver ceiling is not physical). A 29-32 GB model leaves about 20-22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85-100 KiB per token with q8_0 K and f16 V (measured on the large service at 602100 tokens = about 50 GB). That budget gives a total context pool of roughly 200k-260k tokens across both nodes.

Caveats

Every node must run the exact same llama.cpp build. Version skew hangs the handshake.
The slowest node sets the pace for every token.
Model load crosses the network once per start. Expect minutes, not seconds, for the split load.
These are estimates. The exact combination of an RTX 3090 desktop and a Strix Halo NUC has no published benchmark yet.

3.8 KiB Raw Blame History