Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
98 lines
3.8 KiB
Markdown
98 lines
3.8 KiB
Markdown
---
|
|
type: reference
|
|
title: Expected Performance
|
|
description: "Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on."
|
|
tags:
|
|
- pipeline-parallelism
|
|
- performance
|
|
- estimate
|
|
timestamp: 2026-08-14
|
|
---
|
|
|
|
# Expected Performance
|
|
|
|
This page estimates the speed of a 25-40B model split across two machines
|
|
over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
|
|
and the mixed-vendor measurements in [[Multi-Node Runtimes]].
|
|
|
|
## The Bandwidth Ceiling Is Not the Limit
|
|
|
|
A pipeline boundary moves only the hidden state. For a 27B model (hidden
|
|
5120), that is 10240 bytes per token per boundary in fp16. Even at 64
|
|
tokens per second with batch 64, the traffic is about 262 Mbit/s. 1 GbE can
|
|
carry it. The real limits are latency, RPC overhead, and the slowest node.
|
|
|
|
The Petals paper measured this directly. Dropping from 1 Gbit/s to
|
|
100 Mbit/s changed throughput by 3 percent. Raising round-trip latency from
|
|
5 ms to 100 ms dropped throughput by 28 percent. Latency matters more than
|
|
bandwidth for pipeline inference.
|
|
|
|
## Real Benchmark Anchors
|
|
|
|
| Experiment | Model | Network | Decode (t/s) |
|
|
| ---------- | ----- | ------- | ------------ |
|
|
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-72B Q4 | 10 GbE | 6 |
|
|
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-7B Q4 | 10 GbE | 53 |
|
|
| RTX 4070 + MI50 + 3050 (CUDA+ROCm) | Qwen3-VL-30B Q5 | 1 GbE | 51.5 clean, 17.8 at 32k ctx |
|
|
| Two Strix Halos (ROCm RPC) | MiniMax-M2.5-139B Q8 | USB4 | 15.35 |
|
|
| AMD four-node Strix Halo (5 GbE) | Kimi K2.5 (1T) | 5 GbE | 8.30 |
|
|
| Tesla V100 + R9700x2 (Vulkan, one box) | Qwen3.6-35B-A3B Q8 | PCIe | 62 |
|
|
|
|
Pattern: RPC decode runs about 2-5 times slower than the best single node.
|
|
Prefill benefits from more compute across nodes. The 30B-class model over
|
|
1 GbE with warm cache reached 51.5 t/s on a short context and 17.8 t/s at
|
|
32k context.
|
|
|
|
## Estimate for Qwen3.6-27B Q8, Two Nodes
|
|
|
|
Scenario: RTX 3090 (24 GB VRAM, NVIDIA desktop) plus Strix Halo NUC
|
|
(Radeon 8060S, 30 GB physical RAM). The model is the dense 27B at Q8
|
|
(29-32 GB). Each node holds part of the weights and KV cache.
|
|
|
|
| Link | Decode (t/s) | Prefill (t/s) | Notes |
|
|
| ---- | ------------ | ------------- | ----- |
|
|
| 1 GbE | 8-15 | 200-500 | Latency-bound; per-token RPC round trip |
|
|
| 2.5 GbE | 10-18 | 250-600 | Some headroom; latency still dominant |
|
|
| 10 GbE | 12-20 | 300-800 | Closest to the published 10 GbE runs |
|
|
|
|
The dense 27B decodes slower per token than the MoE 30B A3B class. Expect
|
|
the lower half of the ranges at full context.
|
|
|
|
### With Speculative Decoding
|
|
|
|
Speculative decoding (MTP, DFlash, or ngram-mod) raises effective decode.
|
|
Tightwad measured 1.86x on a slow RPC pool. Estimate the effective rate at
|
|
1.4-1.8x the raw RPC decode:
|
|
|
|
| Link | Raw decode (t/s) | With spec (t/s) |
|
|
| ---- | ---------------- | --------------- |
|
|
| 1 GbE | 8-15 | 12-24 |
|
|
| 2.5 GbE | 10-18 | 15-28 |
|
|
| 10 GbE | 12-20 | 18-32 |
|
|
|
|
## Context That Fits
|
|
|
|
The two real machines give about 51 GiB of usable memory combined
|
|
(24 GiB VRAM plus about 27 GiB on the Strix Halo, which has 30 GiB physical
|
|
RAM; the 96 GiB driver ceiling is not physical). A 29-32 GB model leaves
|
|
about 20-22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85-100
|
|
KiB per token with q8_0 K and f16 V (measured on the large service at
|
|
602100 tokens = about 50 GB). That budget gives a total context pool of
|
|
roughly 200k-260k tokens across both nodes.
|
|
|
|
## Caveats
|
|
|
|
- Every node must run the exact same llama.cpp build. Version skew hangs
|
|
the handshake.
|
|
- The slowest node sets the pace for every token.
|
|
- Model load crosses the network once per start. Expect minutes, not
|
|
seconds, for the split load.
|
|
- These are estimates. The exact combination of an RTX 3090 desktop and a
|
|
Strix Halo NUC has no published benchmark yet.
|
|
|
|
## See Also
|
|
|
|
- [[Ethernet Bandwidth Math]]
|
|
- [[Multi-Node Runtimes]]
|
|
- [[Sample Script Set]]
|