dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: measured PP link math (H=5120 verified, 20.5 MB/ubatch wire), NIC inventory

Browse Source 
- Ethernet Bandwidth Math: new section 0 with live measurements
  (iperf 281-294 MB/s, ping RTT both paths, prefill 457.9 t/s at
  2.5GbE, per-ubatch wire 20.5 MB, 98k KV snapshot 3.45 GB blocking
  the slot 13.6 s, validated link-speed model: 386/458/505 t/s)
- System specs: NIC tables for both boxes; the 1 GbE USB-C adapter on
  the claw was silently routing the RPC path (unplugged 2026-08-17)
- PP Deployment: fix hidden-state size (10 KiB/direction, H=5120) and
  RTT-vs-bandwidth claim (decode RTT-bound; prefill+snapshot BW-bound)
This commit is contained in:
Frank
2026-08-17 05:06:41 -07:00
parent 968066ff23
commit 136ee080c9
4 changed files with 148 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Dankclaw System Specs.md
+13
View File
@@ -66,6 +66,19 @@ The NPU shares the 30 GiB unified pool. FLM allocates roughly half of system RAM
|------------|------|------|-------|------| |------------|------|------|-------|------|
| / (nvme0n1p2) | 1.9T | 661G | 1.1T | 38% | | / (nvme0n1p2) | 1.9T | 661G | 1.1T | 38% |
### Network
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carries the RPC path to Dankserver since 2026-08-17. Default route via 192.168.0.1. |
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. |
| docker0 | bridge | - | 172.17.0.1 | Usually down. |
The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at
least 2.5 GbE-capable. Whether it is 10G-capable is unverified — check
the device itself before buying 10G USB-C adapters; otherwise a direct
cable between claw and server is required for 10G.
### OS and Software ### OS and Software
| Property | Value | | Property | Value |
docs/Dankserver System Specs.md
+9
View File
@@ -41,6 +41,15 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
| / (nvme0n1p2) | 915G | 729G | 140G | 84% | | / (nvme0n1p2) | 915G | 729G | 140G | 84% |
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% | | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
### Network
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. |
| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). |
| wlp8s0 | WiFi | - | - | Unused (carrier down). |
| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. |
### OS and Software ### OS and Software
| Property | Value | | Property | Value |
docs/Pipeline Parallelism/Ethernet Bandwidth Math.md
+115 -3
View File
@@ -5,11 +5,121 @@ description: Bytes-per-token math for pipeline vs tensor parallelism, feasibilit
tags: tags:
- pipeline-parallelism - pipeline-parallelism
- research - research
timestamp: 2026-08-14 timestamp: 2026-08-17
--- ---
# Bandwidth Analysis: Multi-Node LLM Inference over Ethernet # Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17)
Everything in this section was measured on the live deployment
([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on
both ends. The theoretical tables below use H=4096/L=32; the real model
is larger (see constants), which raises the per-token bytes but not the
conclusions.
### 0.1 Verified constants (from the GGUF + env, not guessed)
| Constant | Value | Source |
| -------- | ----- | ------ |
| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) |
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-default-large.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured |
| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured |
| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured |
The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more
jitter (mdev 0.334 vs 0.211 ms).
### 0.2 Measured wire bytes (the number that matters)
Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**.
The *measured* bytes per position per direction are ~4x that:
| Direction | Measured | Raw hidden |
| --------- | -------- | ---------- |
| head -> worker | 39.8 KB/position | 10.0 KB |
| worker -> head | 38.4 KB/position | 10.0 KB |
The extra ~30 KB/position is the serialized RPC graph (per-position
nodes for all 45 worker blocks: attention, KV append, SSM ops)
plus protocol framing. It scales with position count just like the
payload, so it does not shrink away.
Per ubatch (256 positions) the measured crossing is **20.5 MB**
(10.4 MB head->worker + 10.1 MB worker->head).
### 0.3 Measured throughput and the validated model
Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE
path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition:
| Term | 2.5 GbE (measured anchor) |
| ---- | ------------------------- |
| wire per ubatch (20.5 MB) | 70 ms |
| compute + overhead per ubatch (residual) | 489 ms |
| ubatch total | 559 ms = 2.18 ms/token |
Using that residual and only swapping the wire time per link speed:
| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty |
| ---- | ----------- | ----------------- | ------------ | -------------------- |
| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% |
| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% |
| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% |
Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**.
### 0.4 The KV snapshot: where the link really saturates
On task release, `llama-server` saves the conversation to the prompt
cache. The KV for the 11 FA layers living on the worker GPU must be
copied **worker -> head** for that snapshot, and the slot is BLOCKED
while it runs (`PARALLEL=1` = one user queues behind it).
Measured for a 98,100-token conversation: 3,454 MB copied (model check:
98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg,
308 MB/s peak).
| Link | snapshot duration (slot blocked) |
| ---- | -------------------------------- |
| 1 GbE | ~31 s |
| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") |
| 10 GbE | ~5 s |
This is the only phase that runs the link at ~100% utilization for a
long time. Prefill runs at 12-26% of line rate in bursts (per ubatch)
and ~6-16% on average; decode runs at ~2% or less.
### 0.5 Decode (MTP verify step, 5 positions)
| Link | net time per step | share of 164 ms step | decode t/s |
| ---- | ----------------- | -------------------- | ---------- |
| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 |
| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) |
| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 |
Decode is latency-touched but bandwidth-trivial. Average wire rate
~19 Mbit/s in each direction.
### 0.6 Verdict for the 10 GbE USB-C adapter question
- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for
free: the claw's native 2.5 GbE `eno1` now carries the RPC path
(the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the
path, the RPC TCP re-attached on the new interface).
- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking.
It requires the L2 device at 192.168.0.1 to be 10G-capable, or a
direct cable between the two machines (dedicated link, second IP
pair, or a lower-metric static route).
- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable.
## 1. Bandwidth Math: Bytes per Token per Layer Boundary ## 1. Bandwidth Math: Bytes per Token per Layer Boundary
### 1.1 Pipeline Parallelism (PP) ### 1.1 Pipeline Parallelism (PP)
@@ -24,8 +134,10 @@ bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes]
**Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length. **Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length.
**For a 27B dense model (e.g., Qwen3.6-27B):** **For a 27B dense model (e.g., Qwen3.6-27B):**
- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures) - Hidden size: H = 4096 (standard for Llama/Qwen-style architectures).
- Layers: L = 32 (typical) Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120
(GGUF-verified); see section 0, which supersedes these theoretical
numbers with measurements.
- Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB` - Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok` - With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok`
docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md
+11 -4
View File
@@ -132,14 +132,21 @@ Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
llama-server b10433 ggml-rpc-server b10433 llama-server b10433 ggml-rpc-server b10433
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052 --rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
model file (31.5 GB) -d CUDA0 -c (tensor cache) model file (31.5 GB) -d CUDA0 -c (tensor cache)
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0..~31 [RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0-44 (45 of 65 blocks)
Strix: layers ~32..64 + lm_head Strix: layers 45-64 + lm_head
hidden state ~20 KiB crosses the link once per token, each direction hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
measured wire bytes ~40 KB/position/direction (serialized graph
dominates); 20.5 MB per ubatch both directions. Details:
[[Ethernet Bandwidth Math]] section 0.
``` ```
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on - **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
the worker, the rest on the head, in order. One hidden-state handoff per the worker, the rest on the head, in order. One hidden-state handoff per
token step. RTT-bound, not bandwidth-bound. token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5%
of step time). PREFILL and the prompt-cache KV snapshot ARE
bandwidth-bound: the link runs at line rate for the wire burst of every
ubatch and ~100% during the snapshot copy. Measured numbers:
[[Ethernet Bandwidth Math]] section 0.
- **Why the head is on Dankclaw:** the model file is already there; the - **Why the head is on Dankclaw:** the model file is already there; the
worker streams its tensor share at load (no 31.5 GB copy to the server). worker streams its tensor share at load (no 31.5 GB copy to the server).
- **Both sides must run the identical build** (b10433 / 9b05354ec). The - **Both sides must run the identical build** (b10433 / 9b05354ec). The