diff --git a/docs/Dankclaw System Specs.md b/docs/Dankclaw System Specs.md index 90f4799..a0cbae1 100644 --- a/docs/Dankclaw System Specs.md +++ b/docs/Dankclaw System Specs.md @@ -66,6 +66,19 @@ The NPU shares the 30 GiB unified pool. FLM allocates roughly half of system RAM |------------|------|------|-------|------| | / (nvme0n1p2) | 1.9T | 661G | 1.1T | 38% | +### Network + +| Interface | Type | Speed | IP | Notes | +|-----------|------|-------|----|-------| +| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carries the RPC path to Dankserver since 2026-08-17. Default route via 192.168.0.1. | +| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. | +| docker0 | bridge | - | 172.17.0.1 | Usually down. | + +The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at +least 2.5 GbE-capable. Whether it is 10G-capable is unverified — check +the device itself before buying 10G USB-C adapters; otherwise a direct +cable between claw and server is required for 10G. + ### OS and Software | Property | Value | diff --git a/docs/Dankserver System Specs.md b/docs/Dankserver System Specs.md index af42a51..8fbb861 100644 --- a/docs/Dankserver System Specs.md +++ b/docs/Dankserver System Specs.md @@ -41,6 +41,15 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0 | / (nvme0n1p2) | 915G | 729G | 140G | 84% | | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% | +### Network + +| Interface | Type | Speed | IP | Notes | +|-----------|------|-------|----|-------| +| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. | +| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). | +| wlp8s0 | WiFi | - | - | Unused (carrier down). | +| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. | + ### OS and Software | Property | Value | diff --git a/docs/Pipeline Parallelism/Ethernet Bandwidth Math.md b/docs/Pipeline Parallelism/Ethernet Bandwidth Math.md index a260ab0..2bb5b93 100644 --- a/docs/Pipeline Parallelism/Ethernet Bandwidth Math.md +++ b/docs/Pipeline Parallelism/Ethernet Bandwidth Math.md @@ -5,11 +5,121 @@ description: Bytes-per-token math for pipeline vs tensor parallelism, feasibilit tags: - pipeline-parallelism - research -timestamp: 2026-08-14 +timestamp: 2026-08-17 --- # Bandwidth Analysis: Multi-Node LLM Inference over Ethernet +## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17) + +Everything in this section was measured on the live deployment +([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on +both ends. The theoretical tables below use H=4096/L=32; the real model +is larger (see constants), which raises the per-token bytes but not the +conclusions. + +### 0.1 Verified constants (from the GGUF + env, not guessed) + +| Constant | Value | Source | +| -------- | ----- | ------ | +| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) | +| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` | +| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` | +| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed | +| ubatch | 256 | `UBATCH_SIZE` in `frank-default-large.env` | +| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` | +| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s | +| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends | +| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured | +| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured | +| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured | + +The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more +jitter (mdev 0.334 vs 0.211 ms). + +### 0.2 Measured wire bytes (the number that matters) + +Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**. +The *measured* bytes per position per direction are ~4x that: + +| Direction | Measured | Raw hidden | +| --------- | -------- | ---------- | +| head -> worker | 39.8 KB/position | 10.0 KB | +| worker -> head | 38.4 KB/position | 10.0 KB | + +The extra ~30 KB/position is the serialized RPC graph (per-position +nodes for all 45 worker blocks: attention, KV append, SSM ops) +plus protocol framing. It scales with position count just like the +payload, so it does not shrink away. + +Per ubatch (256 positions) the measured crossing is **20.5 MB** +(10.4 MB head->worker + 10.1 MB worker->head). + +### 0.3 Measured throughput and the validated model + +Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE +path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition: + +| Term | 2.5 GbE (measured anchor) | +| ---- | ------------------------- | +| wire per ubatch (20.5 MB) | 70 ms | +| compute + overhead per ubatch (residual) | 489 ms | +| ubatch total | 559 ms = 2.18 ms/token | + +Using that residual and only swapping the wire time per link speed: + +| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty | +| ---- | ----------- | ----------------- | ------------ | -------------------- | +| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% | +| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% | +| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% | + +Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**. + +### 0.4 The KV snapshot: where the link really saturates + +On task release, `llama-server` saves the conversation to the prompt +cache. The KV for the 11 FA layers living on the worker GPU must be +copied **worker -> head** for that snapshot, and the slot is BLOCKED +while it runs (`PARALLEL=1` = one user queues behind it). + +Measured for a 98,100-token conversation: 3,454 MB copied (model check: +98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg, +308 MB/s peak). + +| Link | snapshot duration (slot blocked) | +| ---- | -------------------------------- | +| 1 GbE | ~31 s | +| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") | +| 10 GbE | ~5 s | + +This is the only phase that runs the link at ~100% utilization for a +long time. Prefill runs at 12-26% of line rate in bursts (per ubatch) +and ~6-16% on average; decode runs at ~2% or less. + +### 0.5 Decode (MTP verify step, 5 positions) + +| Link | net time per step | share of 164 ms step | decode t/s | +| ---- | ----------------- | -------------------- | ---------- | +| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 | +| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) | +| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 | + +Decode is latency-touched but bandwidth-trivial. Average wire rate +~19 Mbit/s in each direction. + +### 0.6 Verdict for the 10 GbE USB-C adapter question + +- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for + free: the claw's native 2.5 GbE `eno1` now carries the RPC path + (the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the + path, the RPC TCP re-attached on the new interface). +- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking. + It requires the L2 device at 192.168.0.1 to be 10G-capable, or a + direct cable between the two machines (dedicated link, second IP + pair, or a lower-metric static route). +- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable. + ## 1. Bandwidth Math: Bytes per Token per Layer Boundary ### 1.1 Pipeline Parallelism (PP) @@ -24,8 +134,10 @@ bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes] **Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length. **For a 27B dense model (e.g., Qwen3.6-27B):** -- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures) -- Layers: L = 32 (typical) +- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures). + Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120 + (GGUF-verified); see section 0, which supersedes these theoretical + numbers with measurements. - Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB` - With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok` diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md index 8fd60ca..d5290c7 100644 --- a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -132,14 +132,21 @@ Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235) llama-server b10433 ggml-rpc-server b10433 --rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052 model file (31.5 GB) -d CUDA0 -c (tensor cache) - [RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0..~31 - Strix: layers ~32..64 + lm_head - hidden state ~20 KiB crosses the link once per token, each direction + [RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0-44 (45 of 65 blocks) + Strix: layers 45-64 + lm_head + hidden state 10 KiB/position/direction (H=5120, GGUF-verified); + measured wire bytes ~40 KB/position/direction (serialized graph + dominates); 20.5 MB per ubatch both directions. Details: + [[Ethernet Bandwidth Math]] section 0. ``` - **Layer-sequential split** (true pipeline parallelism): layers 0..N run on the worker, the rest on the head, in order. One hidden-state handoff per - token step. RTT-bound, not bandwidth-bound. + token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5% + of step time). PREFILL and the prompt-cache KV snapshot ARE + bandwidth-bound: the link runs at line rate for the wire burst of every + ubatch and ~100% during the snapshot copy. Measured numbers: + [[Ethernet Bandwidth Math]] section 0. - **Why the head is on Dankclaw:** the model file is already there; the worker streams its tensor share at load (no 31.5 GB copy to the server). - **Both sides must run the identical build** (b10433 / 9b05354ec). The