Files

T

Frank 219a097b39 docs(vault): one page per endpoint, forever -- 45 era pages fold into three

Every payload swap a live endpoint has ever run is now one row in that
endpoint's Eras table, with its rollback path in the same row. No per-attempt
page, no per-swap ledger, nowhere else.

- Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md
  (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork
  on the 3090). Each carries Current shape, Eras, Laws, wire contract, files.
- Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the
  pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the
  benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras),
  the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus
  frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing.
- Rewired: every wikilink and markdown-path reference to a folded page points
  at the endpoint page instead; the three Research see-also lists and the
  ubatch sweep's runbook link included. No anchored links existed in the set.
- Regenerated both index pages as pure locators -- one row, one clause, no
  numbers -- so the docs_completeness locator law holds for all 153 pages.

tests/docs_completeness.bats green: index links every page, no unresolved
links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored,
and git history holds every byte).

2026-10-19 00:49:03 -07:00

2.8 KiB

Raw Blame History

Pipeline Parallelism

This section covers running one large model across two machines. It focuses on pipeline parallelism (layer split) over ethernet. The research answers what speeds are possible and which tools work. The sample scripts show a two-node Qwen3.6-27B Q8 split for reading only.

Terminology

Pipeline parallelism (PP) splits model layers across machines. Each machine computes its layers and passes hidden states to the next machine. Tensor parallelism (TP) splits each layer across machines and requires an all-reduce per layer. PP is the only realistic option over ethernet.

Speculative decoding is a separate feature. It uses a small draft model (or n-grams) to propose tokens that the main model verifies in one pass. It can run together with PP in llama.cpp.

The Two Runtimes

graph LR
    subgraph llamaRPC[llama.cpp RPC]
        A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
    end
    subgraph vllmPP[vLLM PP]
        B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
    end

llama.cpp RPC — the working, production-capable path for two heterogeneous nodes over ethernet.
vLLM Pipeline Parallelism — supported but heavier; needs Ray or multiprocessing and does not compose with speculative decoding yet.

What the Research Found

Ethernet Bandwidth Math — a PP boundary moves only the hidden state (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of tokens per second on paper; the real limit is latency and RPC overhead.
Mixed-Vendor Experiments — NVIDIA + Strix Halo multi-node inference works today via llama.cpp RPC with separate CUDA and ROCm builds. The exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with published numbers yet.
Qwen3.6 27B Spec Methods — Q8 GGUF variants, and which speculative methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
Expected Performance — estimated tokens per second for a 27B Q8 split across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.

Deployment

Qwen3.8 PP Deployment — the real Qwen3.8-27B split: head on Dankclaw (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in scripts/, morning-of test runbook. Verified gotchas: no --device with --rpc, default split is proportional to reported free memory (pin 1,1), the iGPU side defaults to the ROCm backend.

Sample Scripts

Sample Script Set — a two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for the largest context that fits. For educational reading only. Do not run it in production.

2.8 KiB Raw Blame History