Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

2.9 KiB

Raw Blame History

Pipeline Parallelism

This section covers running one large model across two machines. It focuses on pipeline parallelism (layer split) over ethernet. The research answers what speeds are possible and which tools work. The sample scripts show a two-node Qwen3.6-27B Q8 split for reading only.

Terminology

Pipeline parallelism (PP) splits model layers across machines. Each machine computes its layers and passes hidden states to the next machine. Tensor parallelism (TP) splits each layer across machines and requires an all-reduce per layer. PP is the only realistic option over ethernet.

Speculative decoding is a separate feature. It uses a small draft model (or n-grams) to propose tokens that the main model verifies in one pass. It can run together with PP in llama.cpp.

The Two Runtimes

graph LR
    subgraph llamaRPC[llama.cpp RPC]
        A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
    end
    subgraph vllmPP[vLLM PP]
        B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
    end

heterogeneous nodes over ethernet. multiprocessing and does not compose with speculative decoding yet.

What the Research Found

Ethernet Bandwidth Math — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of tokens per second on paper; the real limit is latency and RPC overhead. works today via llama.cpp RPC with separate CUDA and ROCm builds. The exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with published numbers yet. methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
Expected Performance — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.

Deployment

Qwen3.8 PP Deployment — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in scripts/, morning-of test runbook. Verified gotchas: no --device with --rpc, default split is proportional to reported free memory (pin 1,1), the iGPU side defaults to the ROCm backend.

Sample Scripts

Sample Script Set — An educational two-node RPC split script set, for reading only — never run it or deploy it. Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for the largest context that fits. For educational reading only. Do not run it in production.

Pages

Page	Owns
Multi-Node Runtimes	How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts.

2.9 KiB Raw Blame History