Files

T

Frank 87721ab0fe rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)

Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:

- Service trio: scripts/{frank-small.env,frank-small.service,
  serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
  frank-secondary-*.md (9 files); the FreeToken runbook stays
  freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
  LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
  3090, and the Dankserver service that became frank-medium) are
  preserved verbatim — they describe what the service was called
  at the time.

2026-10-06 15:37:13 -07:00

2.8 KiB

Raw Blame History

Pipeline Parallelism

This section covers running one large model across two machines. It focuses on pipeline parallelism (layer split) over ethernet. The research answers what speeds are possible and which tools work. The sample scripts show a two-node Qwen3.6-27B Q8 split for reading only.

Terminology

Pipeline parallelism (PP) splits model layers across machines. Each machine computes its layers and passes hidden states to the next machine. Tensor parallelism (TP) splits each layer across machines and requires an all-reduce per layer. PP is the only realistic option over ethernet.

Speculative decoding is a separate feature. It uses a small draft model (or n-grams) to propose tokens that the main model verifies in one pass. It can run together with PP in llama.cpp.

The Two Runtimes

graph LR
    subgraph llamaRPC[llama.cpp RPC]
        A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
    end
    subgraph vllmPP[vLLM PP]
        B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
    end

llama.cpp RPC — the working, production-capable path for two heterogeneous nodes over ethernet.
vLLM Pipeline Parallelism — supported but heavier; needs Ray or multiprocessing and does not compose with speculative decoding yet.

What the Research Found

Ethernet Bandwidth Math — a PP boundary moves only the hidden state (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of tokens per second on paper; the real limit is latency and RPC overhead.
Mixed-Vendor Experiments — NVIDIA + Strix Halo multi-node inference works today via llama.cpp RPC with separate CUDA and ROCm builds. The exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with published numbers yet.
Qwen3.6 27B Spec Methods — Q8 GGUF variants, and which speculative methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
Expected Performance — estimated tokens per second for a 27B Q8 split across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.

Deployment

Qwen3.8 PP Deployment — the real Qwen3.8-27B split: head on Dankclaw (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in scripts/, morning-of test runbook. Verified gotchas: no --device with --rpc, default split is proportional to reported free memory (pin 1,1), the iGPU side defaults to the ROCm backend.

Sample Scripts

Sample Script Set — a two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for the largest context that fits. For educational reading only. Do not run it in production.

2.8 KiB Raw Blame History