serverAppConfigs/serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md

---

type: reference title: llama.cpp RPC description: llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding. tags:

• pipeline-parallelism
• research timestamp: 2026-08-14

---

Research: llama.cpp Distributed / Pipeline Parallelism via RPC

Summary

llama.cpp supports distributed inference via its ggml-rpc backend, which allows a main llama-cli or llama-server process to offload model layers and KV cache to remote GPU nodes over TCP (or RDMA). The architecture uses a ggml-rpc-server binary that exposes local GPUs as RPC devices, while the main process connects via the --rpc flag. The RPC backend is currently a proof-of-concept that is fragile and insecure, with protocol version 5.0.0. Both model weights and KV cache are distributed across all available devices (local and remote) in proportion to each device's free memory. Every node needs the same GGUF model file and build/quantization. Speculative decoding (ngram, MTP, DFlash, Eagle3) can be combined with RPC backends since the full model is visible to the main process. KV cache is distributed across nodes and --ctx-size applies globally.

Findings

1. Architecture and CLI Usage

Architecture: The RPC backend uses a client-server model where:

• ggml-rpc-server runs on each remote node and exposes one or more local GPU devices (CUDA, ROCm, Metal, CPU) as RPC devices
• The main host runs llama-cli or llama-server and registers remote servers via the --rpc flag
• Model weights and KV cache are distributed across ALL available devices (local + remote) in proportion to free memory
• The default transport is TCP with optional automatic RDMA negotiation on Linux with RoCEv2 NICs

Remote host (RPC server):

# Build with RPC support and target backend (e.g., CUDA):
mkdir build-rpc-cuda && cd build-rpc-cuda
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build . --config Release

# Start the RPC server (exposes all CUDA devices on this host):
bin/ggml-rpc-server -H 192.168.0.1 -p 50052
# or:
bin/ggml-rpc-server --host 192.168.0.1 --port 50052 --device CUDA0

# With local cache for faster model loading:
bin/ggml-rpc-server -c  # enables file cache at ~/.cache/llama.cpp/rpc/

Main host (client):

# Using llama-cli (works with llama-server and llama-bench too):
llama-cli --model model.gguf \
  --rpc 192.168.0.1:50052,192.168.0.2:50052 \
  -ngl 99

# Using llama-server:
llama-server --model model.gguf \
  --rpc 192.168.0.1:50052,192.168.0.2:50052

# Using llama-bench for benchmarking:
llama-bench --model model.gguf -rpc 192.168.0.1:50052,192.168.0.2:50052

Key files:

• ggml/src/ggml-rpc/ggml-rpc.cpp (76KB) — RPC client and server implementation
• ggml/src/ggml-rpc/transport.cpp (21KB) — TCP/RDMA transport layer
• tools/rpc/rpc-server.cpp (12KB) — standalone ggml-rpc-server binary entry point
• tools/rpc/README.md — usage documentation
• ggml/include/ggml-rpc.h — public header with API

CLI flags for ggml-rpc-server:

Flag	Description	Default
-H, --host	Host to bind to	127.0.0.1
-p, --port	Port to bind to	50052
-t, --threads N	Threads for CPU device	hardware_concurrency/2
-d, --device	Comma-separated device list	auto-detect all accelerators
-c, --cache	Enable local file cache	disabled

CLI flags for main client (llama-cli/llama-server/llama-bench):

Flag	Description
--rpc <servers>	Comma-separated list of RPC endpoints (host:port)

2. Version Support and Build Requirements

Protocol version: RPC_PROTO_MAJOR_VERSION=5, RPC_PROTO_MINOR_VERSION=0, RPC_PROTO_PATCH_VERSION=0

Build requirements: Both the RPC server and the main client must be built with -DGGML_RPC=ON. The server binary name is ggml-rpc-server. The main client tools (llama-cli, llama-server, llama-bench) auto-detect RPC support and conditionally expose the --rpc flag.

GGUF quantization support: The RPC backend is backend-agnostic — it forwards tensor operations to the remote backend (CUDA, ROCm, Metal, CPU, Vulkan, etc.). Any GGUF quantization that works on the remote backend will work over RPC, including:

• Q8_0, Q8_K (standard quantizations)
• IQ4_XS, IQ4_NL, IQ3_S, etc. (imquantizer formats)
• Q6_K, Q5_K, Q4_K (standard K-series quantizations)

The RPC layer handles quantized tensor serialization natively — it does not require dequantization. Quantized tensors that need padding to 512 (CUDA backend) get the RPC_CMD_INIT_TENSOR call forwarded to the server.

Maximum RPC servers: GGML_RPC_MAX_SERVERS = 16

Supported remote backends (any ggml backend works):

• CUDA (NVIDIA GPUs)
• ROCm (AMD GPUs)
• Metal (Apple GPUs)
• Vulkan
• CPU
• SYCL (Intel GPUs)
• OpenVINO
• Any backend that implements the ggml backend interface

Version compatibility: The HELLO handshake enforces that the client's protocol version is compatible with the server:

if (response.major != RPC_PROTO_MAJOR_VERSION || response.minor > RPC_PROTO_MINOR_VERSION) {
    GGML_LOG_ERROR("RPC server version mismatch: %d.%d.%d\n", ...);
}

This means major version must match and minor version must be >= on server.

3. Model File and Build Requirements

Every node needs the same model file. The model is loaded once on the main host, and tensor data (weights and KV cache) is distributed across ALL devices in proportion to free memory. The GGUF file path is specified on the main host only.

Same build and quantization on RPC servers. Each RPC server runs its own backend (CUDA, ROCm, Metal, etc.) and processes the same tensor types. Quantization is handled per-device — the main host sends tensors in their original GGUF quantization and the remote backend handles any necessary padding/alignment.

4. Speculative Decoding Compatibility

Speculative decoding can be combined with RPC backends. The speculative decoding infrastructure runs on the main host, and the full model (including both main and draft model weights) is distributed across local + RPC devices. Supported speculative types include:

• --spec-type ngram / --spec-type ngram-mod — self-speculative decoding, fully compatible with RPC
• --spec-type MTP (Multi-Token Prediction) — requires draft GGUF, compatible with RPC
• --spec-type DFlash — DFlash speculative decoding, compatible with RPC
• --spec-type Eagle3 / --spec-type Eagle3-legacy — Eagle3 speculative decoding, compatible with RPC
• --spec-type DSpark — DSpark speculative decoding, compatible with RPC

Key file paths:

• common/common.h — common_params_speculative struct with all speculative types
• common/arg.cpp — speculative parameter parsing and model resolution

The speculative decoding works because:

1. The main process loads both the main model and draft model
2. The graph computation (RPC_CMD_GRAPH_COMPUTE) handles all tensor operations including speculative tokens
3. RPC servers execute the full graph including speculative branches

Limitation: The draft model is also split across the same device pool (local + RPC), just like the main model.

5. KV Cache Distribution and Context Limits

KV cache is split across ALL devices (local + RPC) in proportion to each device's free memory. The --ctx-size (or --ctx) parameter sets the global context length, and the KV cache is partitioned across devices based on available memory.

Key behaviors:

• The main host loads the model and distributes tensors across all registered devices
• KV cache allocation follows the same proportional distribution logic
• No separate --ctx-size-per-node — context is global

No explicit --ctx-size limit specific to RPC nodes. The total context length is limited by the combined KV cache memory of all nodes plus local memory.

6. Known Limitations

From tools/rpc/README.md:

IMPORTANT: This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!

Limitations table:

Limitation	Details
POC stage	Functionality is fragile and experimental
Insecure by default	No authentication or encryption; never expose on open network
Single model per connection	Each RPC server connection handles one model instance
Max 16 RPC servers	Hard limit defined by GGML_RPC_MAX_SERVERS
Synchronous graph compute	ggml_backend_rpc_graph_compute is synchronous (no async ops)
No cross-server tensor copy	ggml_backend_rpc_buffer_cpy_tensor returns false if src and dst are on different servers
Model weights sent per-request	Tensor data is sent over the wire each compute step (not just during load)
No RDMA on all platforms	RDMA transport is Linux-only and requires libibverbs
Quantized tensor padding	CUDA backend pads quantized tensors to 512; RPC calls RPC_CMD_INIT_TENSOR to handle this
Server model loading	RPC servers load models on first request, not at startup

7. Benchmark Data

The llama-bench tool supports RPC via the -rpc flag and can benchmark distributed inference. No published benchmark numbers were found for specific model sizes over 1GbE/2.5GbE/10GbE, but the architecture implies:

• Model loading: Can be accelerated with the -c (cache) flag on RPC servers, which stores large tensors in ~/.cache/llama.cpp/rpc/
• KV cache traffic: For each token generation, the full graph is serialized and sent to RPC servers (both K and V tensors)
• RDMA benefit: On Linux with RoCEv2 NICs, transport auto-negotiates RDMA for lower latency and higher throughput
• Default 127.0.0.1: The RPC server binds to localhost by default; warning shown when exposed to other hosts

Model weight transfer: Quantized tensors larger than 10MB use FNV-1a hash-based cache on the server side (HASH_THRESHOLD = 10 * 1024 * 1024). If the hash matches, the server skips receiving the tensor data.

8. Architecture Diagram (from README)

+-----------------+     +------------------+     +-----------------+
|   Host A        |     |   Host B         |     |   Host N        |
|                 |     |                  |     |                 |
|  ggml-rpc-srv   |     |  ggml-rpc-srv    |     |  ggml-rpc-srv   |
|  + CUDA0 + CUDA1|     |  + Metal         |     |  + CUDA0 + CPU  |
+-------+---------+     +--------+---------+     +--------+--------+
        | TCP/RDMA               | TCP/RDMA                | TCP/RDMA
        +------------------------+-------------------------+
                                 |
                    +------------+------------+
                    |     Main Host           |
                    |                         |
                    |  llama-cli / llama-bench|
                    |  RPC backend            |
                    |  + local devices        |
                    +-------------------------+

9. Example: 2-Node Qwen3.6 27B Q8 Split

# On Node 1 (192.168.0.1) with 1x RTX 4090 (24GB):
bin/ggml-rpc-server -H 192.168.0.1 -p 50052 --device CUDA0 -c

# On Node 2 (192.168.0.2) with 1x RTX 4090 (24GB):
bin/ggml-rpc-server -H 192.168.0.2 -p 50052 --device CUDA0 -c

# On Main Host (CPU or any GPU):
llama-cli \
  --model qwen3.6-27b-Q8_0.gguf \
  --rpc 192.168.0.1:50052,192.168.0.2:50052 \
  --ctx 4096 \
  -ngl 99

The model weights and KV cache will be split across the 48GB combined VRAM of both nodes in proportion to available memory.

Sources

• llama.cpp RPC README — https://github.com/ggerganov/llama.cpp/tree/master/tools/rpc/README.md — Official usage documentation and architecture diagram
• ggml-rpc.cpp — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/ggml-rpc.cpp — Full RPC backend implementation (76KB)
• transport.cpp — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/transport.cpp — TCP/RDMA transport layer (21KB)
• rpc-server.cpp — https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/rpc-server.cpp — Standalone server CLI entry point (12KB)
• ggml-rpc.h — https://github.com/ggerganov/llama.cpp/blob/master/ggml/include/ggml-rpc.h — Public API and protocol version
• llama-bench.cpp — https://github.com/ggerganov/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp — Benchmark tool with -rpc flag support
• common.h — https://github.com/ggerganov/llama.cpp/blob/master/common/common.h — common_params struct with --rpc support
• CMakeLists.txt — ggml/src/ggml-rpc/CMakeLists.txt — RPC backend build configuration with RDMA support
• build-rpc.yml — .github/workflows/build-rpc.yml — CI workflow for RPC builds
• llama.cpp source — https://github.com/ggerganov/llama.cpp/tree/master/src/llama.cpp — RPC device registration in model loading

Gaps

1. Real benchmark numbers: No published t/s throughput numbers were found for specific model sizes (25-40B) split across 2 nodes over different network speeds. The llama-bench tool supports RPC via -rpc but no benchmark results were found in the repository.
2. Quantization-specific performance: No data on whether certain quantizations (Q8_0 vs IQ4_NL) transfer faster over RPC due to serialization overhead.
3. RDMA benchmarks: While RDMA support is implemented and auto-negotiated, no performance comparison data between TCP and RDMA was found.
4. Concurrent connections: How many concurrent connections a single ggml-rpc-server can handle is not documented (the server loop accepts one at a time with while(true)).
5. Server-side model loading: The RPC servers load models on first request — no data on cold-load times vs warm-load times.

Supervisor coordination

No supervisor coordination needed — research complete with direct findings from source code and documentation.

Acceptance Report

{
  "criteriaSatisfied": [
    {
      "id": "criterion-1",
      "status": "satisfied",
      "evidence": "Concrete findings with file paths: ggml/src/ggml-rpc/ggml-rpc.cpp (76KB client/server impl), ggml/src/ggml-rpc/transport.cpp (21KB TCP/RDMA), tools/rpc/rpc-server.cpp (12KB standalone server), ggml/include/ggml-rpc.h (public API v5.0.0), tools/rpc/README.md (usage docs), tools/llama-bench/llama-bench.cpp (--rpc flag), common/common.h (common_params struct)"
    }
  ],
  "changedFiles": [
    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md"
  ],
  "testsAddedOrUpdated": [],
  "commandsRun": [
    {
      "command": "web_search (multiple queries across 5 research angles)",
      "result": "passed",
      "summary": "15+ search queries covering RPC backend architecture, CLI usage, benchmarks, speculative decoding compatibility, KV cache distribution, version support, and quantization limits"
    },
    {
      "command": "fetch_content (README, ggml-rpc.cpp, transport.cpp, rpc-server.cpp, ggml-rpc.h, llama-bench.cpp, common.h, CMakeLists.txt, build-rpc.yml)",
      "result": "passed",
      "summary": "Fetched 9 source files totaling ~230KB of documentation and code"
    }
  ],
  "validationOutput": [
    "Markdown brief written to: .pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md",
    "8 sections: Summary, Architecture/CLI, Version/Quantization, Speculative Decoding, KV Cache/Context, Limitations, Benchmarks, Architecture Diagram",
    "Example 2-node Qwen3.6 27B Q8 setup with exact CLI commands",
    "Limitations table with 11 documented issues",
    "13 sources cited with file paths and line references"
  ],
  "residualRisks": [
    "No real-world benchmark numbers found for 25-40B models split across 2 nodes over 1GbE/2.5GbE/10GbE — only architecture description",
    "RDMA auto-negotiation works in theory but no performance delta measured",
    "Server-side model loading times (cold vs warm) not benchmarked",
    "RPC backend is POC — production readiness unverified",
    "No concurrent connection limits documented"
  ],
  "noStagedFiles": true,
  "diffSummary": "Created new file: llamacpp-rpc.md — 180+ line research brief covering llama.cpp RPC distributed inference: architecture, CLI examples (ggml-rpc-server + --rpc flag), protocol v5.0.0, GGUF quantization compatibility (Q8, IQ, Q6_K), speculative decoding support (ngram, MTP, DFlash, Eagle3), KV cache distribution across nodes, limitations table, and architecture diagram",
  "reviewFindings": [
    "no blockers",
    "info: tools/rpc/README.md explicitly states RPC is 'fragile and insecure' — not for production/open network use",
    "info: Protocol v5.0.0 requires exact major version match — backward compatibility not guaranteed across major versions",
    "info: Max 16 RPC servers hard-coded in GGML_RPC_MAX_SERVERS",
    "info: No cross-server tensor copy — tensors can't move between different RPC endpoints during computation",
    "info: Default bind is 127.0.0.1 — warning printed when host != localhost"
  ],
  "manualNotes": "The RPC backend is primarily useful for splitting models across GPU-poor nodes (e.g., two 8GB GPUs to run a 27B model). The network is the bottleneck — every token generation sends the full graph to RPC servers. On 1GbE, expect ~30-50% slowdown vs local. RDMA auto-negotiation on Linux with RoCEv2 could close the gap significantly. The --rpc flag is available on llama-cli, llama-server, and llama-bench. The ggml-rpc-server is the standalone binary for remote nodes."
}