# llama.cpp Multi-GPU Parallelism

## Split modes (--split-mode / -sm)
- **layer** (default): Pipeline parallelism. Each GPU holds a contiguous layer slice. Token flows sequentially through GPUs. Minimizes inter-GPU data movement; tolerates slow interconnects (PCIe fine). Best for prefill and fitting large models. Weakness: needs many tokens to keep pipeline full, poor low-batch latency.
- **tensor** (experimental): True tensor parallelism. Splits each layer's weights AND KV cache across all GPUs via 'meta device', doing cross-GPU all-reduce per layer (NCCL). Minimizes latency, parallelizes any workload size. Heavily bottlenecked by interconnect speed (wants NVLink + NCCL). Best for fast token generation.
- **row** (deprecated): Row-split dense weights only. Predecessor to tensor; comparatively poor performance.
- **none**: Single GPU, selected via --main-gpu.

## Key constraints on tensor mode
- Requires flash attention ON (-fa on)
- KV cache must be non-quantized: f32/f16/bf16 only
- NOT implemented for: MoE/hybrid, Mamba/SSM, and several other architectures (hard-fails with SPLIT_MODE_TENSOR not implemented)
- --fit is disabled; must size --ctx-size / --parallel manually

## Separate axis (NOT model parallelism)
- **--parallel / -np**: Number of concurrent server slots (each with own KV cache). Serving-level concurrency/batch parallelism. On single GPU this is the main throughput knob.
- **CPU+GPU offload** (--n-gpu-layers, --n-cpu-moe): Hybrid inference when model exceeds VRAM. Device-distribution strategy, not 'parallelism' in the strict sense.

## What does NOT exist in llama.cpp
- No first-class context/sequence parallelism mode (unlike vLLM/TensorRT-LLM). If that term is applied to llama.cpp, it's loose third-party language.

## Hardware context (this environment)
- Single RTX 4070 Max-Q (~12 GB VRAM). Multi-GPU split modes do not apply. Relevant knobs: --parallel for concurrency, CPU+GPU offloading for models >12 GB.