chore(memory): snapshot auto-learned memory and renamed workstation skills
--home-blank-.omp--: MEMORY.md +140 / learned.md / memory_summary.md / raw_memories.md +1002 rewritten by auto-learn after the model rename, frank-medium split-balance and bottleneck studies. Skills renamed to single-topic buckets: dotfile-git-workflow -> ghostty-text-aa + niri-display-rotation + noctalia-theme, handy-service-mgmt -> handy-flm-inference, inference-runtime -> llama-cpp-parallelism. --home-blank-Projects-serverAppConfig--: 38 new rollout summaries, raw_memories.md/MEMORY.md refreshed, skills re-bucketed (bench-protocol, llm-fleet-model-swap, rootless-quadlet-log-retrieval, orchestration-quadlet). Per-worktree bucket (wt-20261006-122204) is now gitignored, not committed.
This commit is contained in:
1 parent
a34322aeb0
commit
5b08896e4e
72 files changed
+1461
-175
No files matched your search
@@ -0,0 +1,23 @@
|
||||
# llama.cpp Multi-GPU Parallelism
|
||||
|
||||
## Split modes (--split-mode / -sm)
|
||||
- **layer** (default): Pipeline parallelism. Each GPU holds a contiguous layer slice. Token flows sequentially through GPUs. Minimizes inter-GPU data movement; tolerates slow interconnects (PCIe fine). Best for prefill and fitting large models. Weakness: needs many tokens to keep pipeline full, poor low-batch latency.
|
||||
- **tensor** (experimental): True tensor parallelism. Splits each layer's weights AND KV cache across all GPUs via 'meta device', doing cross-GPU all-reduce per layer (NCCL). Minimizes latency, parallelizes any workload size. Heavily bottlenecked by interconnect speed (wants NVLink + NCCL). Best for fast token generation.
|
||||
- **row** (deprecated): Row-split dense weights only. Predecessor to tensor; comparatively poor performance.
|
||||
- **none**: Single GPU, selected via --main-gpu.
|
||||
|
||||
## Key constraints on tensor mode
|
||||
- Requires flash attention ON (-fa on)
|
||||
- KV cache must be non-quantized: f32/f16/bf16 only
|
||||
- NOT implemented for: MoE/hybrid, Mamba/SSM, and several other architectures (hard-fails with SPLIT_MODE_TENSOR not implemented)
|
||||
- --fit is disabled; must size --ctx-size / --parallel manually
|
||||
|
||||
## Separate axis (NOT model parallelism)
|
||||
- **--parallel / -np**: Number of concurrent server slots (each with own KV cache). Serving-level concurrency/batch parallelism. On single GPU this is the main throughput knob.
|
||||
- **CPU+GPU offload** (--n-gpu-layers, --n-cpu-moe): Hybrid inference when model exceeds VRAM. Device-distribution strategy, not 'parallelism' in the strict sense.
|
||||
|
||||
## What does NOT exist in llama.cpp
|
||||
- No first-class context/sequence parallelism mode (unlike vLLM/TensorRT-LLM). If that term is applied to llama.cpp, it's loose third-party language.
|
||||
|
||||
## Hardware context (this environment)
|
||||
- Single RTX 4070 Max-Q (~12 GB VRAM). Multi-GPU split modes do not apply. Relevant knobs: --parallel for concurrency, CPU+GPU offloading for models >12 GB.
|
||||
Reference in new issue
Block a user