chore(memory): snapshot auto-learned memory and renamed workstation skills

--home-blank-.omp--: MEMORY.md +140 / learned.md / memory_summary.md /
raw_memories.md +1002 rewritten by auto-learn after the model rename,
frank-medium split-balance and bottleneck studies. Skills renamed to
single-topic buckets: dotfile-git-workflow -> ghostty-text-aa +
niri-display-rotation + noctalia-theme, handy-service-mgmt ->
handy-flm-inference, inference-runtime -> llama-cpp-parallelism.

--home-blank-Projects-serverAppConfig--: 38 new rollout summaries,
raw_memories.md/MEMORY.md refreshed, skills re-bucketed (bench-protocol,
llm-fleet-model-swap, rootless-quadlet-log-retrieval, orchestration-quadlet).

Per-worktree bucket (wt-20261006-122204) is now gitignored, not committed.
This commit is contained in:
Frank committed 2026-10-09 13:17:53 -07:00
1 parent a34322aeb0
commit 5b08896e4e
72 files changed
+1461 -175

No files matched your search

@@ -0,0 +1,23 @@
# llama.cpp Multi-GPU Parallelism
## Split modes (--split-mode / -sm)
- **layer** (default): Pipeline parallelism. Each GPU holds a contiguous layer slice. Token flows sequentially through GPUs. Minimizes inter-GPU data movement; tolerates slow interconnects (PCIe fine). Best for prefill and fitting large models. Weakness: needs many tokens to keep pipeline full, poor low-batch latency.
- **tensor** (experimental): True tensor parallelism. Splits each layer's weights AND KV cache across all GPUs via 'meta device', doing cross-GPU all-reduce per layer (NCCL). Minimizes latency, parallelizes any workload size. Heavily bottlenecked by interconnect speed (wants NVLink + NCCL). Best for fast token generation.
- **row** (deprecated): Row-split dense weights only. Predecessor to tensor; comparatively poor performance.
- **none**: Single GPU, selected via --main-gpu.
## Key constraints on tensor mode
- Requires flash attention ON (-fa on)
- KV cache must be non-quantized: f32/f16/bf16 only
- NOT implemented for: MoE/hybrid, Mamba/SSM, and several other architectures (hard-fails with SPLIT_MODE_TENSOR not implemented)
- --fit is disabled; must size --ctx-size / --parallel manually
## Separate axis (NOT model parallelism)
- **--parallel / -np**: Number of concurrent server slots (each with own KV cache). Serving-level concurrency/batch parallelism. On single GPU this is the main throughput knob.
- **CPU+GPU offload** (--n-gpu-layers, --n-cpu-moe): Hybrid inference when model exceeds VRAM. Device-distribution strategy, not 'parallelism' in the strict sense.
## What does NOT exist in llama.cpp
- No first-class context/sequence parallelism mode (unlike vLLM/TensorRT-LLM). If that term is applied to llama.cpp, it's loose third-party language.
## Hardware context (this environment)
- Single RTX 4070 Max-Q (~12 GB VRAM). Multi-GPU split modes do not apply. Relevant knobs: --parallel for concurrency, CPU+GPU offloading for models >12 GB.