- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
* no --device with --rpc (RPC device silently dropped otherwise)
* default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
* default split proportional to reported free mem = ~20/80 here, pin 1,1
* -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
quant landscape, 2026-08-16 test matrix), PP deployment runbook,
index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not
10), the ubatch findings (256 is the safe/fast point), and the premature-stop
sampling-variance investigation.
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.
ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP,
bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into
the KB under docs/Pipeline Parallelism/ with an index, expected
performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node
Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark
noted as vLLM-only). Nothing here is deployed or intended to run.
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
decode cost. Docs synced (benchmark Chosen State, deployment, specs).
Frank