PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py - head design verified in b10433 source + live 4B smoke test: * no --device with --rpc (RPC device silently dropped otherwise) * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan) * default split proportional to reported free mem = ~20/80 here, pin 1,1 * -dio required for large model over RPC - docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers, quant landscape, 2026-08-16 test matrix), PP deployment runbook, index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs) - worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC) - second head build build-vulkan-rpc (HIP off) for backend A/B
This commit is contained in:
1 parent
c775077016
commit
9544c36afc
13 files changed
+1037
-5
No files matched your search
@@ -6,7 +6,23 @@ inference runtime tuning, and server/OS/hardware settings. Each item lists a
|
||||
priority and the exact action.
|
||||
|
||||
Dates: researched 2026-08. Hardware is verified on the live machine unless
|
||||
marked "verify first".
|
||||
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
||||
|
||||
## Corrections (re-verified 2026-08-15/16)
|
||||
|
||||
- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
|
||||
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
|
||||
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
|
||||
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
|
||||
hold on the 8060S. See [[frank-default-large-benchmark]].
|
||||
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
||||
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
||||
`performance`, forced level `auto`. This item is N/A.
|
||||
- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
|
||||
- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
|
||||
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
|
||||
- **TTM**: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still
|
||||
pending a reboot.
|
||||
|
||||
## Verified Memory Picture (measured 2026-08-11)
|
||||
|
||||
@@ -47,8 +63,8 @@ Verified against the running service and llama.cpp source.
|
||||
|
||||
| Change | Recommendation | Priority |
|
||||
| ------ | -------------- | -------- |
|
||||
| Backend | Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU. | Done |
|
||||
| KV cache | Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed. | Keep |
|
||||
| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
|
||||
| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
|
||||
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
|
||||
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
|
||||
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
|
||||
@@ -62,7 +78,7 @@ All require root and a reboot for the modprobe change.
|
||||
| Setting | Action | Effect | Priority |
|
||||
| ------- | ------ | ------ | -------- |
|
||||
| TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High |
|
||||
| Power profile | `echo performance > /sys/class/drm/card0/device/power_profile` | Sustained inference speed; apply at boot via a udev/systemd unit | Medium |
|
||||
| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
|
||||
| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
|
||||
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
|
||||
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
|
||||
|
||||
Reference in new issue
Block a user