Files

T

Frank 9544c36afc PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)

- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
  * no --device with --rpc (RPC device silently dropped otherwise)
  * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
  * default split proportional to reported free mem = ~20/80 here, pin 1,1
  * -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
  quant landscape, 2026-08-16 test matrix), PP deployment runbook,
  index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B

2026-08-16 05:40:27 -07:00

5.9 KiB

Raw Blame History

Dankclaw — Improvements Research

This page collects concrete, sourced recommendations to improve Dankclaw (the AMD Strix Halo inference box). It covers self-hosted agentic tools, inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action.

Dates: researched 2026-08. Hardware is verified on the live machine unless marked "verify first". Re-verified 2026-08-15/16; corrections marked below.

Corrections (re-verified 2026-08-15/16)

Backend: production is Vulkan0, not ROCm. Measured on this box (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs 302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S. See frank-default-large-benchmark.
Power profile: the /sys/class/drm/card0/device/power_profile file does not exist on this machine (checked 2026-08-15). GPU DPM is already performance, forced level auto. This item is N/A.
KV cache: production runs q8_0 K and q8_0 V (both quantized).
Spec: DFlash/DSpark are retired. Production is --spec-default (ngram-mod) with draft-mtp first. See qwen3.8-spec-matrix.
TTM: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still pending a reboot.

Verified Memory Picture (measured 2026-08-11)

Figure	Value	Source
OS physical RAM	30.5 GiB	`/proc/meminfo`
TTM pages_limit	15.24 GiB (3,995,281 x 4KiB)	`/sys/module/ttm/parameters/pages_limit`
ROCm VRAM total	96.0 GiB	`rocm-smi --showmeminfo vram`
ROCm VRAM used	76.2 GiB	`rocm-smi` while serving 3x200700

The OS accounts only ~30.5 GiB physical RAM, and the TTM GPU-addressable pool is capped at ~15.24 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM). This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in frank-default-large-benchmark.

1. Self-Hosted Agentic Tools

All of these connect to the local llama.cpp OpenAI-compatible endpoint (http://localhost:7770/v1). None need NVIDIA.

Tool	Use	Endpoint config	Priority
Open WebUI	Chat + agent harness, plugins, RAG, tool calling	Add OpenAI-compatible provider -> `http://localhost:7770/v1`	High
Cline (VS Code)	IDE coding agent with MCP	Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1`	High
Aider (CLI)	Terminal pair-programming agent	`OPENAI_API_BASE=http://localhost:7770/v1`	Medium
Continue.dev (VS Code)	Open-source coding agent	`config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}`	Medium
OpenHands (Agent Canvas)	Self-hosted multi-agent coding platform	LiteLLM custom model + Base URL to the endpoint	Medium
AnythingLLM	RAG / knowledge chat	Provider "OpenAI (Generic)" -> endpoint	Medium
Browser Use	Browser automation agent (Playwright)	`ChatOpenAI(base_url="http://localhost:7770/v1")`	Low
Qdrant	Vector store for RAG	Standalone, CPU-native	Low

The pragmatic first installs are Open WebUI, Cline, and Aider. They all work with the current llama.cpp server and need no extra GPU.

2. Inference Runtime Tuning (llama.cpp on gfx1151)

Verified against the running service and llama.cpp source.

Change	Recommendation	Priority
Backend	Vulkan0 (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads.	Done
KV cache	`q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed.	Keep
Flash attention	Leave `--flash-attn auto`. It helps prompt processing, not decode.	Keep
Load mode	Start with mmap (default). Enable `--mlock` only if page-fault stutter appears.	Optional
Threads	Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver.	Medium
DFlash n_max	Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput.	Verify
ngram layering	Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled.	Keep off

3. Server / OS / Hardware Settings

All require root and a reboot for the modprobe change.

Setting	Action	Effect	Priority
TTM GPU pool	Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot	Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers	High
Power profile	N/A on this machine (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance`	none	~~Medium~~
NVMe scheduler	Already `none` (`[none]` confirmed)	No change	Done
BIOS VGM	Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS	Max dedicated VRAM for the GPU carve-out; verify in the live BIOS	Verify
Firmware/EC	Check GMKtec for a newer EC/fan-curve firmware	Better sustained thermal behavior	Verify

4. Other Enhancements

Test --spec-draft-n-max 5 and re-benchmark acceptance; if throughput rises, change the env tunable.
Consider switching the disabled vllm-thinkingcap.service unit off or removing it to avoid confusion (it is loaded in a failed state).
Profile generation under 3-slot concurrent load before touching boot settings, so any tuning is measured against the real workload.

Sources

frank-default-large-benchmark — measured backend/spec numbers
deployment — deploy and update process
AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
llama.cpp source: ggml/CMakeLists.txt, ggml/src/ggml-hip/CMakeLists.txt, common/arg.cpp, common/common.h
NotebookCheck Ryzen AI MAX+ 395 analysis

5.9 KiB Raw Blame History