Dankclaw — Improvements Research

This page collects concrete, sourced recommendations to improve Dankclaw (the AMD Strix Halo inference box). It covers self-hosted agentic tools, inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action.

Dates: researched 2026-08. Hardware is verified on the live machine unless marked "verify first".

Verified Memory Picture (measured 2026-08-11)

Figure	Value	Source
OS physical RAM	30.5 GiB	`/proc/meminfo`
TTM pages_limit	15.24 GiB (3,995,281 x 4KiB)	`/sys/module/ttm/parameters/pages_limit`
ROCm VRAM total	96.0 GiB	`rocm-smi --showmeminfo vram`
ROCm VRAM used	76.2 GiB	`rocm-smi` while serving 3x200700

The OS accounts only ~30.5 GiB physical RAM, and the TTM GPU-addressable pool is capped at ~15.24 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM). This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in frank-default-large-benchmark.

1. Self-Hosted Agentic Tools

All of these connect to the local llama.cpp OpenAI-compatible endpoint (http://localhost:7770/v1). None need NVIDIA.

Tool	Use	Endpoint config	Priority
Open WebUI	Chat + agent harness, plugins, RAG, tool calling	Add OpenAI-compatible provider -> `http://localhost:7770/v1`	High
Cline (VS Code)	IDE coding agent with MCP	Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1`	High
Aider (CLI)	Terminal pair-programming agent	`OPENAI_API_BASE=http://localhost:7770/v1`	Medium
Continue.dev (VS Code)	Open-source coding agent	`config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}`	Medium
OpenHands (Agent Canvas)	Self-hosted multi-agent coding platform	LiteLLM custom model + Base URL to the endpoint	Medium
AnythingLLM	RAG / knowledge chat	Provider "OpenAI (Generic)" -> endpoint	Medium
Browser Use	Browser automation agent (Playwright)	`ChatOpenAI(base_url="http://localhost:7770/v1")`	Low
Qdrant	Vector store for RAG	Standalone, CPU-native	Low

The pragmatic first installs are Open WebUI, Cline, and Aider. They all work with the current llama.cpp server and need no extra GPU.

2. Inference Runtime Tuning (llama.cpp on gfx1151)

Verified against the running service and llama.cpp source.

Change	Recommendation	Priority
Backend	Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU.	Done
KV cache	Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed.	Keep
Flash attention	Leave `--flash-attn auto`. It helps prompt processing, not decode.	Keep
Load mode	Start with mmap (default). Enable `--mlock` only if page-fault stutter appears.	Optional
Threads	Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver.	Medium
DFlash n_max	Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput.	Verify
ngram layering	Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled.	Keep off

3. Server / OS / Hardware Settings

All require root and a reboot for the modprobe change.

Setting	Action	Effect	Priority
TTM GPU pool	Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot	Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers	High
Power profile	`echo performance > /sys/class/drm/card0/device/power_profile`	Sustained inference speed; apply at boot via a udev/systemd unit	Medium
NVMe scheduler	Already `none` (`[none]` confirmed)	No change	Done
BIOS VGM	Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS	Max dedicated VRAM for the GPU carve-out; verify in the live BIOS	Verify
Firmware/EC	Check GMKtec for a newer EC/fan-curve firmware	Better sustained thermal behavior	Verify

4. Other Enhancements

Test --spec-draft-n-max 5 and re-benchmark acceptance; if throughput rises, change the env tunable.
Consider switching the disabled vllm-thinkingcap.service unit off or removing it to avoid confusion (it is loaded in a failed state).
Profile generation under 3-slot concurrent load before touching boot settings, so any tuning is measured against the real workload.

Sources

frank-default-large-benchmark — measured backend/spec numbers
deployment — deploy and update process
AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
llama.cpp source: ggml/CMakeLists.txt, ggml/src/ggml-hip/CMakeLists.txt, common/arg.cpp, common/common.h
NotebookCheck Ryzen AI MAX+ 395 analysis

5.0 KiB Raw Blame History