serverAppConfigs/serverAppConfigs-docs/LLM/Dankclaw Improvements.md

Dankclaw — Improvements Research

This page collects concrete, sourced recommendations to improve Dankclaw (the AMD Strix Halo inference box). It covers self-hosted agentic tools, inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action.

Dates: researched 2026-08. Hardware is verified on the live machine unless marked "verify first". Re-verified 2026-08-15/16; corrections marked below.

Corrections (re-verified 2026-08-15/16)

• Backend: production is Vulkan0, not ROCm. Measured on this box (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs 302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S. See frank-default-large-benchmark.
• Power profile: the /sys/class/drm/card0/device/power_profile file does not exist on this machine (checked 2026-08-15). GPU DPM is already performance, forced level auto. This item is N/A.
• KV cache: production runs q8_0 K and q8_0 V (both quantized).
• Spec: DFlash/DSpark are retired. Production is --spec-default (ngram-mod) with draft-mtp first. See qwen3.8-spec-matrix.
• TTM: raised to 114 GiB on 2026-08-25, set live (no reboot) and persisted in /etc/modprobe.d/ttm.conf. The effective VRAM ceiling that once held the service at ~15 GiB is gone.

Verified Memory Picture (measured 2026-08-11)

Figure	Value	Source
OS physical RAM	30.5 GiB	/proc/meminfo
TTM pages_limit	114 GiB (29,883,536 x 4KiB) — raised 2026-08-25	/sys/module/ttm/parameters/pages_limit
ROCm VRAM total	96.0 GiB	rocm-smi --showmeminfo vram
ROCm VRAM used	76.2 GiB	rocm-smi while serving 3x200700

The OS accounts only ~30.5 GiB physical RAM. The TTM GPU-addressable pool was capped at ~15.24 GiB (the real pre-2026-08-25 ceiling); that cap is now lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a different, driver-reported figure that has always been larger than the kernel cap. This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in frank-default-large-benchmark.

1. Self-Hosted Agentic Tools

All of these connect to the local llama.cpp OpenAI-compatible endpoint (http://localhost:7770/v1). None need NVIDIA.

Tool	Use	Endpoint config	Priority
Open WebUI	Chat + agent harness, plugins, RAG, tool calling	Add OpenAI-compatible provider -> http://localhost:7770/v1	High
Cline (VS Code)	IDE coding agent with MCP	Provider "OpenAI Compatible", API_URL=http://localhost:7770/v1	High
Aider (CLI)	Terminal pair-programming agent	OPENAI_API_BASE=http://localhost:7770/v1	Medium
Continue.dev (VS Code)	Open-source coding agent	config.json -> {type:"openai", apiBase:"http://localhost:7770"}	Medium
OpenHands (Agent Canvas)	Self-hosted multi-agent coding platform	LiteLLM custom model + Base URL to the endpoint	Medium
AnythingLLM	RAG / knowledge chat	Provider "OpenAI (Generic)" -> endpoint	Medium
Browser Use	Browser automation agent (Playwright)	ChatOpenAI(base_url="http://localhost:7770/v1")	Low
Qdrant	Vector store for RAG	Standalone, CPU-native	Low

The pragmatic first installs are Open WebUI, Cline, and Aider. They all work with the current llama.cpp server and need no extra GPU.

2. Inference Runtime Tuning (llama.cpp on gfx1151)

Verified against the running service and llama.cpp source.

Change	Recommendation	Priority
Backend	Vulkan0 (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads.	Done
KV cache	q8_0 K / q8_0 V in production. Try q4_0 only if more context is needed.	Keep
Flash attention	Leave --flash-attn auto. It helps prompt processing, not decode.	Keep
Load mode	Start with mmap (default). Enable --mlock only if page-fault stutter appears.	Optional
Threads	Use --threads 12 --threads-batch 16 --threads-http 1 for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver.	Medium
DFlash n_max	Current 8 (block 16) is high. Try --spec-draft-n-max 5; it may reduce rejection cost at the same throughput.	Verify
ngram layering	Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled.	Keep off

3. Server / OS / Hardware Settings

All require root and a reboot for the modprobe change.

Setting	Action	Effect	Priority
TTM GPU pool	/etc/modprobe.d/ttm.conf → options ttm pages_limit=29883536 page_pool_size=29883536 (applied 2026-08-25, set live then persisted)	Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot	Done
Power profile	N/A on this machine (2026-08-15): no power_profile sysfs; GPU DPM already performance	none	Medium
NVMe scheduler	Already none ([none] confirmed)	No change	Done
BIOS VGM	Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS	Max dedicated VRAM for the GPU carve-out; verify in the live BIOS	Verify
Firmware/EC	Check GMKtec for a newer EC/fan-curve firmware	Better sustained thermal behavior	Verify

4. Other Enhancements

• Test --spec-draft-n-max 5 and re-benchmark acceptance; if throughput rises, change the env tunable.
• Consider switching the disabled vllm-thinkingcap.service unit off or removing it to avoid confusion (it is loaded in a failed state).
• Profile generation under 3-slot concurrent load before touching boot settings, so any tuning is measured against the real workload.

Sources

• frank-default-large-benchmark — measured backend/spec numbers
• deployment — deploy and update process
• AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
• llama.cpp source: ggml/CMakeLists.txt, ggml/src/ggml-hip/CMakeLists.txt, common/arg.cpp, common/common.h
• NotebookCheck Ryzen AI MAX+ 395 analysis

See Also

• frank-default-large-benchmark
• DFlash DSpark Speculative Decoding
• deployment
• index