serverAppConfigs/serverAppConfigs-docs/LLM/Dankclaw Improvements.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	This page collects concrete, sourced recommendations to improve Dankclaw	llm	2026-09-03

Dankclaw — Improvements Research

This page collects concrete, sourced recommendations to improve Dankclaw (the AMD Strix Halo inference box). It covers self-hosted agentic tools, inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action.

Dates: researched 2026-08. Hardware is verified on the live machine unless marked "verify first". Re-verified 2026-08-15/16; corrections marked below.

Live today (2026-10-18). The era described below — llama.cpp on the 8060S iGPU — is retired; what runs on this box now (both endpoints, their engines and ports) is frank-dragoon's and frank-zealot's pages, and both run under their repo units today — no drift open as of 2026-10-19 (frank-zealot active since 2026-10-18 21:06, the deployer's 2-slot reslice; frank-dragoon active). The OS facts are this page's: iommu=pt since 2026-10-18 (dankclaw-iommu-translation-off), and the ttm ceiling and NVMe scheduler rows below were re-checked 2026-10-18 and still hold. Every benchmark number below is a measurement of the era it was taken in.

Corrections (re-verified 2026-08-15/16)

• Backend: that era's production was Vulkan0, not ROCm. On this box ROCm wins prefill and Vulkan wins decode, and the decode gap widens with context: the 64k A/B is §4, the 262k A/B is §High-Context Backend Comparison. The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S.
• Power profile: the /sys/class/drm/card0/device/power_profile file does not exist on this machine (checked 2026-08-15). GPU DPM is already performance, forced level auto. This item is N/A.
• KV cache: production ran q8_0 K and q8_0 V (both quantized). What frank-zealot runs now is its own Current shape — this row is not it.
• Spec: DFlash/DSpark are retired. That era's production was --spec-default (ngram-mod) with draft-mtp first — timeline on ; the spec setting each endpoint runs today is on its own page.
• TTM: the pages_limit raise that removed the ~15 GiB ceiling is recorded once on Dankclaw System Specs (Memory).

Verified Memory Picture (measured 2026-08-11)

Figure	Value	Source
OS physical RAM	30.5 GiB	/proc/meminfo
ROCm VRAM total	96.0 GiB	rocm-smi --showmeminfo vram under the then firmware VGM carve
ROCm VRAM used	76.2 GiB	rocm-smi while serving 3x200700

That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as "dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is recorded in §Why Vulkan + GTT; the 76.2 GiB row is an era frank-dragoon load reading). Every live number on this box — MemTotal the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated carve and the ttm page cap — belongs to Dankclaw System Specs, re-read 2026-10-18.

This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in frank-dragoon.

1. Self-Hosted Agentic Tools

All of these connect to the local OpenAI-compatible endpoint on :7770 (which engine owns that port is frank-dragoon's to say), and the address must be the IPv4 literal: curl http://localhost:7770/v1/models returns nothing while http://127.0.0.1:7770/v1/models answers 200 (checked 2026-10-18 — the rootless podman publish is IPv4-only). None need NVIDIA.

Tool	Use	Endpoint config	Priority
Open WebUI	Chat + agent harness, plugins, RAG, tool calling	Add OpenAI-compatible provider -> http://localhost:7770/v1	High
Cline (VS Code)	IDE coding agent with MCP	Provider "OpenAI Compatible", API_URL=http://localhost:7770/v1	High
Aider (CLI)	Terminal pair-programming agent	OPENAI_API_BASE=http://localhost:7770/v1	Medium
Continue.dev (VS Code)	Open-source coding agent	config.json -> {type:"openai", apiBase:"http://localhost:7770"}	Medium
OpenHands (Agent Canvas)	Self-hosted multi-agent coding platform	LiteLLM custom model + Base URL to the endpoint	Medium
AnythingLLM	RAG / knowledge chat	Provider "OpenAI (Generic)" -> endpoint	Medium
Browser Use	Browser automation agent (Playwright)	ChatOpenAI(base_url="http://localhost:7770/v1")	Low
Qdrant	Vector store for RAG	Standalone, CPU-native	Low

The pragmatic first installs are Open WebUI, Cline, and Aider. They all work with the endpoint on 7770 and need no extra GPU.

2. Inference Runtime Tuning (llama.cpp on gfx1151)

Verified against that era's running service and llama.cpp source; the era is retired (frank-dragoon), so this table is history and a method, not a live TODO list.

Change	Recommendation	Priority
Backend	Vulkan0 (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads.	Done
KV cache	q8_0 K / q8_0 V in production. Try q4_0 only if more context is needed.	Keep
Flash attention	Leave --flash-attn auto. It helps prompt processing, not decode.	Keep
Load mode	Start with mmap (default). Enable --mlock only if page-fault stutter appears.	Optional
Threads	Use --threads 12 --threads-batch 16 --threads-http 1 for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver.	Medium
DFlash n_max	Depth 8 (block 16) was high; a shallower draft cuts rejection cost at the same throughput. Sweep depth per the A/B protocol in DFlash DSpark Speculative Decoding.	Verify
ngram layering	Layering ngram on the DFlash draft measured slower than the draft alone at 4k — figures on DFlash DSpark Speculative Decoding.	Keep off

3. Server / OS / Hardware Settings

All require root and a reboot for the modprobe change.

Setting	Action	Effect	Priority
TTM GPU pool	Raised 2026-08-25; the knob, its value and its persistence file are Dankclaw System Specs's (Memory row)	Lifted the ~15 GiB kernel-page ceiling so a full 262k KV plus a ~30 GB payload fits one slot	Done
Power profile	N/A on this machine (2026-08-15; re-checked 2026-10-18: no power_profile under /sys/class/drm/card0/device/ or card1/device/); GPU DPM already performance	none	Medium
NVMe scheduler	Already none ([none] mq-deadline in /sys/block/nvme0n1/queue/scheduler, re-read 2026-10-18)	No change	Done
BIOS VGM	Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS	Max dedicated VRAM for the GPU carve-out; verify in the live BIOS	Verify
Firmware/EC	Check GMKtec for a newer EC/fan-curve firmware	Better sustained thermal behavior	Verify

4. Other Enhancements

• Test --spec-draft-n-max 5 and re-benchmark acceptance; if throughput rises, change the env tunable.
• The vllm-thinkingcap.service unit this item told you to clean up is gone: systemctl is-enabled vllm-thinkingcap.service → not-found (2026-10-18).
• Profile generation under 3-slot concurrent load before touching boot settings, so any tuning is measured against the real workload.

Sources

• frank-dragoon — measured backend/spec numbers
• deploy and update process
• AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
• llama.cpp source: ggml/CMakeLists.txt, ggml/src/ggml-hip/CMakeLists.txt common/arg.cpp, common/common.h
• NotebookCheck Ryzen AI MAX+ 395 analysis

See Also

• frank-dragoon
• DFlash DSpark Speculative Decoding