--- type: reference description: "This page collects concrete, sourced recommendations to improve Dankclaw" tags: - llm timestamp: 2026-09-03 --- # Dankclaw — Improvements Research This page collects concrete, sourced recommendations to improve Dankclaw (the AMD Strix Halo inference box). It covers self-hosted agentic tools, inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action. Dates: researched 2026-08. Hardware is verified on the live machine unless marked "verify first". Re-verified 2026-08-15/16; corrections marked below. **Live today (2026-10-18).** The era described below — llama.cpp on the 8060S iGPU behind :7770 — is retired. Dankclaw now serves [[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2, port 7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18 03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18 ([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of the era they were taken in; the `ttm` ceiling and the NVMe scheduler rows were re-checked 2026-10-18 and still hold. ## Corrections (re-verified 2026-08-15/16) - **Backend**: that era's production was **Vulkan0**, not ROCm. Measured on this box (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs 302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S. See [[frank-dragoon-benchmark]]. - **Power profile**: the `/sys/class/drm/card0/device/power_profile` file does not exist on this machine (checked 2026-08-15). GPU DPM is already `performance`, forced level `auto`. This item is N/A. - **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). The law still holds on `frank-zealot` (`CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` — `scripts/frank-zealot.env:55-56`). - **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default` (ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. Today `frank-zealot` is configured with speculation **off** (`SPEC_TYPE=none` — `scripts/frank-zealot.env:67`). - **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that once held the service at ~15 GiB is gone. ## Verified Memory Picture (measured 2026-08-11) | Figure | Value | Source | | ------ | ----- | ------ | | OS physical RAM | 30.5 GiB | `/proc/meminfo` | | TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` | | ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve | | ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 | That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as "dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT). The OS now accounts **121 GiB** (`MemTotal 127108080 kB`, 2026-10-18) and `rocm-smi --showmeminfo vram` reports **2.0 GiB** (2,147,483,648 B) of dedicated VRAM — the carve is gone and the GPU budget is the GTT aperture, not the SMI figure ([[Dankclaw System Specs]]). The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and `/etc/modprobe.d/ttm.conf` still carries the `options ttm` line (both read 2026-10-18). This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in [[frank-dragoon-benchmark]]. ## 1. Self-Hosted Agentic Tools All of these connect to the local OpenAI-compatible endpoint (`http://127.0.0.1:7770/v1`). Today that port is **halogen-flash-server**, not a llama.cpp server, and the address must be the IPv4 literal: `curl http://localhost:7770/v1/models` returns nothing while `http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18 — the rootless podman publish is IPv4-only). None need NVIDIA. | Tool | Use | Endpoint config | Priority | | ---- | --- | --------------- | -------- | | Open WebUI | Chat + agent harness, plugins, RAG, tool calling | Add OpenAI-compatible provider -> `http://localhost:7770/v1` | High | | Cline (VS Code) | IDE coding agent with MCP | Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1` | High | | Aider (CLI) | Terminal pair-programming agent | `OPENAI_API_BASE=http://localhost:7770/v1` | Medium | | Continue.dev (VS Code) | Open-source coding agent | `config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}` | Medium | | OpenHands (Agent Canvas) | Self-hosted multi-agent coding platform | LiteLLM custom model + Base URL to the endpoint | Medium | | AnythingLLM | RAG / knowledge chat | Provider "OpenAI (Generic)" -> endpoint | Medium | | Browser Use | Browser automation agent (Playwright) | `ChatOpenAI(base_url="http://localhost:7770/v1")` | Low | | Qdrant | Vector store for RAG | Standalone, CPU-native | Low | The pragmatic first installs are Open WebUI, Cline, and Aider. They all work with the endpoint on 7770 and need no extra GPU. The 7770 endpoint today is halogen-flash-server, not llama.cpp. ## 2. Inference Runtime Tuning (llama.cpp on gfx1151) Verified against the running service and llama.cpp source. | Change | Recommendation | Priority | | ------ | -------------- | -------- | | Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done | | KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep | | Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep | | Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional | | Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium | | DFlash n_max | Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput. | Verify | | ngram layering | Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled. | Keep off | ## 3. Server / OS / Hardware Settings All require root and a reboot for the modprobe change. | Setting | Action | Effect | Priority | | ------- | ------ | ------ | -------- | | TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done | | Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ | | NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done | | BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify | | Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify | ## 4. Other Enhancements - Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises, change the env tunable. - The `vllm-thinkingcap.service` unit this item told you to clean up is gone: `systemctl is-enabled vllm-thinkingcap.service` → `not-found` (2026-10-18). - Profile generation under 3-slot concurrent load before touching boot settings, so any tuning is measured against the real workload. ## Sources - [[frank-dragoon-benchmark]] — measured backend/spec numbers - [[deployment]] — deploy and update process - AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS) - llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`, `common/arg.cpp`, `common/common.h` - NotebookCheck Ryzen AI MAX+ 395 analysis ## See Also - [[frank-dragoon-benchmark]] - [[DFlash DSpark Speculative Decoding]] - [[deployment]] - [[index]]