From f20d6ce288ea9cb111f20b27292956aeffa54efd Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 11 Aug 2026 18:14:44 -0700 Subject: [PATCH] docs: add Dankclaw improvements research (agentic tools, runtime tuning, server/OS) --- docs/Dankclaw Improvements.md | 93 +++++++++++++++++++++++++++++++++++ docs/index.md | 1 + 2 files changed, 94 insertions(+) create mode 100644 docs/Dankclaw Improvements.md diff --git a/docs/Dankclaw Improvements.md b/docs/Dankclaw Improvements.md new file mode 100644 index 0000000..68aa224 --- /dev/null +++ b/docs/Dankclaw Improvements.md @@ -0,0 +1,93 @@ +# Dankclaw — Improvements Research + +This page collects concrete, sourced recommendations to improve Dankclaw +(the AMD Strix Halo inference box). It covers self-hosted agentic tools, +inference runtime tuning, and server/OS/hardware settings. Each item lists a +priority and the exact action. + +Dates: researched 2026-08. Hardware is verified on the live machine unless +marked "verify first". + +## Verified Memory Picture (measured 2026-08-11) + +| Figure | Value | Source | +| ------ | ----- | ------ | +| OS physical RAM | 30.5 GiB | `/proc/meminfo` | +| TTM pages_limit | 15.24 GiB (3,995,281 x 4KiB) | `/sys/module/ttm/parameters/pages_limit` | +| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` | +| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 | + +The OS accounts only ~30.5 GiB physical RAM, and the TTM GPU-addressable pool +is capped at ~15.24 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM). +This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory +limits documented in [[frank-default-large-benchmark]]. + +## 1. Self-Hosted Agentic Tools + +All of these connect to the local llama.cpp OpenAI-compatible endpoint +(`http://localhost:7770/v1`). None need NVIDIA. + +| Tool | Use | Endpoint config | Priority | +| ---- | --- | --------------- | -------- | +| Open WebUI | Chat + agent harness, plugins, RAG, tool calling | Add OpenAI-compatible provider -> `http://localhost:7770/v1` | High | +| Cline (VS Code) | IDE coding agent with MCP | Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1` | High | +| Aider (CLI) | Terminal pair-programming agent | `OPENAI_API_BASE=http://localhost:7770/v1` | Medium | +| Continue.dev (VS Code) | Open-source coding agent | `config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}` | Medium | +| OpenHands (Agent Canvas) | Self-hosted multi-agent coding platform | LiteLLM custom model + Base URL to the endpoint | Medium | +| AnythingLLM | RAG / knowledge chat | Provider "OpenAI (Generic)" -> endpoint | Medium | +| Browser Use | Browser automation agent (Playwright) | `ChatOpenAI(base_url="http://localhost:7770/v1")` | Low | +| Qdrant | Vector store for RAG | Standalone, CPU-native | Low | + +The pragmatic first installs are Open WebUI, Cline, and Aider. They all work +with the current llama.cpp server and need no extra GPU. + +## 2. Inference Runtime Tuning (llama.cpp on gfx1151) + +Verified against the running service and llama.cpp source. + +| Change | Recommendation | Priority | +| ------ | -------------- | -------- | +| Backend | Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU. | Done | +| KV cache | Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed. | Keep | +| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep | +| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional | +| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium | +| DFlash n_max | Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput. | Verify | +| ngram layering | Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled. | Keep off | + +## 3. Server / OS / Hardware Settings + +All require root and a reboot for the modprobe change. + +| Setting | Action | Effect | Priority | +| ------- | ------ | ------ | -------- | +| TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High | +| Power profile | `echo performance > /sys/class/drm/card0/device/power_profile` | Sustained inference speed; apply at boot via a udev/systemd unit | Medium | +| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done | +| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify | +| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify | + +## 4. Other Enhancements + +- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises, + change the env tunable. +- Consider switching the disabled `vllm-thinkingcap.service` unit off or + removing it to avoid confusion (it is loaded in a failed state). +- Profile generation under 3-slot concurrent load before touching boot + settings, so any tuning is measured against the real workload. + +## Sources + +- [[frank-default-large-benchmark]] — measured backend/spec numbers +- [[deployment]] — deploy and update process +- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS) +- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`, + `common/arg.cpp`, `common/common.h` +- NotebookCheck Ryzen AI MAX+ 395 analysis + +## See Also + +- [[frank-default-large-benchmark]] +- [[DFlash DSpark Speculative Decoding]] +- [[deployment]] +- [[index]] diff --git a/docs/index.md b/docs/index.md index e870383..8d90728 100644 --- a/docs/index.md +++ b/docs/index.md @@ -52,6 +52,7 @@ DFlash draft, port 7770] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark that drove the Dankclaw llama.cpp deployment +- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings ## Speculative Decoding & Quantization