Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
122 lines
6.3 KiB
Markdown
122 lines
6.3 KiB
Markdown
---
|
|
type: reference
|
|
description: "This page collects concrete, sourced recommendations to improve Dankclaw"
|
|
tags:
|
|
- llm
|
|
timestamp: 2026-09-03
|
|
---
|
|
|
|
# Dankclaw — Improvements Research
|
|
|
|
This page collects concrete, sourced recommendations to improve Dankclaw
|
|
(the AMD Strix Halo inference box). It covers self-hosted agentic tools,
|
|
inference runtime tuning, and server/OS/hardware settings. Each item lists a
|
|
priority and the exact action.
|
|
|
|
Dates: researched 2026-08. Hardware is verified on the live machine unless
|
|
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
|
|
|
## Corrections (re-verified 2026-08-15/16)
|
|
|
|
- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
|
|
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
|
|
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
|
|
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
|
|
hold on the 8060S. See [[frank-dragoon-benchmark]].
|
|
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
|
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
|
`performance`, forced level `auto`. This item is N/A.
|
|
- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
|
|
- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
|
|
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
|
|
- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and
|
|
persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that
|
|
once held the service at ~15 GiB is gone.
|
|
|
|
## Verified Memory Picture (measured 2026-08-11)
|
|
|
|
| Figure | Value | Source |
|
|
| ------ | ----- | ------ |
|
|
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
|
|
| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` |
|
|
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` |
|
|
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
|
|
|
|
The OS accounts only ~30.5 GiB physical RAM. The TTM GPU-addressable pool
|
|
was capped at ~15.24 GiB (the real pre-2026-08-25 ceiling); that cap is now
|
|
lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
|
|
different, driver-reported figure that has always been larger than the kernel
|
|
cap.
|
|
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
|
|
limits documented in [[frank-dragoon-benchmark]].
|
|
|
|
## 1. Self-Hosted Agentic Tools
|
|
|
|
All of these connect to the local llama.cpp OpenAI-compatible endpoint
|
|
(`http://localhost:7770/v1`). None need NVIDIA.
|
|
|
|
| Tool | Use | Endpoint config | Priority |
|
|
| ---- | --- | --------------- | -------- |
|
|
| Open WebUI | Chat + agent harness, plugins, RAG, tool calling | Add OpenAI-compatible provider -> `http://localhost:7770/v1` | High |
|
|
| Cline (VS Code) | IDE coding agent with MCP | Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1` | High |
|
|
| Aider (CLI) | Terminal pair-programming agent | `OPENAI_API_BASE=http://localhost:7770/v1` | Medium |
|
|
| Continue.dev (VS Code) | Open-source coding agent | `config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}` | Medium |
|
|
| OpenHands (Agent Canvas) | Self-hosted multi-agent coding platform | LiteLLM custom model + Base URL to the endpoint | Medium |
|
|
| AnythingLLM | RAG / knowledge chat | Provider "OpenAI (Generic)" -> endpoint | Medium |
|
|
| Browser Use | Browser automation agent (Playwright) | `ChatOpenAI(base_url="http://localhost:7770/v1")` | Low |
|
|
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
|
|
|
|
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
|
|
with the current llama.cpp server and need no extra GPU.
|
|
|
|
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
|
|
|
|
Verified against the running service and llama.cpp source.
|
|
|
|
| Change | Recommendation | Priority |
|
|
| ------ | -------------- | -------- |
|
|
| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
|
|
| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
|
|
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
|
|
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
|
|
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
|
|
| DFlash n_max | Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput. | Verify |
|
|
| ngram layering | Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled. | Keep off |
|
|
|
|
## 3. Server / OS / Hardware Settings
|
|
|
|
All require root and a reboot for the modprobe change.
|
|
|
|
| Setting | Action | Effect | Priority |
|
|
| ------- | ------ | ------ | -------- |
|
|
| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done |
|
|
| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
|
|
| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
|
|
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
|
|
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
|
|
|
|
## 4. Other Enhancements
|
|
|
|
- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises,
|
|
change the env tunable.
|
|
- Consider switching the disabled `vllm-thinkingcap.service` unit off or
|
|
removing it to avoid confusion (it is loaded in a failed state).
|
|
- Profile generation under 3-slot concurrent load before touching boot
|
|
settings, so any tuning is measured against the real workload.
|
|
|
|
## Sources
|
|
|
|
- [[frank-dragoon-benchmark]] — measured backend/spec numbers
|
|
- [[deployment]] — deploy and update process
|
|
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
|
|
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
|
|
`common/arg.cpp`, `common/common.h`
|
|
- NotebookCheck Ryzen AI MAX+ 395 analysis
|
|
|
|
## See Also
|
|
|
|
- [[frank-dragoon-benchmark]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[deployment]]
|
|
- [[index]]
|