Frank
04f4e2c6d4
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
140 lines
7.7 KiB
Markdown
140 lines
7.7 KiB
Markdown
---
|
|
type: reference
|
|
description: "This page collects concrete, sourced recommendations to improve Dankclaw"
|
|
tags:
|
|
- llm
|
|
timestamp: 2026-09-03
|
|
---
|
|
|
|
# Dankclaw — Improvements Research
|
|
|
|
This page collects concrete, sourced recommendations to improve Dankclaw
|
|
(the AMD Strix Halo inference box). It covers self-hosted agentic tools,
|
|
inference runtime tuning, and server/OS/hardware settings. Each item lists a
|
|
priority and the exact action.
|
|
|
|
Dates: researched 2026-08. Hardware is verified on the live machine unless
|
|
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
|
|
|
**Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
|
|
iGPU — is retired; what runs on this box now (both endpoints, their engines and
|
|
ports) is [[frank-dragoon]]'s and [[frank-zealot]]'s pages. Drift worth knowing
|
|
before trusting either page: the frank-zealot unit is `failed` since 2026-10-18
|
|
03:40 and a hand-run 9B-MTP process answers its port. The OS facts are this
|
|
page's: `iommu=pt` since 2026-10-18 ([[dankclaw-iommu-translation-off]]), and
|
|
the `ttm` ceiling and NVMe scheduler rows below were re-checked 2026-10-18 and
|
|
still hold. Every benchmark number below is a measurement of the era it was
|
|
taken in.
|
|
|
|
## Corrections (re-verified 2026-08-15/16)
|
|
|
|
- **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm
|
|
wins prefill and Vulkan wins decode, and the decode gap widens with context:
|
|
the 64k A/B is [[qwen3.8-strix-optimizations]] §4, the 262k A/B is
|
|
[[qwen3.8-spec-matrix]] §High-Context Backend Comparison. The "HIP
|
|
out-performs Vulkan for decode" claim did not hold on the 8060S.
|
|
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
|
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
|
`performance`, forced level `auto`. This item is N/A.
|
|
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What
|
|
`frank-zealot` runs now is its own Current shape — this row is not it.
|
|
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
|
|
(ngram-mod) with draft-mtp first — timeline on [[qwen3.8-spec-matrix]]; the
|
|
spec setting each endpoint runs today is on its own page.
|
|
- **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded
|
|
once on [[Dankclaw System Specs]] (Memory).
|
|
|
|
## Verified Memory Picture (measured 2026-08-11)
|
|
|
|
| Figure | Value | Source |
|
|
| ------ | ----- | ------ |
|
|
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
|
|
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve |
|
|
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
|
|
|
|
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
|
|
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
|
|
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT; the 76.2 GiB row is an
|
|
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`,
|
|
the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated
|
|
carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read
|
|
2026-10-18.
|
|
|
|
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
|
|
limits documented in [[frank-dragoon]].
|
|
|
|
## 1. Self-Hosted Agentic Tools
|
|
|
|
All of these connect to the local OpenAI-compatible endpoint on
|
|
`:7770` (which engine owns that port is [[frank-dragoon]]'s to say), and the
|
|
address must be the IPv4 literal: `curl http://localhost:7770/v1/models` returns
|
|
nothing while `http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18
|
|
— the rootless podman publish is IPv4-only). None need NVIDIA.
|
|
|
|
| Tool | Use | Endpoint config | Priority |
|
|
| ---- | --- | --------------- | -------- |
|
|
| Open WebUI | Chat + agent harness, plugins, RAG, tool calling | Add OpenAI-compatible provider -> `http://localhost:7770/v1` | High |
|
|
| Cline (VS Code) | IDE coding agent with MCP | Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1` | High |
|
|
| Aider (CLI) | Terminal pair-programming agent | `OPENAI_API_BASE=http://localhost:7770/v1` | Medium |
|
|
| Continue.dev (VS Code) | Open-source coding agent | `config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}` | Medium |
|
|
| OpenHands (Agent Canvas) | Self-hosted multi-agent coding platform | LiteLLM custom model + Base URL to the endpoint | Medium |
|
|
| AnythingLLM | RAG / knowledge chat | Provider "OpenAI (Generic)" -> endpoint | Medium |
|
|
| Browser Use | Browser automation agent (Playwright) | `ChatOpenAI(base_url="http://localhost:7770/v1")` | Low |
|
|
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
|
|
|
|
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
|
|
with the endpoint on 7770 and need no extra GPU.
|
|
|
|
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
|
|
|
|
Verified against that era's running service and llama.cpp source; the era is
|
|
retired ([[frank-dragoon]]), so this table is history and a method, not a live
|
|
TODO list.
|
|
|
|
| Change | Recommendation | Priority |
|
|
| ------ | -------------- | -------- |
|
|
| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
|
|
| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
|
|
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
|
|
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
|
|
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
|
|
| DFlash n_max | Depth 8 (block 16) was high; a shallower draft cuts rejection cost at the same throughput. Sweep depth per the A/B protocol in [[DFlash DSpark Speculative Decoding]]. | Verify |
|
|
| ngram layering | Layering ngram on the DFlash draft measured slower than the draft alone at 4k — figures on [[DFlash DSpark Speculative Decoding]]. | Keep off |
|
|
|
|
## 3. Server / OS / Hardware Settings
|
|
|
|
All require root and a reboot for the modprobe change.
|
|
|
|
| Setting | Action | Effect | Priority |
|
|
| ------- | ------ | ------ | -------- |
|
|
| TTM GPU pool | Raised 2026-08-25; the knob, its value and its persistence file are [[Dankclaw System Specs]]'s (Memory row) | Lifted the ~15 GiB kernel-page ceiling so a full 262k KV plus a ~30 GB payload fits one slot | Done |
|
|
| Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ |
|
|
| NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done |
|
|
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
|
|
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
|
|
|
|
## 4. Other Enhancements
|
|
|
|
- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises,
|
|
change the env tunable.
|
|
- The `vllm-thinkingcap.service` unit this item told you to clean up is gone:
|
|
`systemctl is-enabled vllm-thinkingcap.service` → `not-found` (2026-10-18).
|
|
- Profile generation under 3-slot concurrent load before touching boot
|
|
settings, so any tuning is measured against the real workload.
|
|
|
|
## Sources
|
|
|
|
- [[frank-dragoon]] — measured backend/spec numbers
|
|
- [[deployment]] — deploy and update process
|
|
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
|
|
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
|
|
`common/arg.cpp`, `common/common.h`
|
|
- NotebookCheck Ryzen AI MAX+ 395 analysis
|
|
|
|
## See Also
|
|
|
|
- [[frank-dragoon]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[deployment]]
|
|
- [[index]]
|