Frank
327e78b950
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
138 lines
7.6 KiB
Markdown
138 lines
7.6 KiB
Markdown
---
|
|
type: reference
|
|
description: "This page collects concrete, sourced recommendations to improve Dankclaw"
|
|
tags:
|
|
- llm
|
|
timestamp: 2026-09-03
|
|
---
|
|
|
|
# Dankclaw — Improvements Research
|
|
|
|
This page collects concrete, sourced recommendations to improve Dankclaw
|
|
(the AMD Strix Halo inference box). It covers self-hosted agentic tools,
|
|
inference runtime tuning, and server/OS/hardware settings. Each item lists a
|
|
priority and the exact action.
|
|
|
|
Dates: researched 2026-08. Hardware is verified on the live machine unless
|
|
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
|
|
|
**Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
|
|
iGPU — is retired; what runs on this box now (both endpoints, their engines and
|
|
ports) is [[frank-dragoon]]'s and [[frank-zealot]]'s pages, and both run under their
|
|
repo units today — no drift open as of 2026-10-19 (`frank-zealot` active since
|
|
2026-10-18 21:06, the deployer's 2-slot reslice; `frank-dragoon` active). The OS facts
|
|
are this page's: `iommu=pt` since 2026-10-18 ([[dankclaw-iommu-translation-off]]), and
|
|
the `ttm` ceiling and NVMe scheduler rows below were re-checked 2026-10-18 and
|
|
still hold. Every benchmark number below is a measurement of the era it was
|
|
taken in.
|
|
|
|
## Corrections (re-verified 2026-08-15/16)
|
|
|
|
- **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm
|
|
wins prefill and Vulkan wins decode, and the decode gap widens with context:
|
|
the 64k A/B is §4, the 262k A/B is
|
|
§High-Context Backend Comparison. The "HIP
|
|
out-performs Vulkan for decode" claim did not hold on the 8060S.
|
|
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
|
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
|
`performance`, forced level `auto`. This item is N/A.
|
|
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What
|
|
`frank-zealot` runs now is its own Current shape — this row is not it.
|
|
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
|
|
(ngram-mod) with draft-mtp first — timeline on ; the
|
|
spec setting each endpoint runs today is on its own page.
|
|
- **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded
|
|
once on [[Dankclaw System Specs]] (Memory).
|
|
|
|
## Verified Memory Picture (measured 2026-08-11)
|
|
|
|
| Figure | Value | Source |
|
|
| ------ | ----- | ------ |
|
|
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
|
|
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve |
|
|
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
|
|
|
|
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
|
|
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
|
|
recorded in §Why Vulkan + GTT; the 76.2 GiB row is an
|
|
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`
|
|
the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated
|
|
carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read
|
|
2026-10-18.
|
|
|
|
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
|
|
limits documented in [[frank-dragoon]].
|
|
|
|
## 1. Self-Hosted Agentic Tools
|
|
|
|
All of these connect to the local OpenAI-compatible endpoint on
|
|
`:7770` (which engine owns that port is [[frank-dragoon]]'s to say), and the
|
|
address must be the IPv4 literal: `curl http://localhost:7770/v1/models` returns
|
|
nothing while `http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18
|
|
— the rootless podman publish is IPv4-only). None need NVIDIA.
|
|
|
|
| Tool | Use | Endpoint config | Priority |
|
|
| ---- | --- | --------------- | -------- |
|
|
| Open WebUI | Chat + agent harness, plugins, RAG, tool calling | Add OpenAI-compatible provider -> `http://localhost:7770/v1` | High |
|
|
| Cline (VS Code) | IDE coding agent with MCP | Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1` | High |
|
|
| Aider (CLI) | Terminal pair-programming agent | `OPENAI_API_BASE=http://localhost:7770/v1` | Medium |
|
|
| Continue.dev (VS Code) | Open-source coding agent | `config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}` | Medium |
|
|
| OpenHands (Agent Canvas) | Self-hosted multi-agent coding platform | LiteLLM custom model + Base URL to the endpoint | Medium |
|
|
| AnythingLLM | RAG / knowledge chat | Provider "OpenAI (Generic)" -> endpoint | Medium |
|
|
| Browser Use | Browser automation agent (Playwright) | `ChatOpenAI(base_url="http://localhost:7770/v1")` | Low |
|
|
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
|
|
|
|
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
|
|
with the endpoint on 7770 and need no extra GPU.
|
|
|
|
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
|
|
|
|
Verified against that era's running service and llama.cpp source; the era is
|
|
retired ([[frank-dragoon]]), so this table is history and a method, not a live
|
|
TODO list.
|
|
|
|
| Change | Recommendation | Priority |
|
|
| ------ | -------------- | -------- |
|
|
| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
|
|
| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
|
|
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
|
|
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
|
|
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
|
|
| DFlash n_max | Depth 8 (block 16) was high; a shallower draft cuts rejection cost at the same throughput. Sweep depth per the A/B protocol in [[DFlash DSpark Speculative Decoding]]. | Verify |
|
|
| ngram layering | Layering ngram on the DFlash draft measured slower than the draft alone at 4k — figures on [[DFlash DSpark Speculative Decoding]]. | Keep off |
|
|
|
|
## 3. Server / OS / Hardware Settings
|
|
|
|
All require root and a reboot for the modprobe change.
|
|
|
|
| Setting | Action | Effect | Priority |
|
|
| ------- | ------ | ------ | -------- |
|
|
| TTM GPU pool | Raised 2026-08-25; the knob, its value and its persistence file are [[Dankclaw System Specs]]'s (Memory row) | Lifted the ~15 GiB kernel-page ceiling so a full 262k KV plus a ~30 GB payload fits one slot | Done |
|
|
| Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ |
|
|
| NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done |
|
|
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
|
|
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
|
|
|
|
## 4. Other Enhancements
|
|
|
|
- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises,
|
|
change the env tunable.
|
|
- The `vllm-thinkingcap.service` unit this item told you to clean up is gone:
|
|
`systemctl is-enabled vllm-thinkingcap.service` → `not-found` (2026-10-18).
|
|
- Profile generation under 3-slot concurrent load before touching boot
|
|
settings, so any tuning is measured against the real workload.
|
|
|
|
## Sources
|
|
|
|
- [[frank-dragoon]] — measured backend/spec numbers
|
|
- deploy and update process
|
|
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
|
|
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`
|
|
`common/arg.cpp`, `common/common.h`
|
|
- NotebookCheck Ryzen AI MAX+ 395 analysis
|
|
|
|
## See Also
|
|
|
|
- [[frank-dragoon]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|