Files

T

Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.

2026-10-18 13:15:43 -07:00

8.0 KiB

Raw Permalink Blame History

type, description, tags, timestamp

type

description

Dankclaw — Improvements Research

This page collects concrete, sourced recommendations to improve Dankclaw (the AMD Strix Halo inference box). It covers self-hosted agentic tools, inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action.

Dates: researched 2026-08. Hardware is verified on the live machine unless marked "verify first". Re-verified 2026-08-15/16; corrections marked below.

Live today (2026-10-18). The era described below — llama.cpp on the 8060S iGPU behind :7770 — is retired. Dankclaw now serves frank-dragoon-flash-server-v2-native (halogen-flash-server 0.16.2, port 7770) and frank-zealot-llama-prism-bonsai2-pq20 (llama.cpp PrismML fork on the RTX 3090, port 7771 — drift today: that unit is failed since 2026-10-18 03:40 and a hand-run 9B-MTP process answers the port), and boots iommu=pt since 2026-10-18 (dankclaw-iommu-translation-off). The numbers below are the measurements of the era they were taken in; the ttm ceiling and the NVMe scheduler rows were re-checked 2026-10-18 and still hold.

Corrections (re-verified 2026-08-15/16)

Backend: that era's production was Vulkan0, not ROCm. Measured on this box (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs 302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S. See frank-dragoon-benchmark.
Power profile: the /sys/class/drm/card0/device/power_profile file does not exist on this machine (checked 2026-08-15). GPU DPM is already performance, forced level auto. This item is N/A.
KV cache: production ran q8_0 K and q8_0 V (both quantized). The law still holds on frank-zealot (CACHE_TYPE_K=q8_0, CACHE_TYPE_V=q8_0 — scripts/frank-zealot.env:55-56).
Spec: DFlash/DSpark are retired. That era's production was --spec-default (ngram-mod) with draft-mtp first. See qwen3.8-spec-matrix. Today frank-zealot is configured with speculation off (SPEC_TYPE=none — scripts/frank-zealot.env:67).
TTM: raised to 114 GiB on 2026-08-25, set live (no reboot) and persisted in /etc/modprobe.d/ttm.conf. The effective VRAM ceiling that once held the service at ~15 GiB is gone.

Verified Memory Picture (measured 2026-08-11)

Figure	Value	Source
OS physical RAM	30.5 GiB	`/proc/meminfo`
TTM pages_limit	114 GiB (29,883,536 x 4KiB) — raised 2026-08-25	`/sys/module/ttm/parameters/pages_limit`
ROCm VRAM total	96.0 GiB	`rocm-smi --showmeminfo vram` under the then firmware VGM carve
ROCm VRAM used	76.2 GiB	`rocm-smi` while serving 3x200700

That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as "dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is recorded in ling-3.0-flash-trial §Why Vulkan + GTT). The OS now accounts 121 GiB (MemTotal 127108080 kB, 2026-10-18) and rocm-smi --showmeminfo vram reports 2.0 GiB (2,147,483,648 B) of dedicated VRAM — the carve is gone and the GPU budget is the GTT aperture, not the SMI figure (Dankclaw System Specs). The TTM row above is still live: pages_limit = 29883536 (114 GiB) and /etc/modprobe.d/ttm.conf still carries the options ttm line (both read 2026-10-18).

This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in frank-dragoon-benchmark.

1. Self-Hosted Agentic Tools

All of these connect to the local OpenAI-compatible endpoint (http://127.0.0.1:7770/v1). Today that port is halogen-flash-server, not a llama.cpp server, and the address must be the IPv4 literal: curl http://localhost:7770/v1/models returns nothing while http://127.0.0.1:7770/v1/models answers 200 (checked 2026-10-18 — the rootless podman publish is IPv4-only). None need NVIDIA.

Tool	Use	Endpoint config	Priority
Open WebUI	Chat + agent harness, plugins, RAG, tool calling	Add OpenAI-compatible provider -> `http://localhost:7770/v1`	High
Cline (VS Code)	IDE coding agent with MCP	Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1`	High
Aider (CLI)	Terminal pair-programming agent	`OPENAI_API_BASE=http://localhost:7770/v1`	Medium
Continue.dev (VS Code)	Open-source coding agent	`config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}`	Medium
OpenHands (Agent Canvas)	Self-hosted multi-agent coding platform	LiteLLM custom model + Base URL to the endpoint	Medium
AnythingLLM	RAG / knowledge chat	Provider "OpenAI (Generic)" -> endpoint	Medium
Browser Use	Browser automation agent (Playwright)	`ChatOpenAI(base_url="http://localhost:7770/v1")`	Low
Qdrant	Vector store for RAG	Standalone, CPU-native	Low

The pragmatic first installs are Open WebUI, Cline, and Aider. They all work with the endpoint on 7770 and need no extra GPU. The 7770 endpoint today is halogen-flash-server, not llama.cpp.

2. Inference Runtime Tuning (llama.cpp on gfx1151)

Verified against the running service and llama.cpp source.

Change	Recommendation	Priority
Backend	Vulkan0 (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads.	Done
KV cache	`q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed.	Keep
Flash attention	Leave `--flash-attn auto`. It helps prompt processing, not decode.	Keep
Load mode	Start with mmap (default). Enable `--mlock` only if page-fault stutter appears.	Optional
Threads	Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver.	Medium
DFlash n_max	Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput.	Verify
ngram layering	Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled.	Keep off

3. Server / OS / Hardware Settings

All require root and a reboot for the modprobe change.

Setting	Action	Effect	Priority
TTM GPU pool	`/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted)	Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot	Done
Power profile	N/A on this machine (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance`	none	~~Medium~~
NVMe scheduler	Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18)	No change	Done
BIOS VGM	Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS	Max dedicated VRAM for the GPU carve-out; verify in the live BIOS	Verify
Firmware/EC	Check GMKtec for a newer EC/fan-curve firmware	Better sustained thermal behavior	Verify

4. Other Enhancements

Test --spec-draft-n-max 5 and re-benchmark acceptance; if throughput rises, change the env tunable.
The vllm-thinkingcap.service unit this item told you to clean up is gone: systemctl is-enabled vllm-thinkingcap.service → not-found (2026-10-18).
Profile generation under 3-slot concurrent load before touching boot settings, so any tuning is measured against the real workload.

Sources

frank-dragoon-benchmark — measured backend/spec numbers
deployment — deploy and update process
AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
llama.cpp source: ggml/CMakeLists.txt, ggml/src/ggml-hip/CMakeLists.txt, common/arg.cpp, common/common.h
NotebookCheck Ryzen AI MAX+ 395 analysis

8.0 KiB Raw Permalink Blame History