Files

T

Frank 3a64c8f3dd docs: fact-check knowledge base against live machines and vLLM source

Correct stale and incorrect facts after exhaustive verification:
- Dankclaw: NUMA node 0 = 31213 MB; disk 661G/1.1T/38%; vLLM 0.26.0 in
  /home/dank/vllm-rocm-venv (not vllm-env); clarify 96 GiB GPU-visible is a
  driver overcommit ceiling over 30 GiB physical (confirmed via sysfs
  mem_info_vram_total=103079215104)
- Dankserver: disk 729G/140G free/84%; /mnt/entropy 910G free; active service
  is vLLM EngineCore (22 GB), not llama-server; CPU model Ryzen 7 7700X
- index.md: Dankclaw 30GiB phys / 96GiB GPU-visible (was '61 GB unified');
  add deployment topology mermaid diagram
- vLLM defaults (from source): gpu_memory_utilization=0.92, max_num_seqs=128,
  swap_space removed (now --kv-offloading-size) across Overview/Index/RAM/Hybrid
- Quantization: FP8 not native on Ampere (RTX 3090) or RDNA 3.5 (gfx1151);
  add per-model hardware quantization notes; AWQ/GPTQ limited on ROCm
- Speculative decoding: add DFlash, DSpark, MTP, EAGLE3 to feature list
- Confirm Qwen3.6-27B and Qwen3.6-35B-A3B specs are correct (HF-verified)
- Production note: Dankclaw primary llama.cpp service runs ROCmFPX
  Ling-3.0-flash RocmFP4 MTP, not vLLM Qwen3.6-27B

2026-08-10 20:59:33 -07:00

3.2 KiB

Raw Blame History

Dankclaw System Specs

Server Overview

Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool.

Hardware Specifications

CPU

Property	Value
Model	AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
Architecture	x86_64
Cores	16 (1 socket, 1 thread per core)
Max MHz	5187.50
Min MHz	625.00
NUMA Nodes	1 (node 0, 31213 MB)

GPU (Integrated)

Property	Value
Model	AMD Radeon 8060S (Strix Halo)
Device ID	0x1586
Architecture	gfx1151 (Strix Halo)
GPU-Visible Memory	96 GiB (driver VRAM ceiling, overcommitted)
Physical RAM	30 GiB (shared CPU + GPU)

ROCm

Property	Value
ROCk Module	Loaded
Runtime Version	1.18
ROCm Version	7.2.4 (from AMD repos)
Agent Name	AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
GPU Arch	gfx1151 (Strix Halo)
GPU Global Memory	96 GiB (driver-reported ceiling; physical is 30 GiB)

Memory

Property	Value
System RAM	30 GiB (shared CPU + GPU)
GPU-Visible (ROCm)	96 GiB (driver ceiling; ROCm pool 1 is 31962252 KB)
Swap	8.0 GiB
/dev/shm	~15 GiB (half of RAM)

Storage

Filesystem	Size	Used	Avail	Use%
/ (nvme0n1p2)	1.9T	661G	1.1T	38%

OS and Software

Property	Value
OS	Ubuntu 26.04 LTS (Resolute Raccoon)
Kernel	7.0.0-29-generic
Python	3.14.4 (system), 3.12.13 (vLLM venv)
vLLM	0.26.0 (in `/home/dank/vllm-rocm-venv`, Python 3.12)
uv	0.12.2

VLLM Relevance

Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
ROCm reports 96 GiB of GPU-visible memory (overcommitted). Physical RAM is 30 GiB. The driver reports a 96 GiB VRAM ceiling (sysfs mem_info_vram_total), but the machine has only 30.5 GB of physical RAM. vLLM sees the large number and will try to use it. Set --gpu-memory-utilization conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see scripts/patches/rocm_patched.py on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
Only 8 GiB swap. Modern vLLM uses --kv-offloading-size instead of the removed --swap-space flag.
The active llama.cpp service on Dankclaw runs on the ROCmFPX runtime (/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server) with the Ling-3.0-flash ROCmFP4 MTP model, not regular llama.cpp.
ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+.
Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv.
Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
1.1 TB free disk space is ample for model storage.
libmpi_cxx.so.40 is missing from the system. Install openmpi-bin or provide a workaround for torch MPI dependency.

3.2 KiB Raw Blame History