serverAppConfigs/serverAppConfigs-docs/LLM/Shared Memory Architecture Dankclaw.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Dankclaw uses AMD Strix Halo APU with unified memory architecture.	llm	2026-08-28

Shared Memory Architecture Dankclaw

Overview

Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.

Unified Memory Architecture

How Unified Memory Works

In unified memory:

1. CPU and GPU share one physical memory pool
2. The OS and GPU driver manage memory allocation
3. There is no separate VRAM or system RAM
4. The total available memory appears as GPU memory to vLLM
5. Memory pages move between CPU and GPU transparently

Implications for vLLM

• vLLM sees the full memory as GPU memory
• No explicit swap_space parameter needed for memory extension
• The GPU driver handles memory paging automatically
• Performance depends on memory bandwidth and latency
• Shared memory means CPU and GPU compete for the same pool

ROCm Configuration

ROCm Version

vLLM supports AMD GPUs with ROCm 6.3 or above (unverified 2026-10-18). Dankclaw has ROCm 7.2.4 from AMD repos (/etc/apt/sources.list.d/rocm.list → deb https://repo.radeon.com/rocm/apt/7.2.4 noble main; rocm-core 7.2.4.70204-93~24.04, read 2026-10-18). No vLLM endpoint runs on this host today — 7770 is halogen-flash-server and 7771 is llama.cpp (Dankclaw System Specs) — so this page is reference material, not the serving path.

Check ROCm version:

rocminfo

GPU Architecture

Strix Halo uses the gfx115x architecture. Check with:

rocminfo | grep gfx

Installation

Install VLLM for ROCm with Python 3.12:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Attention Backend

Choose the best attention backend for Strix Halo:

vllm serve model \
    --attention-backend ROCM_AITER_UNIFIED_ATTN

Available backends for ROCm:

Backend	Description	Best For
TRITON_ATTN	Triton-based attention	General use
ROCM_ATTN	ROCm native attention	MI series GPUs
ROCM_AITER_FA	AITER Flash Attention	Performance
ROCM_AITER_UNIFIED_ATTN	Unified AITER attention	Strix Halo APU
TRITON_MLA	Triton MLA	DeepSeek models
ROCM_AITER_MLA	AITER MLA	DeepSeek models
ROCM_AITER_TRITON_MLA	Combined MLA	DeepSeek models

All seven names exist on current vLLM main (AttentionBackendEnum in vllm/v1/attention/backends/registry.py — source, read 2026-10-18), and --attention-backend, --numa-bind, --numa-bind-nodes are real vllm serve flags (vllm/engine/arg_utils.py:1098,1171-1176).

Special Flags for APU

Memory Configuration

Since memory is unified, set these parameters carefully:

vllm serve model \
    --gpu-memory-utilization 0.85 \
    --max-num-batched-tokens 4096

• gpu_memory_utilization: Set lower (0.8-0.85) to leave room for CPU operations
• max_model_len: Match actual context needs
• The old --swap-space argument was removed from current vLLM — CacheConfig on main carries no swap_space field (source, read 2026-10-18). Use --kv-offloading-size (kv_offloading_size, same file) for CPU KV offload

NUMA Binding

Strix Halo may have multiple NUMA nodes. Use NUMA binding:

vllm serve model \
    --numa-bind

For explicit binding:

vllm serve model \
    --numa-bind \
    --numa-bind-nodes 0 0

Environment Variables

Set these environment variables for optimal performance:

export ROCM_DISABLE_TCE=0
export HSA_ENABLE_SDMA=1
export HSA_FORCE_FINE_GRAIN_PCIE=1

These three variables came out of the September research pass and were not re-checked against current ROCm documentation (unverified 2026-10-18).

GPU "memory limits" and the gfx override

There is no memory-limit knob for the APU here. What the GPU may address is the GTT aperture (mem_info_gtt_total, see §ROCm Overcommit); the firmware VGM/UMA setting in BIOS is what sizes the dedicated carve-out.

HSA_OVERRIDE_GFX_VERSION is not a memory setting — it makes the runtime report a different ISA version:

export HSA_OVERRIDE_GFX_VERSION=11.5.1

Do not set it on this host: the stack already reports gfx1151 natively (rocm-smi --showproductname → GFX Version: gfx1151, read 2026-10-18). The override exists for GPUs whose ISA the installed runtime does not know.

Performance Considerations

Memory Bandwidth

Unified memory has lower bandwidth than dedicated VRAM:

Memory Type	Bandwidth
HBM2e (A100 80 GB)	2 TB/s
GDDR6 (RTX 4090)	1 TB/s
LPDDR5x-8000, 256-bit (Strix Halo)	256 GB/s (amd.com spec page)

ROCm Overcommit

ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture: mem_info_gtt_total = 122402963456 bytes = 114 GiB, read 2026-10-18 under /sys/class/drm/card1/device/; llama-server --list-devices shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM visible to the OS is 121 GiB today (free -g total 121, read 2026-10-18; 64 GiB in the earlier reading on this page, upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set --gpu-memory-utilization to 0.80 or lower to prevent OOM.

Lower bandwidth means slower inference. Plan for lower throughput.

Cache Effects

Strix Halo has large L3 cache. This helps with memory-bound workloads:

• Large L3 cache reduces memory access latency
• Prefetching helps with sequential KV cache access
• Cache eviction may cause stalls during long contexts

Thermal Throttling

APUs generate more heat than discrete GPUs:

• Monitor GPU temperature with rocm-smi
• Ensure adequate cooling
• Thermal throttling reduces performance
• Consider undervolting for sustained workloads

Concurrency Limits

Unified memory limits concurrent requests:

• Each request uses memory from the shared pool
• CPU operations also use the shared pool
• Reduce max_num_seqs to avoid memory pressure
• Monitor memory usage with rocm-smi

Known Limitations

Feature Support

Feature	Support on ROCm
Tensor Parallelism	Yes
FP8 Quantization	Depends on GPU
CUDA Graphs	Yes
Async Output	No
Multi-step Decoding	No
Speculative Decoding	Yes
Prefix Caching	Yes
Chunked Prefill	Yes

The two "No" rows (Async Output, Multi-step Decoding) were transcribed from a vLLM feature-support snapshot and not re-checked against current upstream (unverified 2026-10-18).

Context Length

Some models on ROCm have context length limited to 4096 tokens. Check model documentation.

FP8 Support

FP8 quantization depends on GPU architecture:

• MI300 series supports native FP8
• Radeon RX 7900 supports FP8 via software emulation
• Strix Halo (gfx1151) uses software emulation for FP8 matmuls — RDNA 3.5 has no FP8 tensor instructions (FP8-ROCM); expect memory savings, not full compute speedup. (Conflicting vendor-side evidence exists for an FP8/INT8 WMMA path used by FSR4 on RDNA 3.5, so treat the matmul claim as the settled half.)

For FP4 on Dankclaw the ROCmFPX llama.cpp fork (Q2_0_ROCMFP4) provided native ROCm FP4 kernels; that runtime line is retired (CIRU/Kairic eras, frank-dragoon, frank-dragoon) and no ROCmFPX build serves this host today.

Example Commands

Basic APU Deployment

vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 64

With FP8 Model

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8

With Long Context

vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.8 \
    --max-model-len 16384 \
    --rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'

Monitoring

Check GPU status:

rocm-smi

Check ROCm device info:

rocminfo

Check memory usage:

free -h

Monitor temperature:

rocm-smi --showtemperature

See Also

• VLLM Overview
• VLLM Index
• RAM CPU Offloading in VLLM
• Hybrid CPU GPU Processing Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw