serverAppConfigs/docs/Shared Memory Architecture Dankclaw.md

Shared Memory Architecture Dankclaw

Overview

Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.

Unified Memory Architecture

How Unified Memory Works

In unified memory:

1. CPU and GPU share one physical memory pool
2. The OS and GPU driver manage memory allocation
3. There is no separate VRAM or system RAM
4. The total available memory appears as GPU memory to vLLM
5. Memory pages move between CPU and GPU transparently

Implications for vLLM

• vLLM sees the full memory as GPU memory
• No explicit swap_space parameter needed for memory extension
• The GPU driver handles memory paging automatically
• Performance depends on memory bandwidth and latency
• Shared memory means CPU and GPU compete for the same pool

ROCm Configuration

ROCm Version

vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw has ROCm 7.2.4 from AMD repos.

Check ROCm version:

rocminfo

GPU Architecture

Strix Halo uses the gfx115x architecture. Check with:

rocminfo | grep gfx

Installation

Install VLLM for ROCm with Python 3.12:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Attention Backend

Choose the best attention backend for Strix Halo:

vllm serve model \
    --attention-backend ROCM_AITER_UNIFIED_ATTN

Available backends for ROCm:

Backend	Description	Best For
TRITON_ATTN	Triton-based attention	General use
ROCM_ATTN	ROCm native attention	MI series GPUs
ROCM_AITER_FA	AITER Flash Attention	Performance
ROCM_AITER_UNIFIED_ATTN	Unified AITER attention	Strix Halo APU
TRITON_MLA	Triton MLA	DeepSeek models
ROCM_AITER_MLA	AITER MLA	DeepSeek models
ROCM_AITER_TRITON_MLA	Combined MLA	DeepSeek models

Special Flags for APU

Memory Configuration

Since memory is unified, set these parameters carefully:

vllm serve model \
    --gpu-memory-utilization 0.85 \
    --max-num-batched-tokens 4096

• gpu_memory_utilization: Set lower (0.8-0.85) to leave room for CPU operations
• max_model_len: Match actual context needs
• The old --swap-space argument was removed from current vLLM. Use --kv-offloading-size for CPU KV offload

NUMA Binding

Strix Halo may have multiple NUMA nodes. Use NUMA binding:

vllm serve model \
    --numa-bind

For explicit binding:

vllm serve model \
    --numa-bind \
    --numa-bind-nodes 0 0

Environment Variables

Set these environment variables for optimal performance:

export ROCM_DISABLE_TCE=0
export HSA_ENABLE_SDMA=1
export HSA_FORCE_FINE_GRAIN_PCIE=1

GPU Memory Limits

Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:

export HSA_OVERRIDE_GFX_VERSION=11.5.1

This may be needed for Strix Halo compatibility.

Performance Considerations

Memory Bandwidth

Unified memory has lower bandwidth than dedicated VRAM:

Memory Type	Bandwidth
HBM3 (A100)	2 TB/s
GDDR6 (RTX 4090)	1 TB/s
LPDDR5x (Strix Halo)	~170 GB/s

ROCm Overcommit

ROCm reports 96 GiB GPU memory on Strix Halo APU. Physical RAM is 30 GiB. The GPU can address more than physical RAM exists. vLLM will see 96 GiB and try to allocate accordingly. Set --gpu-memory-utilization to 0.80 or lower to prevent OOM.

Lower bandwidth means slower inference. Plan for lower throughput.

Cache Effects

Strix Halo has large L3 cache. This helps with memory-bound workloads:

• Large L3 cache reduces memory access latency
• Prefetching helps with sequential KV cache access
• Cache eviction may cause stalls during long contexts

Thermal Throttling

APUs generate more heat than discrete GPUs:

• Monitor GPU temperature with rocm-smi
• Ensure adequate cooling
• Thermal throttling reduces performance
• Consider undervolting for sustained workloads

Concurrency Limits

Unified memory limits concurrent requests:

• Each request uses memory from the shared pool
• CPU operations also use the shared pool
• Reduce max_num_seqs to avoid memory pressure
• Monitor memory usage with rocm-smi

Known Limitations

Feature Support

Feature	Support on ROCm
Tensor Parallelism	Yes
FP8 Quantization	Depends on GPU
CUDA Graphs	Yes
Async Output	No
Multi-step Decoding	No
Speculative Decoding	Yes
Prefix Caching	Yes
Chunked Prefill	Yes

Context Length

Some models on ROCm have context length limited to 4096 tokens. Check model documentation.

FP8 Support

FP8 quantization depends on GPU architecture:

• MI300 series supports native FP8
• Radeon RX 7900 supports FP8 via software emulation
• Strix Halo (gfx1151) uses software emulation for FP8 (no native FP8 tensor cores); expect memory savings, not full compute speedup

For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (Q2_0_ROCMFP4) provides native ROCm FP4 kernels.

Example Commands

Basic APU Deployment

vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 64

With FP8 Model

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8 \
    --swap-space 8

With Long Context

vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.8 \
    --swap-space 16 \
    --max-model-len 16384 \
    --rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'

Monitoring

Check GPU status:

rocm-smi

Check ROCm device info:

rocminfo

Check memory usage:

free -h

Monitor temperature:

rocm-smi --showtemperature

See Also

• VLLM Overview
• VLLM Index
• RAM CPU Offloading in VLLM
• Hybrid CPU GPU Processing Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw