serverAppConfigs/docs/LLM/Shared Memory Architecture Dankclaw.md

# Shared Memory Architecture Dankclaw

## Overview

Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.

## Unified Memory Architecture

### How Unified Memory Works

In unified memory:

1. CPU and GPU share one physical memory pool
2. The OS and GPU driver manage memory allocation
3. There is no separate VRAM or system RAM
4. The total available memory appears as GPU memory to vLLM
5. Memory pages move between CPU and GPU transparently

### Implications for vLLM

- vLLM sees the full memory as GPU memory
- No explicit swap_space parameter needed for memory extension
- The GPU driver handles memory paging automatically
- Performance depends on memory bandwidth and latency
- Shared memory means CPU and GPU compete for the same pool

## ROCm Configuration

### ROCm Version

vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw has ROCm 7.2.4 from AMD repos.

Check ROCm version:

```
rocminfo
```

### GPU Architecture

Strix Halo uses the gfx115x architecture. Check with:

```
rocminfo | grep gfx
```

### Installation

Install VLLM for ROCm with Python 3.12:

```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

### Attention Backend

Choose the best attention backend for Strix Halo:

```
vllm serve model \
    --attention-backend ROCM_AITER_UNIFIED_ATTN
```

Available backends for ROCm:

| Backend | Description | Best For |
|---------|-------------|----------|
| TRITON_ATTN | Triton-based attention | General use |
| ROCM_ATTN | ROCm native attention | MI series GPUs |
| ROCM_AITER_FA | AITER Flash Attention | Performance |
| ROCM_AITER_UNIFIED_ATTN | Unified AITER attention | Strix Halo APU |
| TRITON_MLA | Triton MLA | DeepSeek models |
| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |

## Special Flags for APU

### Memory Configuration

Since memory is unified, set these parameters carefully:

```
vllm serve model \
    --gpu-memory-utilization 0.85 \
    --max-num-batched-tokens 4096
```

- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
- `max_model_len`: Match actual context needs
- The old `--swap-space` argument was removed from current vLLM. Use `--kv-offloading-size` for CPU KV offload

### NUMA Binding

Strix Halo may have multiple NUMA nodes. Use NUMA binding:

```
vllm serve model \
    --numa-bind
```

For explicit binding:

```
vllm serve model \
    --numa-bind \
    --numa-bind-nodes 0 0
```

### Environment Variables

Set these environment variables for optimal performance:

```
export ROCM_DISABLE_TCE=0
export HSA_ENABLE_SDMA=1
export HSA_FORCE_FINE_GRAIN_PCIE=1
```

### GPU Memory Limits

Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:

```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
```

This may be needed for Strix Halo compatibility.

## Performance Considerations

### Memory Bandwidth

Unified memory has lower bandwidth than dedicated VRAM:

| Memory Type | Bandwidth |
|-------------|-----------|
| HBM3 (A100) | 2 TB/s |
| GDDR6 (RTX 4090) | 1 TB/s |
| LPDDR5x (Strix Halo) | ~170 GB/s |

### ROCm Overcommit

ROCm reports 96 GiB GPU memory on Strix Halo APU. Physical RAM is 64 GiB (61 GiB visible; corrected 2026-08-17, the box was upgraded from 32 GB). The GPU can address more than physical RAM exists. vLLM will see 96 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.

Lower bandwidth means slower inference. Plan for lower throughput.

### Cache Effects

Strix Halo has large L3 cache. This helps with memory-bound workloads:

- Large L3 cache reduces memory access latency
- Prefetching helps with sequential KV cache access
- Cache eviction may cause stalls during long contexts

### Thermal Throttling

APUs generate more heat than discrete GPUs:

- Monitor GPU temperature with `rocm-smi`
- Ensure adequate cooling
- Thermal throttling reduces performance
- Consider undervolting for sustained workloads

### Concurrency Limits

Unified memory limits concurrent requests:

- Each request uses memory from the shared pool
- CPU operations also use the shared pool
- Reduce `max_num_seqs` to avoid memory pressure
- Monitor memory usage with `rocm-smi`

## Known Limitations

### Feature Support

| Feature | Support on ROCm |
|---------|-----------------|
| Tensor Parallelism | Yes |
| FP8 Quantization | Depends on GPU |
| CUDA Graphs | Yes |
| Async Output | No |
| Multi-step Decoding | No |
| Speculative Decoding | Yes |
| Prefix Caching | Yes |
| Chunked Prefill | Yes |

### Context Length

Some models on ROCm have context length limited to 4096 tokens. Check model documentation.

### FP8 Support

FP8 quantization depends on GPU architecture:

- MI300 series supports native FP8
- Radeon RX 7900 supports FP8 via software emulation
- Strix Halo (gfx1151) uses **software emulation** for FP8 (no native FP8 tensor cores); expect memory savings, not full compute speedup

For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native ROCm FP4 kernels.

## Example Commands

### Basic APU Deployment

```
vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --swap-space 8 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 64
```

### With FP8 Model

```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8 \
    --swap-space 8
```

### With Long Context

```
vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.8 \
    --swap-space 16 \
    --max-model-len 16384 \
    --rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
```

## Monitoring

Check GPU status:

```
rocm-smi
```

Check ROCm device info:

```
rocminfo
```

Check memory usage:

```
free -h
```

Monitor temperature:

```
rocm-smi --showtemperature
```

## See Also

- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]