serverAppConfigs/serverAppConfigs-docs/LLM/Shared Memory Architecture Dankclaw.md

---
type: reference
description: "Dankclaw uses AMD Strix Halo APU with unified memory architecture."
tags:
  - llm
timestamp: 2026-08-28
---

# Shared Memory Architecture Dankclaw

## Overview

Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.

## Unified Memory Architecture

### How Unified Memory Works

In unified memory:

1. CPU and GPU share one physical memory pool
2. The OS and GPU driver manage memory allocation
3. There is no separate VRAM or system RAM
4. The total available memory appears as GPU memory to vLLM
5. Memory pages move between CPU and GPU transparently

### Implications for vLLM

- vLLM sees the full memory as GPU memory
- No explicit swap_space parameter needed for memory extension
- The GPU driver handles memory paging automatically
- Performance depends on memory bandwidth and latency
- Shared memory means CPU and GPU compete for the same pool

## ROCm Configuration

### ROCm Version

vLLM supports AMD GPUs with ROCm 6.3 or above *(unverified 2026-10-18)*. Dankclaw has ROCm 7.2.4 from AMD repos (`/etc/apt/sources.list.d/rocm.list` → `deb https://repo.radeon.com/rocm/apt/7.2.4 noble main`; `rocm-core 7.2.4.70204-93~24.04`, read 2026-10-18). No vLLM endpoint runs on this host today — 7770 is halogen-flash-server and 7771 is llama.cpp ([[Dankclaw System Specs]]) — so this page is reference material, not the serving path.

Check ROCm version:

```
rocminfo
```

### GPU Architecture

Strix Halo uses the gfx115x architecture. Check with:

```
rocminfo | grep gfx
```

### Installation

Install VLLM for ROCm with Python 3.12:

```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

### Attention Backend

Choose the best attention backend for Strix Halo:

```
vllm serve model \
    --attention-backend ROCM_AITER_UNIFIED_ATTN
```

Available backends for ROCm:

| Backend | Description | Best For |
|---------|-------------|----------|
| TRITON_ATTN | Triton-based attention | General use |
| ROCM_ATTN | ROCm native attention | MI series GPUs |
| ROCM_AITER_FA | AITER Flash Attention | Performance |
| ROCM_AITER_UNIFIED_ATTN | Unified AITER attention | Strix Halo APU |
| TRITON_MLA | Triton MLA | DeepSeek models |
| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |

All seven names exist on current vLLM `main` (`AttentionBackendEnum` in
`vllm/v1/attention/backends/registry.py` —
[source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/attention/backends/registry.py),
read 2026-10-18), and `--attention-backend`, `--numa-bind`, `--numa-bind-nodes`
are real `vllm serve` flags (`vllm/engine/arg_utils.py:1098,1171-1176`).

## Special Flags for APU

### Memory Configuration

Since memory is unified, set these parameters carefully:

```
vllm serve model \
    --gpu-memory-utilization 0.85 \
    --max-num-batched-tokens 4096
```

- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
- `max_model_len`: Match actual context needs
- The old `--swap-space` argument was removed from current vLLM — `CacheConfig` on `main` carries no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) for CPU KV offload

### NUMA Binding

Strix Halo may have multiple NUMA nodes. Use NUMA binding:

```
vllm serve model \
    --numa-bind
```

For explicit binding:

```
vllm serve model \
    --numa-bind \
    --numa-bind-nodes 0 0
```

### Environment Variables

Set these environment variables for optimal performance:

```
export ROCM_DISABLE_TCE=0
export HSA_ENABLE_SDMA=1
export HSA_FORCE_FINE_GRAIN_PCIE=1
```

These three variables came out of the September research pass and were not
re-checked against current ROCm documentation *(unverified 2026-10-18)*.

### GPU "memory limits" and the gfx override

There is no memory-limit knob for the APU here. What the GPU may address is the
GTT aperture (`mem_info_gtt_total`, see §ROCm Overcommit); the firmware VGM/UMA
setting in BIOS is what sizes the *dedicated* carve-out.

`HSA_OVERRIDE_GFX_VERSION` is not a memory setting — it makes the runtime report
a different ISA version:

```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
```

Do not set it on this host: the stack already reports `gfx1151` natively
(`rocm-smi --showproductname` → `GFX Version: gfx1151`, read 2026-10-18). The
override exists for GPUs whose ISA the installed runtime does not know.

## Performance Considerations

### Memory Bandwidth

Unified memory has lower bandwidth than dedicated VRAM:

| Memory Type | Bandwidth |
|-------------|-----------|
| HBM2e (A100 80 GB) | 2 TB/s |
| GDDR6 (RTX 4090) | 1 TB/s |
| LPDDR5x-8000, 256-bit (Strix Halo) | 256 GB/s ([amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)) |

### ROCm Overcommit

ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture: `mem_info_gtt_total` = 122402963456 bytes = 114 GiB, read 2026-10-18 under `/sys/class/drm/card1/device/`; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM visible to the OS is **121 GiB** today (`free -g` total 121, read 2026-10-18; 64 GiB in the earlier reading on this page, upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.

Lower bandwidth means slower inference. Plan for lower throughput.

### Cache Effects

Strix Halo has large L3 cache. This helps with memory-bound workloads:

- Large L3 cache reduces memory access latency
- Prefetching helps with sequential KV cache access
- Cache eviction may cause stalls during long contexts

### Thermal Throttling

APUs generate more heat than discrete GPUs:

- Monitor GPU temperature with `rocm-smi`
- Ensure adequate cooling
- Thermal throttling reduces performance
- Consider undervolting for sustained workloads

### Concurrency Limits

Unified memory limits concurrent requests:

- Each request uses memory from the shared pool
- CPU operations also use the shared pool
- Reduce `max_num_seqs` to avoid memory pressure
- Monitor memory usage with `rocm-smi`

## Known Limitations

### Feature Support

| Feature | Support on ROCm |
|---------|-----------------|
| Tensor Parallelism | Yes |
| FP8 Quantization | Depends on GPU |
| CUDA Graphs | Yes |
| Async Output | No |
| Multi-step Decoding | No |
| Speculative Decoding | Yes |
| Prefix Caching | Yes |
| Chunked Prefill | Yes |

The two "No" rows (Async Output, Multi-step Decoding) were transcribed from a
vLLM feature-support snapshot and not re-checked against current upstream
*(unverified 2026-10-18)*.

### Context Length

Some models on ROCm have context length limited to 4096 tokens. Check model documentation.

### FP8 Support

FP8 quantization depends on GPU architecture:

- MI300 series supports native FP8
- Radeon RX 7900 supports FP8 via software emulation
- Strix Halo (gfx1151) uses **software emulation** for FP8 matmuls — RDNA 3.5 has
  no FP8 tensor instructions ([FP8-ROCM](https://github.com/wabibito/FP8-ROCM));
  expect memory savings, not full compute speedup. (Conflicting vendor-side
  evidence exists for an FP8/INT8 WMMA path used by FSR4 on RDNA 3.5, so treat
  the matmul claim as the settled half.)

For FP4 on Dankclaw the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provided native
ROCm FP4 kernels; that runtime line is **retired** (CIRU/Kairic eras,
[[ciru-activefpx-promptforge]], [[frank-dragoon-kairic-edge-qwen38-27b]]) and no
ROCmFPX build serves this host today.

## Example Commands

### Basic APU Deployment

```
vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 64
```

### With FP8 Model

```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8
```

### With Long Context

```
vllm serve Qwen/Qwen3.6-35B-A3B \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --gpu-memory-utilization 0.8 \
    --max-model-len 16384 \
    --rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
```

## Monitoring

Check GPU status:

```
rocm-smi
```

Check ROCm device info:

```
rocminfo
```

Check memory usage:

```
free -h
```

Monitor temperature:

```
rocm-smi --showtemperature
```

## See Also

- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]