docs: fix dankclaw specs (30GiB RAM, 96GiB GPU-visible, ROCm 7.2.4) + add validate.sh
This commit is contained in:
Frank
@@ -2,8 +2,7 @@
|
||||
|
||||
## Server Overview
|
||||
|
||||
Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same 61 GB RAM pool.
|
||||
rr
|
||||
Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool.
|
||||
## Hardware Specifications
|
||||
|
||||
### CPU
|
||||
@@ -22,50 +21,57 @@ rr
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Model | AMD Radeon 8060S (Strix Halo) |
|
||||
| PCIe ID | c4:00.0 |
|
||||
| Architecture | Strix/Strix Halo |
|
||||
| Memory | Unified with system RAM (no separate VRAM) |
|
||||
| Device ID | 0x1586 |
|
||||
| Architecture | gfx1151 (Strix Halo) |
|
||||
| GPU-Visible Memory | 96 GiB (ROCm addressable range) |
|
||||
| Physical RAM | 30 GiB (shared CPU + GPU) |
|
||||
|
||||
### ROCm
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| ROCk Module | Loaded |
|
||||
| Runtime Version | 1.1 |
|
||||
| HIP Version | 7.1.52801 |
|
||||
| Runtime Version | 1.18 |
|
||||
| ROCm Version | 7.2.4 (from AMD repos) |
|
||||
| Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
|
||||
| GPU Arch | gfx1151 (Strix Halo) |
|
||||
| GPU Global Memory | 96 GiB (overcommitted, physical is 30 GiB) |
|
||||
|
||||
### Memory
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| System RAM | 61 GiB (shared CPU + GPU) |
|
||||
| System RAM | 30 GiB (shared CPU + GPU) |
|
||||
| GPU-Visible (ROCm) | 96 GiB (overcommitted addressable range) |
|
||||
| Swap | 8.0 GiB |
|
||||
| /dev/shm | 31 GiB |
|
||||
| /dev/shm | ~15 GiB (half of RAM) |
|
||||
|
||||
### Storage
|
||||
|
||||
| Filesystem | Size | Used | Avail | Use% |
|
||||
|------------|------|------|-------|------|
|
||||
| / (nvme0n1p2) | 1.9T | 333G | 1.5T | 19% |
|
||||
| / (nvme0n1p2) | 1.9T | 391G | 1.4T | 22% |
|
||||
|
||||
### OS and Software
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
|
||||
| Kernel | 7.0.0-28-generic |
|
||||
| Python | 3.14.4 |
|
||||
| Kernel | 7.0.0-29-generic |
|
||||
| Python | 3.14.4 (system), 3.12.13 (vLLM venv) |
|
||||
| vLLM | 0.26.0+rocm723 |
|
||||
| uv | 0.12.2 |
|
||||
|
||||
## VLLM Relevance
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- 61 GB shared memory allows large models without VRAM constraints.
|
||||
- Only 8 GB swap. Set `--swap-space` conservatively (4-8 GB).
|
||||
- ROCm and HIP 7.1 are installed. vLLM ROCm support requires ROCm 6.3+.
|
||||
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Verify compatibility.
|
||||
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151/1150). vLLM lists this as supported.
|
||||
- 1.5 TB free disk space is ample for model storage.
|
||||
- ROCm reports 96 GiB GPU memory (overcommitted). Physical RAM is 30 GiB. vLLM sees 96 GiB and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
|
||||
- Only 8 GiB swap. Set `--swap-space` conservatively (4-8 GiB).
|
||||
- ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+.
|
||||
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv.
|
||||
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
|
||||
- 1.4 TB free disk space is ample for model storage.
|
||||
- `libmpi_cxx.so.40` is missing from the system. Install `openmpi-bin` or provide a workaround for torch MPI dependency.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -26,10 +26,11 @@ Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27
|
||||
|----------|-------|
|
||||
| CPU | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo) |
|
||||
| GPU | Integrated Radeon 8060S (unified memory) |
|
||||
| Unified RAM | 61 GB (shared CPU + GPU) |
|
||||
| Swap | 8 GB |
|
||||
| ROCm/HIP | 7.1 |
|
||||
| Python | 3.14.4 |
|
||||
| Physical RAM | 30 GiB (shared CPU + GPU) |
|
||||
| GPU-Visible (ROCm) | 96 GiB (overcommitted) |
|
||||
| Swap | 8 GiB |
|
||||
| ROCm | 7.2.4 |
|
||||
| Python | 3.12.13 (vLLM venv) |
|
||||
|
||||
## Memory Calculation
|
||||
|
||||
@@ -42,11 +43,11 @@ Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27
|
||||
| Overhead | ~2 GB | ~2 GB |
|
||||
| **Total (approximate)** | ~58 GB | ~16.5 GB |
|
||||
|
||||
The FP8 variant requires approximately 16-17 GB. This fits within the 61 GB unified memory with comfortable headroom for KV cache.
|
||||
The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB. Both fit within the 30 GiB physical RAM, but ROCm reports 96 GiB to vLLM. Set `--gpu-memory-utilization` conservatively (0.80) to avoid OOM.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install vLLM for ROCm. Version 0.19.0 or later is required for Qwen3.6:
|
||||
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
|
||||
|
||||
```
|
||||
uv venv --python 3.12 --seed
|
||||
@@ -54,13 +55,24 @@ source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
Python 3.14.4 is installed on Dankclaw. vLLM ROCm wheels target Python 3.12. Create a virtual environment with Python 3.12 if available. Verify that Python 3.12 is installed:
|
||||
Installed: vLLM 0.26.0+rocm723 in Python 3.12.13 venv at `/home/dank/vllm-env`.
|
||||
|
||||
### MPI Dependency
|
||||
|
||||
Torch requires `libmpi_cxx.so.40`. If missing, install `openmpi-bin`:
|
||||
|
||||
```
|
||||
python3.12 --version
|
||||
sudo apt install -y openmpi-bin
|
||||
```
|
||||
|
||||
If Python 3.12 is not available, install it or use the system Python with `--no-build-isolation`.
|
||||
### Environment Variables
|
||||
|
||||
Set these for ROCm on Strix Halo:
|
||||
|
||||
```
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
export HF_HOME=/home/dank/.cache/huggingface
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@ In unified memory:
|
||||
|
||||
### ROCm Version
|
||||
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw should have ROCm 7.0 or later.
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw has ROCm 7.2.4 from AMD repos.
|
||||
|
||||
Check ROCm version:
|
||||
|
||||
@@ -141,6 +141,10 @@ Unified memory has lower bandwidth than dedicated VRAM:
|
||||
| GDDR6 (RTX 4090) | 1 TB/s |
|
||||
| LPDDR5x (Strix Halo) | ~170 GB/s |
|
||||
|
||||
### ROCm Overcommit
|
||||
|
||||
ROCm reports 96 GiB GPU memory on Strix Halo APU. Physical RAM is 30 GiB. The GPU can address more than physical RAM exists. vLLM will see 96 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
|
||||
|
||||
Lower bandwidth means slower inference. Plan for lower throughput.
|
||||
|
||||
### Cache Effects
|
||||
|
||||
Reference in New Issue
Block a user