docs(vault): fleet-wide fact re-verification pass (67 pages)
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
This commit is contained in:
1 parent
466fd477d8
commit
e7f0d93c86
67 files changed
+2619
-1152
No files matched your search
@@ -36,7 +36,7 @@ In unified memory:
|
||||
|
||||
### ROCm Version
|
||||
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw has ROCm 7.2.4 from AMD repos.
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above *(unverified 2026-10-18)*. Dankclaw has ROCm 7.2.4 from AMD repos (`/etc/apt/sources.list.d/rocm.list` → `deb https://repo.radeon.com/rocm/apt/7.2.4 noble main`; `rocm-core 7.2.4.70204-93~24.04`, read 2026-10-18). No vLLM endpoint runs on this host today — 7770 is halogen-flash-server and 7771 is llama.cpp ([[Dankclaw System Specs]]) — so this page is reference material, not the serving path.
|
||||
|
||||
Check ROCm version:
|
||||
|
||||
@@ -83,6 +83,12 @@ Available backends for ROCm:
|
||||
| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
|
||||
| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |
|
||||
|
||||
All seven names exist on current vLLM `main` (`AttentionBackendEnum` in
|
||||
`vllm/v1/attention/backends/registry.py` —
|
||||
[source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/attention/backends/registry.py),
|
||||
read 2026-10-18), and `--attention-backend`, `--numa-bind`, `--numa-bind-nodes`
|
||||
are real `vllm serve` flags (`vllm/engine/arg_utils.py:1098,1171-1176`).
|
||||
|
||||
## Special Flags for APU
|
||||
|
||||
### Memory Configuration
|
||||
@@ -97,7 +103,7 @@ vllm serve model \
|
||||
|
||||
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
|
||||
- `max_model_len`: Match actual context needs
|
||||
- The old `--swap-space` argument was removed from current vLLM. Use `--kv-offloading-size` for CPU KV offload
|
||||
- The old `--swap-space` argument was removed from current vLLM — `CacheConfig` on `main` carries no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) for CPU KV offload
|
||||
|
||||
### NUMA Binding
|
||||
|
||||
@@ -126,15 +132,25 @@ export HSA_ENABLE_SDMA=1
|
||||
export HSA_FORCE_FINE_GRAIN_PCIE=1
|
||||
```
|
||||
|
||||
### GPU Memory Limits
|
||||
These three variables came out of the September research pass and were not
|
||||
re-checked against current ROCm documentation *(unverified 2026-10-18)*.
|
||||
|
||||
Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:
|
||||
### GPU "memory limits" and the gfx override
|
||||
|
||||
There is no memory-limit knob for the APU here. What the GPU may address is the
|
||||
GTT aperture (`mem_info_gtt_total`, see §ROCm Overcommit); the firmware VGM/UMA
|
||||
setting in BIOS is what sizes the *dedicated* carve-out.
|
||||
|
||||
`HSA_OVERRIDE_GFX_VERSION` is not a memory setting — it makes the runtime report
|
||||
a different ISA version:
|
||||
|
||||
```
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
```
|
||||
|
||||
This may be needed for Strix Halo compatibility.
|
||||
Do not set it on this host: the stack already reports `gfx1151` natively
|
||||
(`rocm-smi --showproductname` → `GFX Version: gfx1151`, read 2026-10-18). The
|
||||
override exists for GPUs whose ISA the installed runtime does not know.
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
@@ -144,13 +160,13 @@ Unified memory has lower bandwidth than dedicated VRAM:
|
||||
|
||||
| Memory Type | Bandwidth |
|
||||
|-------------|-----------|
|
||||
| HBM3 (A100) | 2 TB/s |
|
||||
| HBM2e (A100 80 GB) | 2 TB/s |
|
||||
| GDDR6 (RTX 4090) | 1 TB/s |
|
||||
| LPDDR5x (Strix Halo) | ~170 GB/s |
|
||||
| LPDDR5x-8000, 256-bit (Strix Halo) | 256 GB/s ([amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)) |
|
||||
|
||||
### ROCm Overcommit
|
||||
|
||||
ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture, `gtt_total=122402963456` bytes = 114 GiB; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM is 64 GiB (61 GiB visible; the box was upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
|
||||
ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture: `mem_info_gtt_total` = 122402963456 bytes = 114 GiB, read 2026-10-18 under `/sys/class/drm/card1/device/`; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM visible to the OS is **121 GiB** today (`free -g` total 121, read 2026-10-18; 64 GiB in the earlier reading on this page, upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
|
||||
|
||||
Lower bandwidth means slower inference. Plan for lower throughput.
|
||||
|
||||
@@ -195,6 +211,10 @@ Unified memory limits concurrent requests:
|
||||
| Prefix Caching | Yes |
|
||||
| Chunked Prefill | Yes |
|
||||
|
||||
The two "No" rows (Async Output, Multi-step Decoding) were transcribed from a
|
||||
vLLM feature-support snapshot and not re-checked against current upstream
|
||||
*(unverified 2026-10-18)*.
|
||||
|
||||
### Context Length
|
||||
|
||||
Some models on ROCm have context length limited to 4096 tokens. Check model documentation.
|
||||
@@ -205,9 +225,16 @@ FP8 quantization depends on GPU architecture:
|
||||
|
||||
- MI300 series supports native FP8
|
||||
- Radeon RX 7900 supports FP8 via software emulation
|
||||
- Strix Halo (gfx1151) uses **software emulation** for FP8 (no native FP8 tensor cores); expect memory savings, not full compute speedup
|
||||
- Strix Halo (gfx1151) uses **software emulation** for FP8 matmuls — RDNA 3.5 has
|
||||
no FP8 tensor instructions ([FP8-ROCM](https://github.com/wabibito/FP8-ROCM));
|
||||
expect memory savings, not full compute speedup. (Conflicting vendor-side
|
||||
evidence exists for an FP8/INT8 WMMA path used by FSR4 on RDNA 3.5, so treat
|
||||
the matmul claim as the settled half.)
|
||||
|
||||
For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native ROCm FP4 kernels.
|
||||
For FP4 on Dankclaw the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provided native
|
||||
ROCm FP4 kernels; that runtime line is **retired** (CIRU/Kairic eras,
|
||||
[[ciru-activefpx-promptforge]], [[frank-dragoon-kairic-edge-qwen38-27b]]) and no
|
||||
ROCmFPX build serves this host today.
|
||||
|
||||
## Example Commands
|
||||
|
||||
@@ -217,7 +244,6 @@ For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--max-num-seqs 64
|
||||
```
|
||||
@@ -228,8 +254,7 @@ vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--swap-space 8
|
||||
--kv-cache-dtype fp8
|
||||
```
|
||||
|
||||
### With Long Context
|
||||
@@ -238,7 +263,6 @@ vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--swap-space 16 \
|
||||
--max-model-len 16384 \
|
||||
--rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user