docs: reflect Ling-3 vLLM (frank-default-large) on Dankclaw

Update target maps (AGENTS, README, deployment, deploy.sh) from the retired
ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770
with MTP speculative decoding. Align Dankserver to the running
vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs
(gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4
experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
This commit is contained in:
Frank committed 2026-08-10 23:37:27 -07:00
1 parent 52ad985978
commit 903d54c389
9 files changed
+55 -54

No files matched your search

+6 -6
View File
@@ -58,19 +58,19 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-29-generic |
| Python | 3.14.4 (system), 3.12.13 (vLLM venv) |
| vLLM | 0.26.0 (in `/home/dank/vllm-rocm-venv`, Python 3.12) |
| Python | 3.14.4 (system), 3.12.13 (CIRU vLLM venv) |
| vLLM | 0.26.1rc1.dev467+g838616875.rocm715 (CIRU venv, Python 3.12) |
| uv | 0.12.2 |
## VLLM Relevance
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). Physical RAM is 30 GiB. The driver reports a 96 GiB VRAM ceiling (`sysfs mem_info_vram_total`), but the machine has only 30.5 GB of physical RAM. vLLM sees the large number and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer).
- Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
- Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag.
- The active llama.cpp service on Dankclaw runs on the ROCmFPX runtime (`/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server`) with the Ling-3.0-flash ROCmFP4 MTP model, not regular llama.cpp.
- ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+.
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv.
- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service.
- ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+.
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12.
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
- 1.1 TB free disk space is ample for model storage.
- `libmpi_cxx.so.40` is missing from the system. Install `openmpi-bin` or provide a workaround for torch MPI dependency.