docs: reflect Ling-3 vLLM (frank-default-large) on Dankclaw
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770 with MTP speculative decoding. Align Dankserver to the running vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs (gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4 experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
This commit is contained in:
Frank committed
1 parent
52ad985978
commit
903d54c389
9 files changed
+55
-54
No files matched your search
@@ -146,21 +146,15 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw runs the ROCmFPX llama.cpp runtime as its primary service. The active service uses the Ling-3.0-flash ROCmFP4 MTP model with `--spec-type draft-mtp`. This gives measured results:
|
||||
Dankclaw runs vLLM-ROCm as its primary runtime. The active service (`frank-default-large`, port 7770) serves the Ling-3.0-flash CIRU INT4 (W4A16) model with native **MTP speculative decoding** (K1, lossless). MTP uses a NextN head preserved in the checkpoint, so no separate draft model is needed.
|
||||
|
||||
- 45-50 t/s decode with MTP speculation.
|
||||
- ~37 t/s plain decode.
|
||||
- ~95% draft acceptance with `--spec-draft-n-max 2`.
|
||||
Ling-3.0-flash facts:
|
||||
|
||||
ROCmFPX is a llama.cpp fork by charlie12345. It adds ROCmFP4 and ROCmFP2 GGUF weight formats with HIP/ROCm and Vulkan kernels. It is not merged into upstream llama.cpp.
|
||||
- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active (hybrid linear attention: KDA + MLA).
|
||||
- The MTP head is served in-band; vLLM detects it and shares the target embedding and lm_head weights with the draft head.
|
||||
- The earlier ROCmFPX llama.cpp service (raulvidis ROCmFP4 MTP GGUF) was retired because it was defective. vLLM now handles concurrency and tool calling correctly.
|
||||
|
||||
Ling-3.0-flash-related facts:
|
||||
|
||||
- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active.
|
||||
- The MTP (NextN) head is preserved in the GGUF conversion.
|
||||
- The raulvidis GGUF repo targets Strix Halo (gfx1151).
|
||||
|
||||
If you want DFlash on Dankclaw, use a DFlash GGUF draft. Community conversions exist, for example Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp. The ROCm path for DFlash is newer than the MTP path and less proven on gfx1151.
|
||||
If you want DFlash or DSpark on Dankclaw, use a vLLM speculative-config draft that pairs with the target model. DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. MTP is the proven, lossless path on gfx1151.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -58,19 +58,19 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|
||||
|----------|-------|
|
||||
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
|
||||
| Kernel | 7.0.0-29-generic |
|
||||
| Python | 3.14.4 (system), 3.12.13 (vLLM venv) |
|
||||
| vLLM | 0.26.0 (in `/home/dank/vllm-rocm-venv`, Python 3.12) |
|
||||
| Python | 3.14.4 (system), 3.12.13 (CIRU vLLM venv) |
|
||||
| vLLM | 0.26.1rc1.dev467+g838616875.rocm715 (CIRU venv, Python 3.12) |
|
||||
| uv | 0.12.2 |
|
||||
|
||||
## VLLM Relevance
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). Physical RAM is 30 GiB. The driver reports a 96 GiB VRAM ceiling (`sysfs mem_info_vram_total`), but the machine has only 30.5 GB of physical RAM. vLLM sees the large number and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
|
||||
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer).
|
||||
- Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
|
||||
- Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag.
|
||||
- The active llama.cpp service on Dankclaw runs on the ROCmFPX runtime (`/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server`) with the Ling-3.0-flash ROCmFP4 MTP model, not regular llama.cpp.
|
||||
- ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+.
|
||||
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv.
|
||||
- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service.
|
||||
- ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+.
|
||||
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12.
|
||||
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
- `libmpi_cxx.so.40` is missing from the system. Install `openmpi-bin` or provide a workaround for torch MPI dependency.
|
||||
|
||||
@@ -10,7 +10,7 @@ Use this page when you choose a model format for Dankserver or Dankclaw. The sho
|
||||
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
|
||||
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
|
||||
- AMD CDNA (MI300 series): FP8 or MXFP4.
|
||||
- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4.
|
||||
- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
|
||||
- CPU-only: any GGUF quant.
|
||||
|
||||
## Quantization Types
|
||||
@@ -112,13 +112,14 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
|
||||
|
||||
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
|
||||
|
||||
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats:
|
||||
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
|
||||
|
||||
| Ranking | Format | Why |
|
||||
|---------|--------|-----|
|
||||
| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP |
|
||||
| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp |
|
||||
| 3 | FP8 (vLLM) | software emulation; memory savings only |
|
||||
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
|
||||
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
|
||||
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
|
||||
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -47,7 +47,7 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> **Production note:** Dankclaw's primary llama.cpp service now runs the ROCmFPX runtime with the Ling-3.0-flash ROCmFP4 MTP model (see `scripts/llama-server.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
|
||||
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-default-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
|
||||
|
||||
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
|
||||
|
||||
|
||||
+10
-10
@@ -8,8 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 |
|
||||
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 |
|
||||
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -77,10 +77,10 @@ Run these commands on Dankclaw:
|
||||
```
|
||||
cd /home/dank/vllmScripts
|
||||
git pull origin main
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/llama-server.service /etc/systemd/system/llama-server.service
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart llama-server
|
||||
curl -sf http://127.0.0.1:8080/health
|
||||
sudo systemctl restart frank-default-large
|
||||
curl -sf http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
### Update Dankserver
|
||||
@@ -97,15 +97,15 @@ curl -sf http://127.0.0.1:8000/health
|
||||
|
||||
## Service Units
|
||||
|
||||
### Dankclaw — llama-server
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw runs the ROCmFPX llama.cpp runtime. It serves the Ling-3.0-flash ROCmFP4 MTP model with `draft-mtp` speculative decoding. The unit file is `scripts/llama-server.service` in the repository. The machine symlinks it to `/etc/systemd/system/llama-server.service`.
|
||||
The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95).
|
||||
|
||||
### Dankserver — vllm-qwen36
|
||||
### Dankserver — vllm-awq-qwen36
|
||||
|
||||
The vLLM service on Dankserver runs in a Docker container. The unit file is a physical file on the machine, not a symlink into the repository. The `vllm-qwen36.sh` launcher in the repository passes the model and flags to `docker run`.
|
||||
The active vLLM service on Dankserver runs in a Podman/Docker container and serves the Qwen3.6-35B-A3B AWQ 4-bit model on port 34401. The unit file is a physical file on the machine, not a symlink into the repository.
|
||||
|
||||
> Note: a second unit, `vllm-awq-qwen36.service`, currently runs the AWQ 4-bit model on port 34401. Verify the intended service before a restart. See the repository `git log` for the latest decision.
|
||||
> Note: the failed `vllm-qwen36.service` (FP8) and the retired `llama-server.service` (ROCmFPX llama.cpp) are not active. Verify the intended service before a restart. See the repository `git log` for the latest decision.
|
||||
|
||||
## Validation
|
||||
|
||||
|
||||
+10
-6
@@ -8,20 +8,24 @@ Documentation for running vLLM inference on two local machines.
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[vLLM 0.26.0 in Podman\nvllm-qwen36-dflash]
|
||||
C[Qwen3.6-35B-A3B-FP8\n+ DFlash draft, port 8000]
|
||||
B[vLLM in Podman
|
||||
vllm-awq-qwen36]
|
||||
C[Qwen3.6-35B-A3B AWQ 4bit
|
||||
port 34401]
|
||||
A --> B --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
|
||||
E[ROCmFPX llama.cpp\n/ROCmFPX/build-strix-rocmfp4]
|
||||
F[Ling-3.0-flash ROCmFP4-MTP\ndraft-mtp, port 8080]
|
||||
E[vLLM-ROCm CIRU INT4
|
||||
frank-default-large]
|
||||
F[Ling-3.0-flash CIRU INT4 MTP
|
||||
port 7770]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
G[LLM Clients / API] -->|HTTP :8000| B
|
||||
G -->|HTTP :8080| E
|
||||
G[LLM Clients / API] -->|HTTP :34401| B
|
||||
G -->|HTTP :7770| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
|
||||
Reference in new issue
Block a user