dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reflect Ling-3 vLLM (frank-default-large) on Dankclaw

Browse Source 
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired
ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770
with MTP speculative decoding. Align Dankserver to the running
vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs
(gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4
experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
This commit is contained in:
Frank
2026-08-10 23:37:27 -07:00
parent 52ad985978
commit 903d54c389
9 changed files with 55 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+5 -5
View File
@@ -2,7 +2,7 @@
## Overview
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a ROCmFPX llama.cpp service. Dankserver runs a vLLM service. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process.
This project manages deployment scripts for two LLM inference servers. Both run vLLM. Dankclaw runs a vLLM-ROCm service for Ling-3.0-flash (CIRU INT4). Dankserver runs a vLLM service for Qwen3.6. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process.
## Git Workflow
@@ -22,8 +22,8 @@ All script edits must flow through git. Do not edit files on the remote machine
Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint.
```
./scripts/deploy.sh dankclaw # ROCmFPX llama.cpp (llama-server, port 8080)
./scripts/deploy.sh dankserver # vLLM (vllm-qwen36, port 8000)
./scripts/deploy.sh dankclaw # vLLM (frank-default-large, port 7770)
./scripts/deploy.sh dankserver # vLLM (vllm-awq-qwen36, port 34401)
./scripts/deploy.sh dankclaw --dry-run # preview without changes
```
@@ -42,8 +42,8 @@ Target map:
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 |
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 |
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 |
## Why
README.md
+4 -2
View File
@@ -13,14 +13,16 @@ Scripts for running LLM inference on two local machines.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| llama-server | dankclaw | ROCmFPX llama.cpp | Ling-3.0-flash ROCmFP4 MTP | 8080 |
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 |
| vllm-awq-qwen36 | dankserver | vLLM (Docker) | Qwen3.6-35B-A3B AWQ 4-bit | 34401 |
> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. Its unit file `scripts/llama-server.service` remains for reference.
## Models
| Model | Target | Format |
|-------|--------|--------|
| Ling-3.0-flash ROCmFP4-MTP | dankclaw | ROCmFP4 GGUF |
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ |
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
docs/DFlash DSpark Speculative Decoding.md
+6 -12
View File
@@ -146,21 +146,15 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
Dankclaw runs the ROCmFPX llama.cpp runtime as its primary service. The active service uses the Ling-3.0-flash ROCmFP4 MTP model with `--spec-type draft-mtp`. This gives measured results:
Dankclaw runs vLLM-ROCm as its primary runtime. The active service (`frank-default-large`, port 7770) serves the Ling-3.0-flash CIRU INT4 (W4A16) model with native **MTP speculative decoding** (K1, lossless). MTP uses a NextN head preserved in the checkpoint, so no separate draft model is needed.
- 45-50 t/s decode with MTP speculation.
- ~37 t/s plain decode.
- ~95% draft acceptance with `--spec-draft-n-max 2`.
Ling-3.0-flash facts:
ROCmFPX is a llama.cpp fork by charlie12345. It adds ROCmFP4 and ROCmFP2 GGUF weight formats with HIP/ROCm and Vulkan kernels. It is not merged into upstream llama.cpp.
- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active (hybrid linear attention: KDA + MLA).
- The MTP head is served in-band; vLLM detects it and shares the target embedding and lm_head weights with the draft head.
- The earlier ROCmFPX llama.cpp service (raulvidis ROCmFP4 MTP GGUF) was retired because it was defective. vLLM now handles concurrency and tool calling correctly.
Ling-3.0-flash-related facts:
- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active.
- The MTP (NextN) head is preserved in the GGUF conversion.
- The raulvidis GGUF repo targets Strix Halo (gfx1151).
If you want DFlash on Dankclaw, use a DFlash GGUF draft. Community conversions exist, for example Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp. The ROCm path for DFlash is newer than the MTP path and less proven on gfx1151.
If you want DFlash or DSpark on Dankclaw, use a vLLM speculative-config draft that pairs with the target model. DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. MTP is the proven, lossless path on gfx1151.
## Notes
docs/Dankclaw System Specs.md
+6 -6
View File
@@ -58,19 +58,19 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-29-generic |
| Python | 3.14.4 (system), 3.12.13 (vLLM venv) |
| vLLM | 0.26.0 (in `/home/dank/vllm-rocm-venv`, Python 3.12) |
| Python | 3.14.4 (system), 3.12.13 (CIRU vLLM venv) |
| vLLM | 0.26.1rc1.dev467+g838616875.rocm715 (CIRU venv, Python 3.12) |
| uv | 0.12.2 |
## VLLM Relevance
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). Physical RAM is 30 GiB. The driver reports a 96 GiB VRAM ceiling (`sysfs mem_info_vram_total`), but the machine has only 30.5 GB of physical RAM. vLLM sees the large number and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer).
- Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
- Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag.
- The active llama.cpp service on Dankclaw runs on the ROCmFPX runtime (`/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server`) with the Ling-3.0-flash ROCmFP4 MTP model, not regular llama.cpp.
- ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+.
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv.
- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service.
- ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+.
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12.
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
- 1.1 TB free disk space is ample for model storage.
- `libmpi_cxx.so.40` is missing from the system. Install `openmpi-bin` or provide a workaround for torch MPI dependency.
docs/Quantization by Hardware Architecture.md
+6 -5
View File
@@ -10,7 +10,7 @@ Use this page when you choose a model format for Dankserver or Dankclaw. The sho
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
- AMD CDNA (MI300 series): FP8 or MXFP4.
- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4.
- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
- CPU-only: any GGUF quant.
## Quantization Types
@@ -112,13 +112,14 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats:
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
| Ranking | Format | Why |
|---------|--------|-----|
| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP |
| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp |
| 3 | FP8 (vLLM) | software emulation; memory savings only |
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
## See Also
docs/Running Qwen3.6 27B FP8 on Dankclaw.md
+1 -1
View File
@@ -47,7 +47,7 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB
## Prerequisites
> **Production note:** Dankclaw's primary llama.cpp service now runs the ROCmFPX runtime with the Ling-3.0-flash ROCmFP4 MTP model (see `scripts/llama-server.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-default-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
docs/deployment.md
+10 -10
View File
@@ -8,8 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 |
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 |
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 |
## Workflow
@@ -77,10 +77,10 @@ Run these commands on Dankclaw:
```
cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/llama-server.service /etc/systemd/system/llama-server.service
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart llama-server
curl -sf http://127.0.0.1:8080/health
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```
### Update Dankserver
@@ -97,15 +97,15 @@ curl -sf http://127.0.0.1:8000/health
## Service Units
### Dankclaw — llama-server
### Dankclaw — frank-default-large
The active service on Dankclaw runs the ROCmFPX llama.cpp runtime. It serves the Ling-3.0-flash ROCmFP4 MTP model with `draft-mtp` speculative decoding. The unit file is `scripts/llama-server.service` in the repository. The machine symlinks it to `/etc/systemd/system/llama-server.service`.
The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95).
### Dankserver — vllm-qwen36
### Dankserver — vllm-awq-qwen36
The vLLM service on Dankserver runs in a Docker container. The unit file is a physical file on the machine, not a symlink into the repository. The `vllm-qwen36.sh` launcher in the repository passes the model and flags to `docker run`.
The active vLLM service on Dankserver runs in a Podman/Docker container and serves the Qwen3.6-35B-A3B AWQ 4-bit model on port 34401. The unit file is a physical file on the machine, not a symlink into the repository.
> Note: a second unit, `vllm-awq-qwen36.service`, currently runs the AWQ 4-bit model on port 34401. Verify the intended service before a restart. See the repository `git log` for the latest decision.
> Note: the failed `vllm-qwen36.service` (FP8) and the retired `llama-server.service` (ROCmFPX llama.cpp) are not active. Verify the intended service before a restart. See the repository `git log` for the latest decision.
## Validation
docs/index.md
+10 -6
View File
@@ -8,20 +8,24 @@ Documentation for running vLLM inference on two local machines.
graph LR
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
A[RTX 3090 24GB + 62GB RAM]
B[vLLM 0.26.0 in Podman\nvllm-qwen36-dflash]
C[Qwen3.6-35B-A3B-FP8\n+ DFlash draft, port 8000]
B[vLLM in Podman
vllm-awq-qwen36]
C[Qwen3.6-35B-A3B AWQ 4bit
port 34401]
A --> B --> C
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
E[ROCmFPX llama.cpp\n/ROCmFPX/build-strix-rocmfp4]
F[Ling-3.0-flash ROCmFP4-MTP\ndraft-mtp, port 8080]
E[vLLM-ROCm CIRU INT4
frank-default-large]
F[Ling-3.0-flash CIRU INT4 MTP
port 7770]
D --> E --> F
end
G[LLM Clients / API] -->|HTTP :8000| B
G -->|HTTP :8080| E
G[LLM Clients / API] -->|HTTP :34401| B
G -->|HTTP :7770| E
```
## Servers
scripts/deploy.sh
+7 -7
View File
@@ -12,8 +12,8 @@
# ./deploy.sh <target> [--dry-run]
#
# Targets:
# dankclaw - ROCmFPX llama.cpp server (llama-server.service, port 8080)
# dankserver - vLLM server (vllm-qwen36.service, port 8000)
# dankclaw - vLLM-ROCm server (frank-default-large.service, port 7770)
# dankserver - vLLM server (vllm-awq-qwen36.service, port 34401)
#
# Example:
# ./deploy.sh dankclaw
@@ -28,14 +28,14 @@ declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH
REMOTE_USER[dankclaw]=dank
REMOTE_HOST[dankclaw]=dankclaw
REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
SERVICE[dankclaw]="llama-server"
SERVICE_HEALTH[dankclaw]="http://127.0.0.1:8080/health"
SERVICE[dankclaw]="frank-default-large"
SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
REMOTE_USER[dankserver]=dank
REMOTE_HOST[dankserver]=dankserver
REMOTE_DIR[dankserver]="/home/dank/vLLM"
SERVICE[dankserver]="vllm-qwen36"
SERVICE_HEALTH[dankserver]="http://127.0.0.1:8000/health"
SERVICE[dankserver]="vllm-awq-qwen36"
SERVICE_HEALTH[dankserver]="http://127.0.0.1:34401/health"
TARGET="${1:-}"
DRY_RUN=0
@@ -103,7 +103,7 @@ git pull origin main
# Install / symlink the systemd unit
case '${TARGET}' in
dankclaw)
sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/llama-server.service' /etc/systemd/system/llama-server.service
sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-large.service' /etc/systemd/system/frank-default-large.service
;;
dankserver)
# vllm-qwen36.service is a physical file; no symlink needed.