diff --git a/AGENTS.md b/AGENTS.md index fe2338c..4b9a239 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Dankclaw runs a ROCmFPX llama.cpp service. Dankserver runs a vLLM service. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process. +This project manages deployment scripts for two LLM inference servers. Both run vLLM. Dankclaw runs a vLLM-ROCm service for Ling-3.0-flash (CIRU INT4). Dankserver runs a vLLM service for Qwen3.6. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process. ## Git Workflow @@ -22,8 +22,8 @@ All script edits must flow through git. Do not edit files on the remote machine Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint. ``` -./scripts/deploy.sh dankclaw # ROCmFPX llama.cpp (llama-server, port 8080) -./scripts/deploy.sh dankserver # vLLM (vllm-qwen36, port 8000) +./scripts/deploy.sh dankclaw # vLLM (frank-default-large, port 7770) +./scripts/deploy.sh dankserver # vLLM (vllm-awq-qwen36, port 34401) ./scripts/deploy.sh dankclaw --dry-run # preview without changes ``` @@ -42,8 +42,8 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 | -| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 | +| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 | +| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 | ## Why diff --git a/README.md b/README.md index fa11008..fa217c7 100644 --- a/README.md +++ b/README.md @@ -13,14 +13,16 @@ Scripts for running LLM inference on two local machines. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| -| llama-server | dankclaw | ROCmFPX llama.cpp | Ling-3.0-flash ROCmFP4 MTP | 8080 | +| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 | | vllm-awq-qwen36 | dankserver | vLLM (Docker) | Qwen3.6-35B-A3B AWQ 4-bit | 34401 | +> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. Its unit file `scripts/llama-server.service` remains for reference. + ## Models | Model | Target | Format | |-------|--------|--------| -| Ling-3.0-flash ROCmFP4-MTP | dankclaw | ROCmFP4 GGUF | +| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) | | Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ | | Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) | | Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) | diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index 940fe09..ed2da64 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -146,21 +146,15 @@ docker run --rm --name vllm-qwen36-dflash \ ## Deployment on Dankclaw (AMD Strix Halo gfx1151) -Dankclaw runs the ROCmFPX llama.cpp runtime as its primary service. The active service uses the Ling-3.0-flash ROCmFP4 MTP model with `--spec-type draft-mtp`. This gives measured results: +Dankclaw runs vLLM-ROCm as its primary runtime. The active service (`frank-default-large`, port 7770) serves the Ling-3.0-flash CIRU INT4 (W4A16) model with native **MTP speculative decoding** (K1, lossless). MTP uses a NextN head preserved in the checkpoint, so no separate draft model is needed. -- 45-50 t/s decode with MTP speculation. -- ~37 t/s plain decode. -- ~95% draft acceptance with `--spec-draft-n-max 2`. +Ling-3.0-flash facts: -ROCmFPX is a llama.cpp fork by charlie12345. It adds ROCmFP4 and ROCmFP2 GGUF weight formats with HIP/ROCm and Vulkan kernels. It is not merged into upstream llama.cpp. +- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active (hybrid linear attention: KDA + MLA). +- The MTP head is served in-band; vLLM detects it and shares the target embedding and lm_head weights with the draft head. +- The earlier ROCmFPX llama.cpp service (raulvidis ROCmFP4 MTP GGUF) was retired because it was defective. vLLM now handles concurrency and tool calling correctly. -Ling-3.0-flash-related facts: - -- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active. -- The MTP (NextN) head is preserved in the GGUF conversion. -- The raulvidis GGUF repo targets Strix Halo (gfx1151). - -If you want DFlash on Dankclaw, use a DFlash GGUF draft. Community conversions exist, for example Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp. The ROCm path for DFlash is newer than the MTP path and less proven on gfx1151. +If you want DFlash or DSpark on Dankclaw, use a vLLM speculative-config draft that pairs with the target model. DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. MTP is the proven, lossless path on gfx1151. ## Notes diff --git a/docs/Dankclaw System Specs.md b/docs/Dankclaw System Specs.md index d32ec12..bafd889 100644 --- a/docs/Dankclaw System Specs.md +++ b/docs/Dankclaw System Specs.md @@ -58,19 +58,19 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit |----------|-------| | OS | Ubuntu 26.04 LTS (Resolute Raccoon) | | Kernel | 7.0.0-29-generic | -| Python | 3.14.4 (system), 3.12.13 (vLLM venv) | -| vLLM | 0.26.0 (in `/home/dank/vllm-rocm-venv`, Python 3.12) | +| Python | 3.14.4 (system), 3.12.13 (CIRU vLLM venv) | +| vLLM | 0.26.1rc1.dev467+g838616875.rocm715 (CIRU venv, Python 3.12) | | uv | 0.12.2 | ## VLLM Relevance - Unified memory eliminates separate VRAM limits. The GPU uses system RAM. -- ROCm reports 96 GiB of GPU-visible memory (overcommitted). Physical RAM is 30 GiB. The driver reports a 96 GiB VRAM ceiling (`sysfs mem_info_vram_total`), but the machine has only 30.5 GB of physical RAM. vLLM sees the large number and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory. +- ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer). - Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin. - Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag. -- The active llama.cpp service on Dankclaw runs on the ROCmFPX runtime (`/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server`) with the Ling-3.0-flash ROCmFP4 MTP model, not regular llama.cpp. -- ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+. -- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv. +- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service. +- ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+. +- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12. - Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported. - 1.1 TB free disk space is ample for model storage. - `libmpi_cxx.so.40` is missing from the system. Install `openmpi-bin` or provide a workaround for torch MPI dependency. diff --git a/docs/Quantization by Hardware Architecture.md b/docs/Quantization by Hardware Architecture.md index 7122a65..9aef395 100644 --- a/docs/Quantization by Hardware Architecture.md +++ b/docs/Quantization by Hardware Architecture.md @@ -10,7 +10,7 @@ Use this page when you choose a model format for Dankserver or Dankclaw. The sho - NVIDIA Ada (RTX 40 series): FP8 or AWQ. - NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4. - AMD CDNA (MI300 series): FP8 or MXFP4. -- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4. +- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4. - CPU-only: any GGUF quant. ## Quantization Types @@ -112,13 +112,14 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use: ### Dankclaw (Strix Halo gfx1151, RDNA 3.5) -RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats: +RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache. | Ranking | Format | Why | |---------|--------|-----| -| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP | -| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp | -| 3 | FP8 (vLLM) | software emulation; memory savings only | +| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency | +| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) | +| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only | +| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win | ## See Also diff --git a/docs/Running Qwen3.6 27B FP8 on Dankclaw.md b/docs/Running Qwen3.6 27B FP8 on Dankclaw.md index 8259f3c..515cab3 100644 --- a/docs/Running Qwen3.6 27B FP8 on Dankclaw.md +++ b/docs/Running Qwen3.6 27B FP8 on Dankclaw.md @@ -47,7 +47,7 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB ## Prerequisites -> **Production note:** Dankclaw's primary llama.cpp service now runs the ROCmFPX runtime with the Ling-3.0-flash ROCmFP4 MTP model (see `scripts/llama-server.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B. +> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-default-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B. Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding: diff --git a/docs/deployment.md b/docs/deployment.md index ccbdd02..b04147b 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -8,8 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | ROCmFPX llama.cpp | llama-server | 8080 | -| Dankserver | dank@dankserver | vLLM (Docker) | vllm-qwen36 | 8000 | +| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 | +| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 | ## Workflow @@ -77,10 +77,10 @@ Run these commands on Dankclaw: ``` cd /home/dank/vllmScripts git pull origin main -sudo ln -sf /home/dank/vllmScripts/scripts/llama-server.service /etc/systemd/system/llama-server.service +sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service sudo systemctl daemon-reload -sudo systemctl restart llama-server -curl -sf http://127.0.0.1:8080/health +sudo systemctl restart frank-default-large +curl -sf http://127.0.0.1:7770/health ``` ### Update Dankserver @@ -97,15 +97,15 @@ curl -sf http://127.0.0.1:8000/health ## Service Units -### Dankclaw — llama-server +### Dankclaw — frank-default-large -The active service on Dankclaw runs the ROCmFPX llama.cpp runtime. It serves the Ling-3.0-flash ROCmFP4 MTP model with `draft-mtp` speculative decoding. The unit file is `scripts/llama-server.service` in the repository. The machine symlinks it to `/etc/systemd/system/llama-server.service`. +The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95). -### Dankserver — vllm-qwen36 +### Dankserver — vllm-awq-qwen36 -The vLLM service on Dankserver runs in a Docker container. The unit file is a physical file on the machine, not a symlink into the repository. The `vllm-qwen36.sh` launcher in the repository passes the model and flags to `docker run`. +The active vLLM service on Dankserver runs in a Podman/Docker container and serves the Qwen3.6-35B-A3B AWQ 4-bit model on port 34401. The unit file is a physical file on the machine, not a symlink into the repository. -> Note: a second unit, `vllm-awq-qwen36.service`, currently runs the AWQ 4-bit model on port 34401. Verify the intended service before a restart. See the repository `git log` for the latest decision. +> Note: the failed `vllm-qwen36.service` (FP8) and the retired `llama-server.service` (ROCmFPX llama.cpp) are not active. Verify the intended service before a restart. See the repository `git log` for the latest decision. ## Validation diff --git a/docs/index.md b/docs/index.md index 356fc78..284b735 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,20 +8,24 @@ Documentation for running vLLM inference on two local machines. graph LR subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04] A[RTX 3090 24GB + 62GB RAM] - B[vLLM 0.26.0 in Podman\nvllm-qwen36-dflash] - C[Qwen3.6-35B-A3B-FP8\n+ DFlash draft, port 8000] + B[vLLM in Podman +vllm-awq-qwen36] + C[Qwen3.6-35B-A3B AWQ 4bit +port 34401] A --> B --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] - E[ROCmFPX llama.cpp\n/ROCmFPX/build-strix-rocmfp4] - F[Ling-3.0-flash ROCmFP4-MTP\ndraft-mtp, port 8080] + E[vLLM-ROCm CIRU INT4 +frank-default-large] + F[Ling-3.0-flash CIRU INT4 MTP +port 7770] D --> E --> F end - G[LLM Clients / API] -->|HTTP :8000| B - G -->|HTTP :8080| E + G[LLM Clients / API] -->|HTTP :34401| B + G -->|HTTP :7770| E ``` ## Servers diff --git a/scripts/deploy.sh b/scripts/deploy.sh index 7c83334..a857ad2 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -12,8 +12,8 @@ # ./deploy.sh [--dry-run] # # Targets: -# dankclaw - ROCmFPX llama.cpp server (llama-server.service, port 8080) -# dankserver - vLLM server (vllm-qwen36.service, port 8000) +# dankclaw - vLLM-ROCm server (frank-default-large.service, port 7770) +# dankserver - vLLM server (vllm-awq-qwen36.service, port 34401) # # Example: # ./deploy.sh dankclaw @@ -28,14 +28,14 @@ declare -A REMOTE_USER REMOTE_HOST REMOTE_DIR SERVICE SERVICE_HEALTH REMOTE_USER[dankclaw]=dank REMOTE_HOST[dankclaw]=dankclaw REMOTE_DIR[dankclaw]="/home/dank/vllmScripts" -SERVICE[dankclaw]="llama-server" -SERVICE_HEALTH[dankclaw]="http://127.0.0.1:8080/health" +SERVICE[dankclaw]="frank-default-large" +SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health" REMOTE_USER[dankserver]=dank REMOTE_HOST[dankserver]=dankserver REMOTE_DIR[dankserver]="/home/dank/vLLM" -SERVICE[dankserver]="vllm-qwen36" -SERVICE_HEALTH[dankserver]="http://127.0.0.1:8000/health" +SERVICE[dankserver]="vllm-awq-qwen36" +SERVICE_HEALTH[dankserver]="http://127.0.0.1:34401/health" TARGET="${1:-}" DRY_RUN=0 @@ -103,7 +103,7 @@ git pull origin main # Install / symlink the systemd unit case '${TARGET}' in dankclaw) - sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/llama-server.service' /etc/systemd/system/llama-server.service + sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-large.service' /etc/systemd/system/frank-default-large.service ;; dankserver) # vllm-qwen36.service is a physical file; no symlink needed.