From 35c44c4123de5192e4df60cee51351d8e8db740e Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 11 Aug 2026 01:18:29 -0700 Subject: [PATCH] docs+deploy: Dankserver now llama.cpp frank-default-small (HauhauCS Q6_K_P + DFlash) Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index, Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0 draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16). deploy.sh now maps dankserver to frank-default-small and symlinks its unit. models.json: dankserver baseUrl -> :7770, contextWindow -> 262144. --- AGENTS.md | 6 +++--- README.md | 9 +++++---- docs/DFlash DSpark Speculative Decoding.md | 4 +++- docs/Dankserver System Specs.md | 13 ++++++------- docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md | 2 +- docs/deployment.md | 13 +++++++------ docs/index.md | 10 +++++----- scripts/deploy.sh | 9 ++++----- 8 files changed, 34 insertions(+), 32 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4b9a239..955bdcb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Both run vLLM. Dankclaw runs a vLLM-ROCm service for Ling-3.0-flash (CIRU INT4). Dankserver runs a vLLM service for Qwen3.6. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process. +This project manages deployment scripts for two LLM inference servers. Dankclaw runs a vLLM-ROCm service for Ling-3.0-flash (CIRU INT4). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with a DFlash draft and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process. ## Git Workflow @@ -23,7 +23,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main ``` ./scripts/deploy.sh dankclaw # vLLM (frank-default-large, port 7770) -./scripts/deploy.sh dankserver # vLLM (vllm-awq-qwen36, port 34401) +./scripts/deploy.sh dankserver # llama.cpp (frank-default-small, port 7770) ./scripts/deploy.sh dankclaw --dry-run # preview without changes ``` @@ -43,7 +43,7 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 | -| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 | +| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 | ## Why diff --git a/README.md b/README.md index fa217c7..b78cd8e 100644 --- a/README.md +++ b/README.md @@ -14,23 +14,24 @@ Scripts for running LLM inference on two local machines. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| | frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 | -| vllm-awq-qwen36 | dankserver | vLLM (Docker) | Qwen3.6-35B-A3B AWQ 4-bit | 34401 | +| frank-default-small | dankserver | llama.cpp (DFlash) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 | -> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. Its unit file `scripts/llama-server.service` remains for reference. +> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired `vllm-awq-qwen36` (AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain in `scripts/` for reference. ## Models | Model | Target | Format | |-------|--------|--------| | Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) | -| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ | +| Qwen3.6 35B A3B uncensored Q6_K_P (DFlash draft) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) | +| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) | | Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) | | Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) | ## Structure - `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes -- `scripts/` — Deployment and launch scripts (deploy.sh, serve-vllm-qwen36-dflash.sh, validate-vllm.sh, systemd unit files) +- `scripts/` — Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files) ## Deployment diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index ed2da64..0cd2447 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -115,7 +115,9 @@ The DFlash draft is not a standalone model. It must pair with its target model. ## Deployment on Dankserver (NVIDIA RTX 3090) -Dankserver runs vLLM in a Podman container with the RTX 3090. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: +The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 262144 context, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`. + +The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: - FP8 KV cache is valid on CUDA 12. - FP8 weights compute through a fallback path; no native FP8 tensor cores. diff --git a/docs/Dankserver System Specs.md b/docs/Dankserver System Specs.md index e17a8f4..c3a61b8 100644 --- a/docs/Dankserver System Specs.md +++ b/docs/Dankserver System Specs.md @@ -51,15 +51,14 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0 ### Active Services -A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies approximately 22 GB of the RTX 3090 VRAM (`nvidia-smi` reports `VLLM::EngineCore` at 22164 MiB). +The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 262144 context each. All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). At load it uses about 22 GB of the 24 GB VRAM and roughly 4 GB of resident RAM (experts are lazy-mapped). -## vLLM Relevance +## Hardware Relevance -- 24 GB VRAM limits model size. Use FP8 quantization for large models. -- 62 GB system RAM provides ample swap space for KV cache offloading. -- 127 GB swap supports large `--swap-space` values for CPU KV cache paging. -- Root disk has approximately 140 GB free. Store models on /mnt/entropy. -- CUDA 12.0 and Python 3.12.3 are compatible with vLLM. +- 24 GB VRAM limits the GPU-resident footprint. With 3 x 262144 context, the full-attention KV cache plus non-expert weights consume about 22 GB, leaving about 2 GB of headroom. MoE experts must stay on CPU. +- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load). +- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. +- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). ## See Also diff --git a/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md b/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md index f1c26d4..32cbc20 100644 --- a/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md +++ b/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md @@ -112,7 +112,7 @@ export HF_HOME=/mnt/entropy/huggingface ### Active Service Conflict -A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies roughly 22 GB of the 24 GB VRAM. Stop it before starting a competing service: +The active `frank-default-small` service (llama.cpp, port 7770) uses about 22 GB of the 24 GB VRAM. The retired `vllm-awq-qwen36` (Podman, ~22 GB) and `vllm-qwen36` services are not active. Stop any competing service before starting another: ``` podman stop vllm-qwen36-dflash diff --git a/docs/deployment.md b/docs/deployment.md index b04147b..2eee268 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 | -| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 | +| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 | ## Workflow @@ -90,9 +90,10 @@ Run these commands on Dankserver: ``` cd /home/dank/vLLM git pull origin main +sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service sudo systemctl daemon-reload -sudo systemctl restart vllm-qwen36 -curl -sf http://127.0.0.1:8000/health +sudo systemctl restart frank-default-small +curl -sf http://127.0.0.1:7770/health ``` ## Service Units @@ -101,11 +102,11 @@ curl -sf http://127.0.0.1:8000/health The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95). -### Dankserver — vllm-awq-qwen36 +### Dankserver — frank-default-small -The active vLLM service on Dankserver runs in a Podman/Docker container and serves the Qwen3.6-35B-A3B AWQ 4-bit model on port 34401. The unit file is a physical file on the machine, not a symlink into the repository. +The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses a DFlash draft (`Alittlehammmer/Qwen3.6-35B-A3B-DFlash:Q8_0`) with `--spec-draft-n-max 7`, 3 parallel slots at 262144 context each, and mmproj vision. All 40 layers' MoE experts stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. -> Note: the failed `vllm-qwen36.service` (FP8) and the retired `llama-server.service` (ROCmFPX llama.cpp) are not active. Verify the intended service before a restart. See the repository `git log` for the latest decision. +> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`. ## Validation diff --git a/docs/index.md b/docs/index.md index 284b735..0291ba7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,10 +8,10 @@ Documentation for running vLLM inference on two local machines. graph LR subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04] A[RTX 3090 24GB + 62GB RAM] - B[vLLM in Podman -vllm-awq-qwen36] - C[Qwen3.6-35B-A3B AWQ 4bit -port 34401] + B[llama.cpp DFlash +frank-default-small] + C[Qwen3.6-35B-A3B uncensored Q6_K_P +experts on CPU, port 7770] A --> B --> C end @@ -24,7 +24,7 @@ port 7770] D --> E --> F end - G[LLM Clients / API] -->|HTTP :34401| B + G[LLM Clients / API] -->|HTTP :7770| B G -->|HTTP :7770| E ``` diff --git a/scripts/deploy.sh b/scripts/deploy.sh index a857ad2..31ec1b6 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -13,7 +13,7 @@ # # Targets: # dankclaw - vLLM-ROCm server (frank-default-large.service, port 7770) -# dankserver - vLLM server (vllm-awq-qwen36.service, port 34401) +# dankserver - llama.cpp server (frank-default-small.service, port 7770) # # Example: # ./deploy.sh dankclaw @@ -34,8 +34,8 @@ SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health" REMOTE_USER[dankserver]=dank REMOTE_HOST[dankserver]=dankserver REMOTE_DIR[dankserver]="/home/dank/vLLM" -SERVICE[dankserver]="vllm-awq-qwen36" -SERVICE_HEALTH[dankserver]="http://127.0.0.1:34401/health" +SERVICE[dankserver]="frank-default-small" +SERVICE_HEALTH[dankserver]="http://127.0.0.1:7770/health" TARGET="${1:-}" DRY_RUN=0 @@ -106,8 +106,7 @@ case '${TARGET}' in sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-large.service' /etc/systemd/system/frank-default-large.service ;; dankserver) - # vllm-qwen36.service is a physical file; no symlink needed. - : + sudo ln -sf '${REMOTE_DIR[$TARGET]}/scripts/frank-default-small.service' /etc/systemd/system/frank-default-small.service ;; esac