dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs+deploy: Dankserver now llama.cpp frank-default-small (HauhauCS Q6_K_P + DFlash)

Browse Source 
Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index,
Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream
llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0
draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16).
deploy.sh now maps dankserver to frank-default-small and symlinks its unit.
models.json: dankserver baseUrl -> :7770, contextWindow -> 262144.
This commit is contained in:
Frank committed 2026-08-11 01:18:29 -07:00
1 parent 6a86a9df93
commit 35c44c4123
8 files changed
+34 -32
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/deployment.md
+7 -6
View File
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
| Dankserver | dank@dankserver | vLLM (Docker) | vllm-awq-qwen36 | 34401 |
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 |
## Workflow
@@ -90,9 +90,10 @@ Run these commands on Dankserver:
```
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart vllm-qwen36
curl -sf http://127.0.0.1:8000/health
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:7770/health
```
## Service Units
@@ -101,11 +102,11 @@ curl -sf http://127.0.0.1:8000/health
The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95).
### Dankserver — vllm-awq-qwen36
### Dankserver — frank-default-small
The active vLLM service on Dankserver runs in a Podman/Docker container and serves the Qwen3.6-35B-A3B AWQ 4-bit model on port 34401. The unit file is a physical file on the machine, not a symlink into the repository.
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses a DFlash draft (`Alittlehammmer/Qwen3.6-35B-A3B-DFlash:Q8_0`) with `--spec-draft-n-max 7`, 3 parallel slots at 262144 context each, and mmproj vision. All 40 layers' MoE experts stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
> Note: the failed `vllm-qwen36.service` (FP8) and the retired `llama-server.service` (ROCmFPX llama.cpp) are not active. Verify the intended service before a restart. See the repository `git log` for the latest decision.
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
## Validation