dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: frank-default-large llama.cpp draft-dflash deployment + benchmark results; retire Ling vLLM references

Browse Source
This commit is contained in:
Frank
2026-08-11 17:24:59 -07:00
parent 668083fe76
commit e92c72fd11
4 changed files with 99 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+3 -1
View File
@@ -100,7 +100,9 @@ curl -sf http://127.0.0.1:7770/health
### Dankclaw — frank-default-large
The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash CIRU INT4 (W4A16) model. It serves under the alias `frank-default-large` on port 7770 with native MTP speculative decoding. The unit file is `scripts/frank-default-large.service` in the repository. The machine symlinks it to `/etc/systemd/system/frank-default-large.service`. The service launches through `scripts/serve-ling3-vllm.sh`, which sources the CIRU runtime and passes the production settings (host `0.0.0.0`, port 7770, gpu memory utilization 0.95).
The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses a **DFlash draft model** (`draft-dflash`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `f16` V on GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 2048 / ubatch 512. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
> Note: this service replaced the retired vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the earlier llama-serve/vLLM units on Dankclaw. See [[frank-default-large-benchmark]] for the benchmark that drove the backend and spec choices.
### Dankserver — frank-default-small