docs: frank-default-small now uses ngram-mod, DFlash retired
- SWAT from DFlash (see ngram-mod benchmarks below); draft model not loaded. Active spec: --spec-type ngram-mod n_min=48/n_max=64/n_match=24. - Update Dankserver specs, deployment.md, DFlash doc to ngram-mod. - VRAM footprint: 14 GB (was 20 GB), ~10 GB free headroom.
This commit is contained in:
1 parent
fd528ab364
commit
677b4af2a4
3 files changed
+9
-5
No files matched your search
+2
-2
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -104,7 +104,7 @@ The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash C
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses a DFlash draft (`Alittlehammmer/Qwen3.6-35B-A3B-DFlash:Q8_0`) with `--spec-draft-n-max 7`, 3 parallel slots at 200000 context each (pool 600000), and mmproj vision. All 40 layers' MoE experts stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. It runs batch 3072 / ubatch 1536 with `--ctx-checkpoints 4` and `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), 3 parallel slots at 200000 context each (pool 600000), and mmproj vision. All 40 layers' MoE expert weights stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. It runs batch 3072 / ubatch 1536 with `--ctx-checkpoints 4` and `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
|
||||
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
|
||||
|
||||
|
||||
Reference in new issue
Block a user