docs: frank-default-small now uses ngram-mod, DFlash retired
- SWAT from DFlash (see ngram-mod benchmarks below); draft model not loaded. Active spec: --spec-type ngram-mod n_min=48/n_max=64/n_match=24. - Update Dankserver specs, deployment.md, DFlash doc to ngram-mod. - VRAM footprint: 14 GB (was 20 GB), ~10 GB free headroom.
This commit is contained in:
@@ -115,7 +115,11 @@ The DFlash draft is not a standalone model. It must pair with its target model.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The active Dankserver deployment runs **llama.cpp + DFlash** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 with the `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` Q8_0 draft, 3 slots at 200000 context each, and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The active Dankserver deployment runs **llama.cpp** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 using **ngram-mod** speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
|
||||
|
||||
|
||||
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with a DFlash draft. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 20 GB of the 24 GB VRAM and about 32 GB of RAM.
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs 3 parallel slots at 200000 context each (pool 600000). All 40 layers' MoE expert weights sit on CPU (`--n-cpu-moe 40`) while non-expert tensors and the full-attention KV cache stay on GPU (`--gpu-layers all`, KV K=`q8_0`, V=`f16`). It uses batch 3072 / ubatch 1536 with `--ctx-checkpoints` and `--no-mmap`. At load it uses about 14 GB of the 24 GB VRAM and about 32 GB of RAM.
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context, the full-attention KV cache plus non-expert weights and the batch 3072/1536 activation buffers consume about 20 GB, leaving about 4 GB of headroom. MoE experts must stay on CPU.
|
||||
- 24 GB VRAM limits the GPU-resident footprint. With 3 x 200000 context plus batch 3072/1536 activation buffers, the non-expert weights and KV cache consume about 14 GB, leaving about 10 GB of headroom. MoE experts stay on CPU; `--no-mmap` preloads expert tensors into RAM, eliminating decode page faults (uses ~32 GB of the 62 GB available).
|
||||
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | vLLM-ROCm (CIRU INT4) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + DFlash, MoE offload) | frank-default-small | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -104,7 +104,7 @@ The active service on Dankclaw runs a vLLM-ROCm runtime for the Ling-3.0-flash C
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses a DFlash draft (`Alittlehammmer/Qwen3.6-35B-A3B-DFlash:Q8_0`) with `--spec-draft-n-max 7`, 3 parallel slots at 200000 context each (pool 600000), and mmproj vision. All 40 layers' MoE experts stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. It runs batch 3072 / ubatch 1536 with `--ctx-checkpoints 4` and `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), 3 parallel slots at 200000 context each (pool 600000), and mmproj vision. All 40 layers' MoE expert weights stay on CPU (`--n-cpu-moe 40`) with non-expert tensors on GPU (`--gpu-layers all`); the full-attention KV cache stays on GPU with K quantized to `q8_0` and V at `f16`. It runs batch 3072 / ubatch 1536 with `--ctx-checkpoints 4` and `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
|
||||
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user