dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial vLLM Qwen3.6-35B-A3B-FP8 serve script with DFlash drafting on dankserver

Browse Source 
- Podman container with CUDA GPU access
- DFlash speculative decoding via --spec-method dflash
- FP8 KV cache, 4 parallel streams, 262K context
- CPU expert offloading via --cpu-offload-gb
- systemd service for persistent runtime
This commit is contained in:
Frank
2026-08-07 18:28:38 -07:00
parent 5452bb63f1
commit 9971bdef0f
3 changed files with 224 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/vllm_research_findings.md
+217
View File
@@ -0,0 +1,217 @@
# Research: vLLM Parameters for Specified Configuration
## Summary
vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit.
## Findings
### 1. Speculative Decoding with DFlash Draft Model
**Flag syntax:**
```bash
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8
```
- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`.
- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model.
- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`).
**Additional speculative config options** (from `vllm/engine/arg_utils.py`):
- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings
- `--spec-tokens` — shorthand for `num_speculative_tokens`
**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed.
[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash)
---
### 2. KV Cache Dtype — q8_0 NOT Supported
**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`):
| Value | Description |
|-------|-------------|
| `auto` | Use model data type (default) |
| `float16` | FP16 KV cache |
| `bfloat16` | BF16 KV cache |
| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) |
| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) |
| `fp8_inc` | FP8 for Intel Gaudi (HPU) |
| `fp8_ds_mla` | FP8 for DeepSeek MLA |
| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V |
| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal |
| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V |
| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal |
| `int4_per_token_head` | INT4 per-token-head |
| `int8_per_token_head` | INT8 per-token-head |
| `fp8_per_token_head` | FP8 per-token-head |
| `nvfp4` | NVFP4 quantization |
| `nvfp4_4over6` | NVFP4 4over6 layout |
**Closest alternatives to q8_0:**
- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision
- `int8_per_token_head` — 8-bit integer with per-token-head scales
- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache
**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values)
[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)
---
### 3. CPU Expert Offloading for MoE Models
**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.**
Available MoE-related flags (from `vllm/engine/arg_utils.py`):
- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs)
- `--expert-placement-strategy` — controls expert placement strategy
- `--moe-backend` — selects the MoE kernel backend
Available CPU offloading flags:
- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading)
- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU
- `--kv-cache-memory-bytes` — controls KV cache memory allocation
**How to offload MoE experts to CPU:**
Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights:
```bash
--cpu-offload-gb 32 \
--cpu-offload-params "experts.*"
```
This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.
**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
---
### 4. max-num-seqs Flag — Confirmed
**Flag:** `--max-num-seqs N`
- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None`
- Limits the maximum number of concurrent sequences (requests) in the scheduler
- When `None` (default), vLLM auto-calculates based on available KV cache memory
- Setting this constrains concurrency regardless of available resources
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
---
### 5. Podman + vLLM + CUDA GPU Access
**Exact podman run flags for GPU access:**
```bash
podman run -it \
--gpus all \
--device /dev/nvidia0 \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools \
--device /dev/nvidia-modeset \
--cap-add=SYS_ADMIN \
--security-opt apparmor=unconfined \
-v /path/to/hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:latest \
vllm serve <model>
```
**Minimal form (with NVIDIA Container Toolkit installed):**
```bash
podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>
```
**Key flags:**
- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`)
- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work)
- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations
- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access
**Prerequisites:**
- `nvidia-container-toolkit` must be installed on the host
- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all`
- NVIDIA driver must be installed on the host (not in the container)
[Source: Podman GPU documentation](https://podman.io/docs/gpu-support)
---
### 6. Latest Stable vLLM Version
**Version: v0.26.0** (released July 25, 2026)
- 411 commits from 212 contributors (61 new)
- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
- Available on PyPI: `pip install vllm==0.26.0`
- Docker image: `vllm/vllm-openai:latest`
[Source: PyPI vllm page](https://pypi.org/project/vllm/)
[Source: GitHub releases](https://github.com/vllm-project/vllm/releases)
---
## Complete Example Command
Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:
```bash
podman run -it \
--gpus all \
--device /dev/nvidia0 \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools \
--cap-add=SYS_ADMIN \
--security-opt apparmor=unconfined \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
vllm serve Qwen/Qwen3.6-35B-A3B \
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8 \
--kv-cache-dtype fp8 \
--max-num-seqs 32 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9
```
## Sources
### Kept
- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information
- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions
- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig
- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types
- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples
- [PyPI vllm](https://pypi.org/project/vllm/) — version history
### Dropped
- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
- Wikipedia vLLM article — not authoritative for CLI flags
## Gaps
1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`.
2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.
3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.
4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.
## Residual Risks
- DFlash may require specific attention backends not available on all GPU architectures
- CPU offloading MoE experts will add significant latency; test thoroughly before production use
- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure