feat: initial vLLM Qwen3.6-35B-A3B-FP8 serve script with DFlash drafting on dankserver
- Podman container with CUDA GPU access - DFlash speculative decoding via --spec-method dflash - FP8 KV cache, 4 parallel streams, 262K context - CPU expert offloading via --cpu-offload-gb - systemd service for persistent runtime
This commit is contained in:
Frank
@@ -0,0 +1,217 @@
|
||||
# Research: vLLM Parameters for Specified Configuration
|
||||
|
||||
## Summary
|
||||
vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Speculative Decoding with DFlash Draft Model
|
||||
|
||||
**Flag syntax:**
|
||||
```bash
|
||||
--spec-method dflash \
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
|
||||
--spec-tokens 8
|
||||
```
|
||||
|
||||
- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`.
|
||||
- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model.
|
||||
- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
|
||||
- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
|
||||
- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`).
|
||||
|
||||
**Additional speculative config options** (from `vllm/engine/arg_utils.py`):
|
||||
- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings
|
||||
- `--spec-tokens` — shorthand for `num_speculative_tokens`
|
||||
|
||||
**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed.
|
||||
|
||||
[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
|
||||
[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
|
||||
[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash)
|
||||
|
||||
---
|
||||
|
||||
### 2. KV Cache Dtype — q8_0 NOT Supported
|
||||
|
||||
**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`):
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| `auto` | Use model data type (default) |
|
||||
| `float16` | FP16 KV cache |
|
||||
| `bfloat16` | BF16 KV cache |
|
||||
| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) |
|
||||
| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) |
|
||||
| `fp8_inc` | FP8 for Intel Gaudi (HPU) |
|
||||
| `fp8_ds_mla` | FP8 for DeepSeek MLA |
|
||||
| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V |
|
||||
| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal |
|
||||
| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V |
|
||||
| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal |
|
||||
| `int4_per_token_head` | INT4 per-token-head |
|
||||
| `int8_per_token_head` | INT8 per-token-head |
|
||||
| `fp8_per_token_head` | FP8 per-token-head |
|
||||
| `nvfp4` | NVFP4 quantization |
|
||||
| `nvfp4_4over6` | NVFP4 4over6 layout |
|
||||
|
||||
**Closest alternatives to q8_0:**
|
||||
- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision
|
||||
- `int8_per_token_head` — 8-bit integer with per-token-head scales
|
||||
- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache
|
||||
|
||||
**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values)
|
||||
|
||||
[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)
|
||||
|
||||
---
|
||||
|
||||
### 3. CPU Expert Offloading for MoE Models
|
||||
|
||||
**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.**
|
||||
|
||||
Available MoE-related flags (from `vllm/engine/arg_utils.py`):
|
||||
- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs)
|
||||
- `--expert-placement-strategy` — controls expert placement strategy
|
||||
- `--moe-backend` — selects the MoE kernel backend
|
||||
|
||||
Available CPU offloading flags:
|
||||
- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading)
|
||||
- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU
|
||||
- `--kv-cache-memory-bytes` — controls KV cache memory allocation
|
||||
|
||||
**How to offload MoE experts to CPU:**
|
||||
Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights:
|
||||
```bash
|
||||
--cpu-offload-gb 32 \
|
||||
--cpu-offload-params "experts.*"
|
||||
```
|
||||
|
||||
This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.
|
||||
|
||||
**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.
|
||||
|
||||
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
|
||||
|
||||
---
|
||||
|
||||
### 4. max-num-seqs Flag — Confirmed
|
||||
|
||||
**Flag:** `--max-num-seqs N`
|
||||
|
||||
- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None`
|
||||
- Limits the maximum number of concurrent sequences (requests) in the scheduler
|
||||
- When `None` (default), vLLM auto-calculates based on available KV cache memory
|
||||
- Setting this constrains concurrency regardless of available resources
|
||||
|
||||
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
|
||||
|
||||
---
|
||||
|
||||
### 5. Podman + vLLM + CUDA GPU Access
|
||||
|
||||
**Exact podman run flags for GPU access:**
|
||||
|
||||
```bash
|
||||
podman run -it \
|
||||
--gpus all \
|
||||
--device /dev/nvidia0 \
|
||||
--device /dev/nvidiactl \
|
||||
--device /dev/nvidia-uvm \
|
||||
--device /dev/nvidia-uvm-tools \
|
||||
--device /dev/nvidia-modeset \
|
||||
--cap-add=SYS_ADMIN \
|
||||
--security-opt apparmor=unconfined \
|
||||
-v /path/to/hf-cache:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
vllm serve <model>
|
||||
```
|
||||
|
||||
**Minimal form (with NVIDIA Container Toolkit installed):**
|
||||
```bash
|
||||
podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>
|
||||
```
|
||||
|
||||
**Key flags:**
|
||||
- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`)
|
||||
- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work)
|
||||
- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations
|
||||
- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access
|
||||
|
||||
**Prerequisites:**
|
||||
- `nvidia-container-toolkit` must be installed on the host
|
||||
- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all`
|
||||
- NVIDIA driver must be installed on the host (not in the container)
|
||||
|
||||
[Source: Podman GPU documentation](https://podman.io/docs/gpu-support)
|
||||
|
||||
---
|
||||
|
||||
### 6. Latest Stable vLLM Version
|
||||
|
||||
**Version: v0.26.0** (released July 25, 2026)
|
||||
|
||||
- 411 commits from 212 contributors (61 new)
|
||||
- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
|
||||
- Available on PyPI: `pip install vllm==0.26.0`
|
||||
- Docker image: `vllm/vllm-openai:latest`
|
||||
|
||||
[Source: PyPI vllm page](https://pypi.org/project/vllm/)
|
||||
[Source: GitHub releases](https://github.com/vllm-project/vllm/releases)
|
||||
|
||||
---
|
||||
|
||||
## Complete Example Command
|
||||
|
||||
Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:
|
||||
|
||||
```bash
|
||||
podman run -it \
|
||||
--gpus all \
|
||||
--device /dev/nvidia0 \
|
||||
--device /dev/nvidiactl \
|
||||
--device /dev/nvidia-uvm \
|
||||
--device /dev/nvidia-uvm-tools \
|
||||
--cap-add=SYS_ADMIN \
|
||||
--security-opt apparmor=unconfined \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--spec-method dflash \
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
|
||||
--spec-tokens 8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-num-seqs 32 \
|
||||
--tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
## Sources
|
||||
|
||||
### Kept
|
||||
- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information
|
||||
- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions
|
||||
- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig
|
||||
- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types
|
||||
- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples
|
||||
- [PyPI vllm](https://pypi.org/project/vllm/) — version history
|
||||
|
||||
### Dropped
|
||||
- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
|
||||
- Wikipedia vLLM article — not authoritative for CLI flags
|
||||
|
||||
## Gaps
|
||||
|
||||
1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`.
|
||||
|
||||
2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.
|
||||
|
||||
3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.
|
||||
|
||||
4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.
|
||||
|
||||
## Residual Risks
|
||||
|
||||
- DFlash may require specific attention backends not available on all GPU architectures
|
||||
- CPU offloading MoE experts will add significant latency; test thoroughly before production use
|
||||
- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure
|
||||
@@ -31,8 +31,8 @@ echo ""
|
||||
# Remove existing container if present
|
||||
podman rm -f "${CONTAINER_NAME}" 2>/dev/null || true
|
||||
|
||||
# Run vLLM in Podman container
|
||||
exec podman run -d --name "${CONTAINER_NAME}" \
|
||||
# Run vLLM in Podman container (foreground mode for systemd)
|
||||
exec podman run --name "${CONTAINER_NAME}" \
|
||||
--device /dev/nvidia0 \
|
||||
--device /dev/nvidiactl \
|
||||
--device /dev/nvidia-uvm \
|
||||
@@ -41,7 +41,6 @@ exec podman run -d --name "${CONTAINER_NAME}" \
|
||||
-v "${GPU_LIBS}:/usr/lib/x86_64-linux-gnu:ro" \
|
||||
-v "${HF_CACHE}:/root/.cache/huggingface:ro" \
|
||||
-p "${PORT}:${PORT}" \
|
||||
--cap-add SYS_ADMIN \
|
||||
docker.io/vllm/vllm-openai:latest \
|
||||
vllm serve "${MODEL}" \
|
||||
--port "${PORT}" \
|
||||
@@ -50,7 +49,7 @@ exec podman run -d --name "${CONTAINER_NAME}" \
|
||||
--gpu-memory-utilization "${GPU_MEM_UTIL}" \
|
||||
--max-num-seqs "${MAX_NUM_SEQS}" \
|
||||
--max-model-len "${MAX_MODEL_LEN}" \
|
||||
--speculative-method dflash \
|
||||
--speculative-model "${DRAFT_MODEL}" \
|
||||
--num-speculative-tokens "${NUM_SPEC_TOKENS}" \
|
||||
--spec-method dflash \
|
||||
--spec-model "${DRAFT_MODEL}" \
|
||||
--spec-tokens "${NUM_SPEC_TOKENS}" \
|
||||
--trust-remote-code
|
||||
|
||||
@@ -7,6 +7,7 @@ Wants=network-online.target
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
WorkingDirectory=/home/dank/Projects/vllmScripts
|
||||
Environment="PORT=8000"
|
||||
Environment="GPU_MEM_UTIL=0.95"
|
||||
Environment="MAX_NUM_SEQS=4"
|
||||
@@ -17,16 +18,7 @@ ExecStart=/home/dank/Projects/vllmScripts/scripts/serve-vllm-qwen36-dflash.sh
|
||||
ExecStop=/usr/bin/podman stop vllm-qwen36-dflash
|
||||
ExecStopPost=/usr/bin/podman rm -f vllm-qwen36-dflash
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
TimeoutStopSec=60
|
||||
|
||||
# Ensure GPU devices are available
|
||||
DevicePolicy=closed
|
||||
DeviceAllow=/dev/nvidia0 rwm
|
||||
DeviceAllow=/dev/nvidiactl rwm
|
||||
DeviceAllow=/dev/nvidia-uvm rwm
|
||||
DeviceAllow=/dev/nvidia-uvm-tools rwm
|
||||
DeviceAllow=/dev/nvidia-modeset rwm
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
Reference in New Issue
Block a user