serverAppConfigs/docs/vllm_research_findings.md

Research: vLLM Parameters for Specified Configuration

Summary

vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the --spec-method dflash and --spec-model flags. KV cache dtype q8_0 is not supported — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via --cpu-offload-gb is the closest option. The --max-num-seqs flag is confirmed. Podman GPU access requires --gpus all with the NVIDIA container toolkit.

Findings

1. Speculative Decoding with DFlash Draft Model

Flag syntax:

--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8

• --spec-method dflash — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as DFlashModelTypes = Literal["dflash"] in vllm/config/speculative.py.
• --spec-model z-lab/Qwen3.6-35B-A3B-DFlash — specifies the HuggingFace path to the DFlash draft model.
• --spec-tokens N — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
• DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
• The DFlash draft model is not standalone — it must be paired with its target model (Qwen/Qwen3.6-35B-A3B).

Additional speculative config options (from vllm/engine/arg_utils.py):

• --speculative-config '{"key": "value"}' — full JSON config for advanced settings
• --spec-tokens — shorthand for num_speculative_tokens

Caveat: DFlash may require a non-causal-capable attention backend for the draft model. Set --speculative-config '{"attention_backend": "FLASH_ATTN"}' if needed.

Source: vLLM v0.26.0 release notes Source: vllm/config/speculative.py source Source: z-lab/Qwen3.6-35B-A3B-DFlash model card

---

2. KV Cache Dtype — q8_0 NOT Supported

q8_0 is NOT a valid KV cache dtype in vLLM. The supported CacheDType values (from vllm/config/cache.py):

Value	Description
auto	Use model data type (default)
float16	FP16 KV cache
bfloat16	BF16 KV cache
fp8 / fp8_e4m3	FP8 E4M3 (CUDA 11.8+)
fp8_e5m2	FP8 E5M2 (CUDA 11.8+)
fp8_inc	FP8 for Intel Gaudi (HPU)
fp8_ds_mla	FP8 for DeepSeek MLA
turboquant_k8v4	TurboQuant 8-bit K, 4-bit V
turboquant_4bit_nc	TurboQuant 4-bit non-causal
turboquant_k3v4_nc	TurboQuant 3-bit K, 4-bit V
turboquant_3bit_nc	TurboQuant 3-bit non-causal
int4_per_token_head	INT4 per-token-head
int8_per_token_head	INT8 per-token-head
fp8_per_token_head	FP8 per-token-head
nvfp4	NVFP4 quantization
nvfp4_4over6	NVFP4 4over6 layout

Closest alternatives to q8_0:

• fp8 (alias for fp8_e4m3) — 8-bit FP8, closest to q8_0 in precision
• int8_per_token_head — 8-bit integer with per-token-head scales
• turboquant_k8v4 — 8-bit K cache, 4-bit V cache

CLI flag: --kv-cache-dtype fp8 (or any of the above values)

Source: vllm/config/cache.py source

---

3. CPU Expert Offloading for MoE Models

vLLM does NOT have a dedicated --cpu-offload-gate or --expert-offload-cpu flag.

Available MoE-related flags (from vllm/engine/arg_utils.py):

• --enable-expert-parallel — enables expert parallelism (distributes experts across GPUs)
• --expert-placement-strategy — controls expert placement strategy
• --moe-backend — selects the MoE kernel backend

Available CPU offloading flags:

• --cpu-offload-gb N — general CPU offloading of N GB of model parameters (UVA offloading)
• --cpu-offload-params "pattern" — offload specific parameter patterns to CPU
• --kv-cache-memory-bytes — controls KV cache memory allocation

How to offload MoE experts to CPU: Use --cpu-offload-gb with --cpu-offload-params "experts.*" to offload expert weights:

--cpu-offload-gb 32 \
--cpu-offload-params "experts.*"

This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.

Caveat: CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.

Source: vllm/engine/arg_utils.py source

---

4. max-num-seqs Flag — Confirmed

Flag: --max-num-seqs N

• Confirmed in vllm/engine/arg_utils.py: max_num_seqs: int | None = None
• Limits the maximum number of concurrent sequences (requests) in the scheduler
• When None (default), vLLM auto-calculates based on available KV cache memory
• Setting this constrains concurrency regardless of available resources

Source: vllm/engine/arg_utils.py source

---

5. Podman + vLLM + CUDA GPU Access

Exact podman run flags for GPU access:

podman run -it \
  --gpus all \
  --device /dev/nvidia0 \
  --device /dev/nvidiactl \
  --device /dev/nvidia-uvm \
  --device /dev/nvidia-uvm-tools \
  --device /dev/nvidia-modeset \
  --cap-add=SYS_ADMIN \
  --security-opt apparmor=unconfined \
  -v /path/to/hf-cache:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  vllm serve <model>

Minimal form (with NVIDIA Container Toolkit installed):

podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>

Key flags:

• --gpus all — exposes all GPUs to the container (requires nvidia-container-toolkit)
• --device /dev/nvidia* — manual device passthrough (fallback if --gpus doesn't work)
• --cap-add=SYS_ADMIN — needed for NVIDIA driver access in some Podman configurations
• --security-opt apparmor=unconfined — disables AppArmor which can block GPU access

Prerequisites:

• nvidia-container-toolkit must be installed on the host
• Podman's containers.conf should have nvidia in the devices section, or use --gpus all
• NVIDIA driver must be installed on the host (not in the container)

Source: Podman GPU documentation

---

6. Latest Stable vLLM Version

Version: v0.26.0 (released July 25, 2026)

• 411 commits from 212 contributors (61 new)
• Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
• Available on PyPI: pip install vllm==0.26.0
• Docker image: vllm/vllm-openai:latest

Source: PyPI vllm page Source: GitHub releases

---

Complete Example Command

Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:

podman run -it \
  --gpus all \
  --device /dev/nvidia0 \
  --device /dev/nvidiactl \
  --device /dev/nvidia-uvm \
  --device /dev/nvidia-uvm-tools \
  --cap-add=SYS_ADMIN \
  --security-opt apparmor=unconfined \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3.6-35B-A3B \
  --spec-method dflash \
  --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
  --spec-tokens 8 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9

Sources

Kept

• vLLM v0.26.0 Release Notes — authoritative release information
• vllm/engine/arg_utils.py — complete CLI argument definitions
• vllm/config/cache.py — CacheDType enum and CacheConfig
• vllm/config/speculative.py — SpeculativeConfig and method types
• z-lab/Qwen3.6-35B-A3B-DFlash — DFlash model card with usage examples
• PyPI vllm — version history

Dropped

• Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
• Wikipedia vLLM article — not authoritative for CLI flags

Gaps

1. DFlash attention backend requirements — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need --attention-backend flash-attn or flashinfer.

2. MoE CPU offload performance impact — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.

3. Podman GPU on specific distros — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.

4. DFlash + MoE compatibility — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.

Residual Risks

• DFlash may require specific attention backends not available on all GPU architectures
• CPU offloading MoE experts will add significant latency; test thoroughly before production use
• q8_0 is definitively not supported; using an unsupported dtype will cause startup failure