# Research: vLLM Parameters for Specified Configuration

## Summary
vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit.

## Findings

### 1. Speculative Decoding with DFlash Draft Model

**Flag syntax:**
```bash
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8
```

- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`.
- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model.
- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`).

**Additional speculative config options** (from `vllm/engine/arg_utils.py`):
- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings
- `--spec-tokens` — shorthand for `num_speculative_tokens`

**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed.

[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash)

---

### 2. KV Cache Dtype — q8_0 NOT Supported

**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`):

| Value | Description |
|-------|-------------|
| `auto` | Use model data type (default) |
| `float16` | FP16 KV cache |
| `bfloat16` | BF16 KV cache |
| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) |
| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) |
| `fp8_inc` | FP8 for Intel Gaudi (HPU) |
| `fp8_ds_mla` | FP8 for DeepSeek MLA |
| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V |
| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal |
| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V |
| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal |
| `int4_per_token_head` | INT4 per-token-head |
| `int8_per_token_head` | INT8 per-token-head |
| `fp8_per_token_head` | FP8 per-token-head |
| `nvfp4` | NVFP4 quantization |
| `nvfp4_4over6` | NVFP4 4over6 layout |

**Closest alternatives to q8_0:**
- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision
- `int8_per_token_head` — 8-bit integer with per-token-head scales
- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache

**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values)

[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)

---

### 3. CPU Expert Offloading for MoE Models

**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.**

Available MoE-related flags (from `vllm/engine/arg_utils.py`):
- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs)
- `--expert-placement-strategy` — controls expert placement strategy
- `--moe-backend` — selects the MoE kernel backend

Available CPU offloading flags:
- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading)
- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU
- `--kv-cache-memory-bytes` — controls KV cache memory allocation

**How to offload MoE experts to CPU:**
Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights:
```bash
--cpu-offload-gb 32 \
--cpu-offload-params "experts.*"
```

This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.

**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.

[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)

---

### 4. max-num-seqs Flag — Confirmed

**Flag:** `--max-num-seqs N`

- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None`
- Limits the maximum number of concurrent sequences (requests) in the scheduler
- When `None` (default), vLLM auto-calculates based on available KV cache memory
- Setting this constrains concurrency regardless of available resources

[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)

---

### 5. Podman + vLLM + CUDA GPU Access

**Exact podman run flags for GPU access:**

```bash
podman run -it \
  --gpus all \
  --device /dev/nvidia0 \
  --device /dev/nvidiactl \
  --device /dev/nvidia-uvm \
  --device /dev/nvidia-uvm-tools \
  --device /dev/nvidia-modeset \
  --cap-add=SYS_ADMIN \
  --security-opt apparmor=unconfined \
  -v /path/to/hf-cache:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  vllm serve <model>
```

**Minimal form (with NVIDIA Container Toolkit installed):**
```bash
podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>
```

**Key flags:**
- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`)
- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work)
- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations
- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access

**Prerequisites:**
- `nvidia-container-toolkit` must be installed on the host
- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all`
- NVIDIA driver must be installed on the host (not in the container)

[Source: Podman GPU documentation](https://podman.io/docs/gpu-support)

---

### 6. Latest Stable vLLM Version

**Version: v0.26.0** (released July 25, 2026)

- 411 commits from 212 contributors (61 new)
- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
- Available on PyPI: `pip install vllm==0.26.0`
- Docker image: `vllm/vllm-openai:latest`

[Source: PyPI vllm page](https://pypi.org/project/vllm/)
[Source: GitHub releases](https://github.com/vllm-project/vllm/releases)

---

## Complete Example Command

Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:

```bash
podman run -it \
  --gpus all \
  --device /dev/nvidia0 \
  --device /dev/nvidiactl \
  --device /dev/nvidia-uvm \
  --device /dev/nvidia-uvm-tools \
  --cap-add=SYS_ADMIN \
  --security-opt apparmor=unconfined \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3.6-35B-A3B \
  --spec-method dflash \
  --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
  --spec-tokens 8 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9
```

## Sources

### Kept
- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information
- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions
- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig
- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types
- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples
- [PyPI vllm](https://pypi.org/project/vllm/) — version history

### Dropped
- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
- Wikipedia vLLM article — not authoritative for CLI flags

## Gaps

1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`.

2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.

3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.

4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.

## Residual Risks

- DFlash may require specific attention backends not available on all GPU architectures
- CPU offloading MoE experts will add significant latency; test thoroughly before production use
- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure