dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2a124876538f9cc612378a7ad4c046b0f521dad0
serverAppConfigs/serverAppConfigs-docs/LLM/vllm_research_findings.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

237 lines
10 KiB
Markdown
Raw Blame History

---
type: reference
description: "Upstream support survey for speculative decoding: which flags, releases and pull requests carry each method."
tags:
- llm
- vllm
timestamp: 2026-08-28
---
# Research: vLLM Parameters for Specified Configuration
## Summary
vLLM v0.26.0 (released 2026-07-25, the current stable when this survey was
gathered on 2026-08-28; PyPI shows v0.32.0 as the latest on 2026-10-19, so
re-check this page before acting on it) supports DFlash speculative decoding.
The preferred CLI is the `--speculative-config` JSON object; `--spec-method`
and `--spec-model` are the per-field aliases defined in
`vllm/engine/arg_utils.py`. KV cache dtype `q8_0` is **not supported** — only
fp8 variants and turboquant options are available. vLLM does not have a
dedicated MoE expert CPU offload flag; general CPU offloading via
`--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is
confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container
toolkit.
## Findings
### 1. Speculative Decoding with DFlash Draft Model
**Flag syntax:**
```bash
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8
```
- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`.
- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model.
- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`).
**Additional speculative config options** (from `vllm/engine/arg_utils.py`):
- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings
- `--spec-tokens` — shorthand for `num_speculative_tokens`
**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed.
[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash)
---
### 2. KV Cache Dtype — q8_0 NOT Supported
**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`):
| Value | Description |
|-------|-------------|
| `auto` | Use model data type (default) |
| `float16` | FP16 KV cache |
| `bfloat16` | BF16 KV cache |
| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) |
| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) |
| `fp8_inc` | FP8 for Intel Gaudi (HPU) |
| `fp8_ds_mla` | FP8 for DeepSeek MLA |
| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V |
| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal |
| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V |
| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal |
| `int4_per_token_head` | INT4 per-token-head |
| `int8_per_token_head` | INT8 per-token-head |
| `fp8_per_token_head` | FP8 per-token-head |
| `nvfp4` | NVFP4 quantization |
| `nvfp4_4over6` | NVFP4 4over6 layout |
**Closest alternatives to q8_0:**
- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision
- `int8_per_token_head` — 8-bit integer with per-token-head scales
- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache
**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values)
[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)
---
### 3. CPU Expert Offloading for MoE Models
**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.**
Available MoE-related flags (from `vllm/engine/arg_utils.py`):
- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs)
- `--expert-placement-strategy` — controls expert placement strategy
- `--moe-backend` — selects the MoE kernel backend
Available CPU offloading flags:
- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading)
- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU
- `--kv-cache-memory-bytes` — controls KV cache memory allocation
**How to offload MoE experts to CPU:**
Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights:
```bash
--cpu-offload-gb 32 \
--cpu-offload-params "experts.*"
```
This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.
**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
---
### 4. max-num-seqs Flag — Confirmed
**Flag:** `--max-num-seqs N`
- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None`
- Limits the maximum number of concurrent sequences (requests) in the scheduler
- When `None` (default), vLLM auto-calculates based on available KV cache memory
- Setting this constrains concurrency regardless of available resources
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
---
### 5. Podman + vLLM + CUDA GPU Access
**Exact podman run flags for GPU access:**
```bash
podman run -it \
--gpus all \
--device /dev/nvidia0 \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools \
--device /dev/nvidia-modeset \
--cap-add=SYS_ADMIN \
--security-opt apparmor=unconfined \
-v /path/to/hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:latest \
vllm serve <model>
```
**Minimal form (with NVIDIA Container Toolkit installed):**
```bash
podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>
```
**Key flags:**
- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`)
- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work)
- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations
- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access
**Prerequisites:**
- `nvidia-container-toolkit` must be installed on the host
- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all`
- NVIDIA driver must be installed on the host (not in the container)
[Source: Podman GPU documentation](https://podman.io/docs/gpu-support)
---
### 6. Latest Stable vLLM Version
**Version: v0.26.0** (released July 25, 2026)
- 411 commits from 212 contributors (61 new)
- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
- Available on PyPI: `pip install vllm==0.26.0`
- Docker image: `vllm/vllm-openai:latest`
[Source: PyPI vllm page](https://pypi.org/project/vllm/)
[Source: GitHub releases](https://github.com/vllm-project/vllm/releases)
---
## Complete Example Command
Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:
```bash
podman run -it \
--gpus all \
--device /dev/nvidia0 \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools \
--cap-add=SYS_ADMIN \
--security-opt apparmor=unconfined \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
vllm serve Qwen/Qwen3.6-35B-A3B \
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8 \
--kv-cache-dtype fp8 \
--max-num-seqs 32 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9
```
## Sources
### Kept
- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information
- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions
- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig
- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types
- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples
- [PyPI vllm](https://pypi.org/project/vllm/) — version history
### Dropped
- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
- Wikipedia vLLM article — not authoritative for CLI flags
## Gaps
1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`.
2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.
3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.
4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.
## Residual Risks
- DFlash may require specific attention backends not available on all GPU architectures
- CPU offloading MoE experts will add significant latency; test thoroughly before production use
- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure
Reference in new issue View Git Blame Copy Permalink