From 9971bdef0ff7512db7c956b3a0a5f4db2ad82c42 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 7 Aug 2026 18:28:38 -0700 Subject: [PATCH] feat: initial vLLM Qwen3.6-35B-A3B-FP8 serve script with DFlash drafting on dankserver - Podman container with CUDA GPU access - DFlash speculative decoding via --spec-method dflash - FP8 KV cache, 4 parallel streams, 262K context - CPU expert offloading via --cpu-offload-gb - systemd service for persistent runtime --- docs/vllm_research_findings.md | 217 ++++++++++++++++++++++++++++ scripts/serve-vllm-qwen36-dflash.sh | 11 +- scripts/vllm-qwen36.service | 12 +- 3 files changed, 224 insertions(+), 16 deletions(-) create mode 100644 docs/vllm_research_findings.md diff --git a/docs/vllm_research_findings.md b/docs/vllm_research_findings.md new file mode 100644 index 0000000..65c5384 --- /dev/null +++ b/docs/vllm_research_findings.md @@ -0,0 +1,217 @@ +# Research: vLLM Parameters for Specified Configuration + +## Summary +vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit. + +## Findings + +### 1. Speculative Decoding with DFlash Draft Model + +**Flag syntax:** +```bash +--spec-method dflash \ +--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \ +--spec-tokens 8 +``` + +- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`. +- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model. +- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput. +- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914). +- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`). + +**Additional speculative config options** (from `vllm/engine/arg_utils.py`): +- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings +- `--spec-tokens` — shorthand for `num_speculative_tokens` + +**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed. + +[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) +[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) +[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) + +--- + +### 2. KV Cache Dtype — q8_0 NOT Supported + +**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`): + +| Value | Description | +|-------|-------------| +| `auto` | Use model data type (default) | +| `float16` | FP16 KV cache | +| `bfloat16` | BF16 KV cache | +| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) | +| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) | +| `fp8_inc` | FP8 for Intel Gaudi (HPU) | +| `fp8_ds_mla` | FP8 for DeepSeek MLA | +| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V | +| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal | +| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V | +| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal | +| `int4_per_token_head` | INT4 per-token-head | +| `int8_per_token_head` | INT8 per-token-head | +| `fp8_per_token_head` | FP8 per-token-head | +| `nvfp4` | NVFP4 quantization | +| `nvfp4_4over6` | NVFP4 4over6 layout | + +**Closest alternatives to q8_0:** +- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision +- `int8_per_token_head` — 8-bit integer with per-token-head scales +- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache + +**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values) + +[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) + +--- + +### 3. CPU Expert Offloading for MoE Models + +**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.** + +Available MoE-related flags (from `vllm/engine/arg_utils.py`): +- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs) +- `--expert-placement-strategy` — controls expert placement strategy +- `--moe-backend` — selects the MoE kernel backend + +Available CPU offloading flags: +- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading) +- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU +- `--kv-cache-memory-bytes` — controls KV cache memory allocation + +**How to offload MoE experts to CPU:** +Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights: +```bash +--cpu-offload-gb 32 \ +--cpu-offload-params "experts.*" +``` + +This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing. + +**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization. + +[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) + +--- + +### 4. max-num-seqs Flag — Confirmed + +**Flag:** `--max-num-seqs N` + +- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None` +- Limits the maximum number of concurrent sequences (requests) in the scheduler +- When `None` (default), vLLM auto-calculates based on available KV cache memory +- Setting this constrains concurrency regardless of available resources + +[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) + +--- + +### 5. Podman + vLLM + CUDA GPU Access + +**Exact podman run flags for GPU access:** + +```bash +podman run -it \ + --gpus all \ + --device /dev/nvidia0 \ + --device /dev/nvidiactl \ + --device /dev/nvidia-uvm \ + --device /dev/nvidia-uvm-tools \ + --device /dev/nvidia-modeset \ + --cap-add=SYS_ADMIN \ + --security-opt apparmor=unconfined \ + -v /path/to/hf-cache:/root/.cache/huggingface \ + vllm/vllm-openai:latest \ + vllm serve +``` + +**Minimal form (with NVIDIA Container Toolkit installed):** +```bash +podman run -it --gpus all vllm/vllm-openai:latest vllm serve +``` + +**Key flags:** +- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`) +- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work) +- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations +- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access + +**Prerequisites:** +- `nvidia-container-toolkit` must be installed on the host +- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all` +- NVIDIA driver must be installed on the host (not in the container) + +[Source: Podman GPU documentation](https://podman.io/docs/gpu-support) + +--- + +### 6. Latest Stable vLLM Version + +**Version: v0.26.0** (released July 25, 2026) + +- 411 commits from 212 contributors (61 new) +- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements +- Available on PyPI: `pip install vllm==0.26.0` +- Docker image: `vllm/vllm-openai:latest` + +[Source: PyPI vllm page](https://pypi.org/project/vllm/) +[Source: GitHub releases](https://github.com/vllm-project/vllm/releases) + +--- + +## Complete Example Command + +Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding: + +```bash +podman run -it \ + --gpus all \ + --device /dev/nvidia0 \ + --device /dev/nvidiactl \ + --device /dev/nvidia-uvm \ + --device /dev/nvidia-uvm-tools \ + --cap-add=SYS_ADMIN \ + --security-opt apparmor=unconfined \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + vllm/vllm-openai:latest \ + vllm serve Qwen/Qwen3.6-35B-A3B \ + --spec-method dflash \ + --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \ + --spec-tokens 8 \ + --kv-cache-dtype fp8 \ + --max-num-seqs 32 \ + --tensor-parallel-size 1 \ + --gpu-memory-utilization 0.9 +``` + +## Sources + +### Kept +- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information +- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions +- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig +- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types +- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples +- [PyPI vllm](https://pypi.org/project/vllm/) — version history + +### Dropped +- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details +- Wikipedia vLLM article — not authoritative for CLI flags + +## Gaps + +1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`. + +2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified. + +3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration. + +4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes. + +## Residual Risks + +- DFlash may require specific attention backends not available on all GPU architectures +- CPU offloading MoE experts will add significant latency; test thoroughly before production use +- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure diff --git a/scripts/serve-vllm-qwen36-dflash.sh b/scripts/serve-vllm-qwen36-dflash.sh index 01f8bfa..a2811dc 100755 --- a/scripts/serve-vllm-qwen36-dflash.sh +++ b/scripts/serve-vllm-qwen36-dflash.sh @@ -31,8 +31,8 @@ echo "" # Remove existing container if present podman rm -f "${CONTAINER_NAME}" 2>/dev/null || true -# Run vLLM in Podman container -exec podman run -d --name "${CONTAINER_NAME}" \ +# Run vLLM in Podman container (foreground mode for systemd) +exec podman run --name "${CONTAINER_NAME}" \ --device /dev/nvidia0 \ --device /dev/nvidiactl \ --device /dev/nvidia-uvm \ @@ -41,7 +41,6 @@ exec podman run -d --name "${CONTAINER_NAME}" \ -v "${GPU_LIBS}:/usr/lib/x86_64-linux-gnu:ro" \ -v "${HF_CACHE}:/root/.cache/huggingface:ro" \ -p "${PORT}:${PORT}" \ - --cap-add SYS_ADMIN \ docker.io/vllm/vllm-openai:latest \ vllm serve "${MODEL}" \ --port "${PORT}" \ @@ -50,7 +49,7 @@ exec podman run -d --name "${CONTAINER_NAME}" \ --gpu-memory-utilization "${GPU_MEM_UTIL}" \ --max-num-seqs "${MAX_NUM_SEQS}" \ --max-model-len "${MAX_MODEL_LEN}" \ - --speculative-method dflash \ - --speculative-model "${DRAFT_MODEL}" \ - --num-speculative-tokens "${NUM_SPEC_TOKENS}" \ + --spec-method dflash \ + --spec-model "${DRAFT_MODEL}" \ + --spec-tokens "${NUM_SPEC_TOKENS}" \ --trust-remote-code diff --git a/scripts/vllm-qwen36.service b/scripts/vllm-qwen36.service index fe30dec..6b4c9b5 100644 --- a/scripts/vllm-qwen36.service +++ b/scripts/vllm-qwen36.service @@ -7,6 +7,7 @@ Wants=network-online.target Type=simple User=dank Group=dank +WorkingDirectory=/home/dank/Projects/vllmScripts Environment="PORT=8000" Environment="GPU_MEM_UTIL=0.95" Environment="MAX_NUM_SEQS=4" @@ -17,16 +18,7 @@ ExecStart=/home/dank/Projects/vllmScripts/scripts/serve-vllm-qwen36-dflash.sh ExecStop=/usr/bin/podman stop vllm-qwen36-dflash ExecStopPost=/usr/bin/podman rm -f vllm-qwen36-dflash Restart=on-failure -RestartSec=10 -TimeoutStopSec=60 - -# Ensure GPU devices are available -DevicePolicy=closed -DeviceAllow=/dev/nvidia0 rwm -DeviceAllow=/dev/nvidiactl rwm -DeviceAllow=/dev/nvidia-uvm rwm -DeviceAllow=/dev/nvidia-uvm-tools rwm -DeviceAllow=/dev/nvidia-modeset rwm +RestartSec=5 [Install] WantedBy=multi-user.target