docs(vault): one fact, one owner - de-stale the technique and era pages (C1-C6) #24
@@ -411,7 +411,7 @@ measurement that would settle it.
|
||||
|
||||
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
|
||||
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
|
||||
- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
|
||||
- The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by [[RAM CPU Offloading in VLLM]].
|
||||
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
|
||||
- Use `rejection_sample_method` default `strict` for lossless output.
|
||||
|
||||
|
||||
@@ -56,7 +56,10 @@ Lower values leave more GPU memory for KV cache. This reduces swap frequency.
|
||||
|
||||
### Swap Space / KV Offload
|
||||
|
||||
Recent vLLM releases removed the `--swap-space` engine argument — `CacheConfig` on `main` has no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||||
Current vLLM has no `--swap-space` engine argument, and the CPU KV-offload knob pair
|
||||
is owned by [[RAM CPU Offloading in VLLM]] (that page carries the upstream
|
||||
`CacheConfig` verification). In Podman, mount the host cache read-write for KV offload
|
||||
writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||||
|
||||
| System RAM | Recommended Swap |
|
||||
|------------|------------------|
|
||||
|
||||
@@ -103,7 +103,7 @@ vllm serve model \
|
||||
|
||||
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
|
||||
- `max_model_len`: Match actual context needs
|
||||
- The old `--swap-space` argument was removed from current vLLM — `CacheConfig` on `main` carries no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) for CPU KV offload
|
||||
- `--swap-space` is retired from current vLLM; the CPU KV-offload knob pair, with the upstream `CacheConfig` verification, is owned by [[RAM CPU Offloading in VLLM]]
|
||||
|
||||
### NUMA Binding
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8
|
||||
|
||||
# With GPU memory control
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space 8
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --kv-offloading-size 8 --kv-offloading-backend native
|
||||
```
|
||||
|
||||
### Common API Endpoints
|
||||
@@ -79,7 +79,7 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space
|
||||
|-----------|-------------|---------------|
|
||||
| tensor_parallel_size | Number of GPUs for model sharding | 1-8 |
|
||||
| gpu_memory_utilization | Fraction of GPU memory for KV cache (default 0.92) | 0.7-0.95 |
|
||||
| swap_space | Removed in current vLLM; use `--kv-offloading-size` | n/a |
|
||||
| swap_space | Retired; live knob pair in [[RAM CPU Offloading in VLLM]] | n/a |
|
||||
| max_model_len | Maximum context length | auto |
|
||||
| max_num_batched_tokens | Max tokens per batch | 2048-8192 |
|
||||
| max_num_seqs | Max sequences per batch | 128-512 |
|
||||
|
||||
@@ -140,7 +140,7 @@ curl http://localhost:8000/v1/chat/completions \
|
||||
| `tensor_parallel_size` | 1 | Number of GPUs for tensor parallelism |
|
||||
| `pipeline_parallel_size` | 1 | Number of GPUs for pipeline parallelism |
|
||||
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | removed | Removed in current vLLM; use `--kv-offloading-size` |
|
||||
| `swap_space` | — | retired; the live CPU-KV-offload knob pair is in [[RAM CPU Offloading in VLLM]] |
|
||||
| `enforce_eager` | false | Use eager mode (no CUDA graphs) |
|
||||
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
|
||||
| `max_num_seqs` | 128 | Max sequences per batch |
|
||||
@@ -167,7 +167,7 @@ curl http://localhost:8000/v1/chat/completions \
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | removed | Removed in current vLLM; use `--kv-offloading-size` |
|
||||
| `swap_space` | — | retired; the live CPU-KV-offload knob pair is in [[RAM CPU Offloading in VLLM]] |
|
||||
| `kv_cache_dtype` | auto | KV cache data type (auto, fp8, fp8_e5m2) |
|
||||
| `max_model_len` | auto | Maximum context length |
|
||||
| `num_gpu_blocks` | auto | Number of GPU blocks |
|
||||
|
||||
@@ -14,6 +14,10 @@ timestamp: 2026-09-03
|
||||
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
|
||||
> The `frank-zealot` below is the 2026-08 service: llama.cpp on the RTX 3090
|
||||
> **while that card sat in Dankserver**. Today's [[frank-zealot]] is a different host
|
||||
> (Dankclaw), payload and runtime build, though the same card class - this crash
|
||||
> signature is not a live fault.
|
||||
|
||||
# Dankserver VRAM and RAM Growth Under Load
|
||||
|
||||
|
||||
Reference in New Issue
Block a user