docs: fact-check knowledge base against live machines and vLLM source
Correct stale and incorrect facts after exhaustive verification: - Dankclaw: NUMA node 0 = 31213 MB; disk 661G/1.1T/38%; vLLM 0.26.0 in /home/dank/vllm-rocm-venv (not vllm-env); clarify 96 GiB GPU-visible is a driver overcommit ceiling over 30 GiB physical (confirmed via sysfs mem_info_vram_total=103079215104) - Dankserver: disk 729G/140G free/84%; /mnt/entropy 910G free; active service is vLLM EngineCore (22 GB), not llama-server; CPU model Ryzen 7 7700X - index.md: Dankclaw 30GiB phys / 96GiB GPU-visible (was '61 GB unified'); add deployment topology mermaid diagram - vLLM defaults (from source): gpu_memory_utilization=0.92, max_num_seqs=128, swap_space removed (now --kv-offloading-size) across Overview/Index/RAM/Hybrid - Quantization: FP8 not native on Ampere (RTX 3090) or RDNA 3.5 (gfx1151); add per-model hardware quantization notes; AWQ/GPTQ limited on ROCm - Speculative decoding: add DFlash, DSpark, MTP, EAGLE3 to feature list - Confirm Qwen3.6-27B and Qwen3.6-35B-A3B specs are correct (HF-verified) - Production note: Dankclaw primary llama.cpp service runs ROCmFPX Ling-3.0-flash RocmFP4 MTP, not vLLM Qwen3.6-27B
This commit is contained in:
Frank
@@ -14,7 +14,7 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|
||||
| Cores | 16 (1 socket, 1 thread per core) |
|
||||
| Max MHz | 5187.50 |
|
||||
| Min MHz | 625.00 |
|
||||
| NUMA Nodes | 1 (node 0, 62909 MB) |
|
||||
| NUMA Nodes | 1 (node 0, 31213 MB) |
|
||||
|
||||
### GPU (Integrated)
|
||||
|
||||
@@ -23,7 +23,7 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|
||||
| Model | AMD Radeon 8060S (Strix Halo) |
|
||||
| Device ID | 0x1586 |
|
||||
| Architecture | gfx1151 (Strix Halo) |
|
||||
| GPU-Visible Memory | 96 GiB (ROCm addressable range) |
|
||||
| GPU-Visible Memory | 96 GiB (driver VRAM ceiling, overcommitted) |
|
||||
| Physical RAM | 30 GiB (shared CPU + GPU) |
|
||||
|
||||
### ROCm
|
||||
@@ -35,14 +35,14 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|
||||
| ROCm Version | 7.2.4 (from AMD repos) |
|
||||
| Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
|
||||
| GPU Arch | gfx1151 (Strix Halo) |
|
||||
| GPU Global Memory | 96 GiB (overcommitted, physical is 30 GiB) |
|
||||
| GPU Global Memory | 96 GiB (driver-reported ceiling; physical is 30 GiB) |
|
||||
|
||||
### Memory
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| System RAM | 30 GiB (shared CPU + GPU) |
|
||||
| GPU-Visible (ROCm) | 96 GiB (overcommitted addressable range) |
|
||||
| GPU-Visible (ROCm) | 96 GiB (driver ceiling; ROCm pool 1 is 31962252 KB) |
|
||||
| Swap | 8.0 GiB |
|
||||
| /dev/shm | ~15 GiB (half of RAM) |
|
||||
|
||||
@@ -50,7 +50,7 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|
||||
|
||||
| Filesystem | Size | Used | Avail | Use% |
|
||||
|------------|------|------|-------|------|
|
||||
| / (nvme0n1p2) | 1.9T | 391G | 1.4T | 22% |
|
||||
| / (nvme0n1p2) | 1.9T | 661G | 1.1T | 38% |
|
||||
|
||||
### OS and Software
|
||||
|
||||
@@ -59,18 +59,20 @@ Dankclaw is an AMD APU server for vLLM inference. The host runs Ubuntu 26.04 wit
|
||||
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
|
||||
| Kernel | 7.0.0-29-generic |
|
||||
| Python | 3.14.4 (system), 3.12.13 (vLLM venv) |
|
||||
| vLLM | 0.26.0+rocm723 |
|
||||
| vLLM | 0.26.0 (in `/home/dank/vllm-rocm-venv`, Python 3.12) |
|
||||
| uv | 0.12.2 |
|
||||
|
||||
## VLLM Relevance
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- ROCm reports 96 GiB GPU memory (overcommitted). Physical RAM is 30 GiB. vLLM sees 96 GiB and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
|
||||
- Only 8 GiB swap. Set `--swap-space` conservatively (4-8 GiB).
|
||||
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). Physical RAM is 30 GiB. The driver reports a 96 GiB VRAM ceiling (`sysfs mem_info_vram_total`), but the machine has only 30.5 GB of physical RAM. vLLM sees the large number and will try to use it. Set `--gpu-memory-utilization` conservatively (0.80-0.85) to avoid OOM when the OS needs memory.
|
||||
- Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
|
||||
- Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag.
|
||||
- The active llama.cpp service on Dankclaw runs on the ROCmFPX runtime (`/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server`) with the Ling-3.0-flash ROCmFP4 MTP model, not regular llama.cpp.
|
||||
- ROCm 7.2.4 from AMD repos. vLLM ROCm support requires ROCm 6.3+.
|
||||
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. Use a Python 3.12 venv.
|
||||
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
|
||||
- 1.4 TB free disk space is ample for model storage.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
- `libmpi_cxx.so.40` is missing from the system. Install `openmpi-bin` or provide a workaround for torch MPI dependency.
|
||||
|
||||
## See Also
|
||||
|
||||
@@ -12,6 +12,7 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|----------|-------|
|
||||
| Architecture | x86_64 |
|
||||
| Vendor | AuthenticAMD |
|
||||
| Model | AMD Ryzen 7 7700X |
|
||||
| Cores | 8 (1 socket) |
|
||||
| NUMA Nodes | 1 (node 0, 63503 MB) |
|
||||
|
||||
@@ -37,8 +38,8 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
| Filesystem | Size | Used | Avail | Use% |
|
||||
|------------|------|------|-------|------|
|
||||
| / (nvme0n1p2) | 915G | 849G | 20G | 98% |
|
||||
| /mnt/entropy (dm-0) | 91T | 86T | 977G | 99% |
|
||||
| / (nvme0n1p2) | 915G | 729G | 140G | 84% |
|
||||
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
|
||||
|
||||
### OS and Software
|
||||
|
||||
@@ -50,14 +51,14 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
A llama-server process occupies approximately 20 GB of the RTX 3090 VRAM.
|
||||
A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies approximately 22 GB of the RTX 3090 VRAM (`nvidia-smi` reports `VLLM::EngineCore` at 22164 MiB).
|
||||
|
||||
## VLLM Relevance
|
||||
|
||||
- 24 GB VRAM limits model size. Use FP8 quantization for large models.
|
||||
- 62 GB system RAM provides ample swap space for KV cache offloading.
|
||||
- 127 GB swap supports large `--swap-space` values for CPU KV cache paging.
|
||||
- Root disk has only 20 GB free. Store models on /mnt/entropy.
|
||||
- Root disk has approximately 140 GB free. Store models on /mnt/entropy.
|
||||
- CUDA 12.0 and Python 3.12.3 are compatible with vLLM.
|
||||
|
||||
## See Also
|
||||
|
||||
@@ -41,9 +41,9 @@ Set `--gpu-memory-utilization` based on your GPU:
|
||||
|
||||
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
|
||||
|
||||
### Swap Space
|
||||
### Swap Space / KV Offload
|
||||
|
||||
Set `--swap-space` based on system RAM:
|
||||
Recent vLLM releases removed the `--swap-space` engine argument. Use `--kv-offloading-size` to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||||
|
||||
| System RAM | Recommended Swap |
|
||||
|------------|------------------|
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
|
||||
VLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. VLLM does not offload model weights to CPU. Instead, VLLM moves KV cache data between GPU and CPU.
|
||||
|
||||
> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.
|
||||
|
||||
## How CPU Offloading Works
|
||||
|
||||
VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.
|
||||
@@ -26,19 +28,19 @@ This parameter controls the fraction of GPU memory used for KV cache.
|
||||
|-------|-------------|
|
||||
| 0.7 | Uses 70% of GPU memory for KV cache |
|
||||
| 0.8 | Uses 80% of GPU memory for KV cache |
|
||||
| 0.9 | Uses 90% of GPU memory for KV cache (default) |
|
||||
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
|
||||
| 0.95 | Uses 95% of GPU memory for KV cache |
|
||||
|
||||
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
|
||||
|
||||
### swap_space
|
||||
### swap_space (removed) / kv-offloading-size (current)
|
||||
|
||||
This parameter sets the CPU swap space in GiB.
|
||||
The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| 0 | No CPU swap space |
|
||||
| 4 | 4 GiB swap space (default) |
|
||||
| 4 | 4 GiB swap space (historical default) |
|
||||
| 8 | 8 GiB swap space |
|
||||
| 16 | 16 GiB swap space |
|
||||
| 32 | 32 GiB swap space |
|
||||
@@ -78,8 +80,7 @@ This parameter sets the maximum number of sequences per batch.
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| 128 | Default for online serving |
|
||||
| 256 | Default for offline inference |
|
||||
| 512 | Large batch |
|
||||
| 256 | Large batch |
|
||||
|
||||
### kv_cache_dtype
|
||||
|
||||
|
||||
@@ -47,6 +47,8 @@ The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> **Production note:** Dankclaw's primary llama.cpp service now runs the ROCmFPX runtime with the Ling-3.0-flash ROCmFP4 MTP model (see `scripts/llama-server.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
|
||||
|
||||
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
|
||||
|
||||
```
|
||||
@@ -55,7 +57,7 @@ source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
Installed: vLLM 0.26.0+rocm723 in Python 3.12.13 venv at `/home/dank/vllm-env`.
|
||||
Installed: vLLM 0.26.0 in Python 3.12.13 venv at `/home/dank/vllm-rocm-venv`.
|
||||
|
||||
### MPI Dependency
|
||||
|
||||
@@ -198,7 +200,7 @@ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
## ROCm Notes
|
||||
|
||||
- Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
|
||||
- FP8 may use software emulation on RDNA architecture. Verify with a BF16 test first
|
||||
- FP8 uses **software emulation** on RDNA architecture (gfx1151 has no native FP8 tensor cores). This gives memory savings, not full compute speedup. Compare against a BF16 test first
|
||||
- Attention backends: try `ROCM_AITER_UNIFIED_ATTN` if the default backend underperforms
|
||||
- Context length on ROCm may be limited for some models. Start with 131072 and increase gradually
|
||||
|
||||
|
||||
@@ -49,6 +49,12 @@ Install vLLM. Version 0.19.0 or later is required for Qwen3.6:
|
||||
uv pip install vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
## Hardware Quantization Notes (RTX 3090 / Ampere)
|
||||
|
||||
The RTX 3090 is an Ampere GPU (compute capability 8.6). It has **no native FP8 tensor cores**. FP8 W8A8 compute runs through a fallback/dequantization path, so the FP8 model gives memory savings but not full compute speedup. vLLM loads these models and runs them; verify quality against a BF16 run. `--kv-cache-dtype fp8` is still valid on CUDA 12 for an FP8 KV cache.
|
||||
|
||||
> Do not run on production without a BF16 comparison. The FP8 precision path may differ from native-FP8 hardware.
|
||||
|
||||
## vLLM Serve Commands
|
||||
|
||||
### Text-Only (Recommended for Memory Savings)
|
||||
@@ -91,7 +97,7 @@ vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
|-----------|-------|--------|
|
||||
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
|
||||
| --gpu-memory-utilization 0.75 | 75% of VRAM | Leaves room for all MoE weights |
|
||||
| --swap-space 32 | 32 GB CPU swap | Dankserver has 62 GB RAM |
|
||||
| --swap-space 32 (removed) | 32 GB CPU overflow | Modern vLLM removed swap-space; use --kv-offloading-size |
|
||||
| --max-model-len 131072 | 128K context | Full 262K context is too large for 24 GB |
|
||||
| --max-num-batched-tokens 2048 | Conservative batch | Prevents OOM with MoE |
|
||||
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode output |
|
||||
@@ -106,10 +112,10 @@ export HF_HOME=/mnt/entropy/huggingface
|
||||
|
||||
### Active Service Conflict
|
||||
|
||||
A llama-server process occupies approximately 20 GB of VRAM. Stop it before starting vLLM:
|
||||
A vLLM engine core (Podman container `vllm-qwen36-dflash`) occupies roughly 22 GB of the 24 GB VRAM. Stop it before starting a competing service:
|
||||
|
||||
```
|
||||
pkill llama-server
|
||||
podman stop vllm-qwen36-dflash
|
||||
nvidia-smi # verify GPU is free
|
||||
```
|
||||
|
||||
|
||||
@@ -84,13 +84,12 @@ Since memory is unified, set these parameters carefully:
|
||||
```
|
||||
vllm serve model \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-num-batched-tokens 4096
|
||||
```
|
||||
|
||||
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
|
||||
- `swap_space`: Keep moderate (8-16 GB) for overflow
|
||||
- `max_model_len`: Match actual context needs
|
||||
- The old `--swap-space` argument was removed from current vLLM. Use `--kv-offloading-size` for CPU KV offload
|
||||
|
||||
### NUMA Binding
|
||||
|
||||
@@ -197,8 +196,10 @@ Some models on ROCm have context length limited to 4096 tokens. Check model docu
|
||||
FP8 quantization depends on GPU architecture:
|
||||
|
||||
- MI300 series supports native FP8
|
||||
- Radeon RX 7900 supports FP8 via software
|
||||
- Strix Halo may support FP8 via software emulation
|
||||
- Radeon RX 7900 supports FP8 via software emulation
|
||||
- Strix Halo (gfx1151) uses **software emulation** for FP8 (no native FP8 tensor cores); expect memory savings, not full compute speedup
|
||||
|
||||
For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native ROCm FP4 kernels.
|
||||
|
||||
## Example Commands
|
||||
|
||||
|
||||
+6
-6
@@ -67,20 +67,20 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space
|
||||
| Parameter | Description | Typical Value |
|
||||
|-----------|-------------|---------------|
|
||||
| tensor_parallel_size | Number of GPUs for model sharding | 1-8 |
|
||||
| gpu_memory_utilization | Fraction of GPU memory for KV cache | 0.7-0.95 |
|
||||
| swap_space | CPU swap space in GiB | 4-32 |
|
||||
| gpu_memory_utilization | Fraction of GPU memory for KV cache (default 0.92) | 0.7-0.95 |
|
||||
| swap_space | Removed in current vLLM; use `--kv-offloading-size` | n/a |
|
||||
| max_model_len | Maximum context length | auto |
|
||||
| max_num_batched_tokens | Max tokens per batch | 2048-8192 |
|
||||
| max_num_seqs | Max sequences per batch | 128-256 |
|
||||
| max_num_seqs | Max sequences per batch | 128-512 |
|
||||
|
||||
### Quantization Methods
|
||||
|
||||
| Method | GPU Support | Description |
|
||||
|--------|-------------|-------------|
|
||||
| FP8 | Hopper, Ada, Ampere | Native FP8 inference |
|
||||
| FP8 | Hopper, Ada (native); Ampere W8A16 via Marlin; RDNA via emulation | FP8 E4M3 inference |
|
||||
| INT8 | Most GPUs | Weight-only INT8 |
|
||||
| GPTQ | Most GPUs | Post-training GPTQ |
|
||||
| AWQ | Turing+ | Activation-aware weight quantization |
|
||||
| GPTQ | Most GPUs (NVIDIA; limited on ROCm) | Post-training GPTQ |
|
||||
| AWQ | Turing+ (NVIDIA) | Activation-aware weight quantization |
|
||||
| GGUF | CPU, GPU | General GPU unification format |
|
||||
|
||||
## See Also
|
||||
|
||||
+11
-9
@@ -11,7 +11,7 @@ VLLM is fast with these features:
|
||||
- Chunked prefill and prefix caching
|
||||
- FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
|
||||
- Tensor, pipeline, data, expert, and context parallelism
|
||||
- Speculative decoding with n-gram, suffix, and EAGLE
|
||||
- Speculative decoding with n-gram, suffix, EAGLE/EAGLE3, DFlash, DSpark, and MTP
|
||||
- Automatic kernel generation with torch.compile
|
||||
- Disaggregated prefill, decode, and encode
|
||||
|
||||
@@ -130,11 +130,11 @@ curl http://localhost:8000/v1/chat/completions \
|
||||
|----------|---------|-------------|
|
||||
| `tensor_parallel_size` | 1 | Number of GPUs for tensor parallelism |
|
||||
| `pipeline_parallel_size` | 1 | Number of GPUs for pipeline parallelism |
|
||||
| `gpu_memory_utilization` | 0.9 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | 4 | CPU swap space in GiB |
|
||||
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | removed | Removed in current vLLM; use `--kv-offloading-size` |
|
||||
| `enforce_eager` | false | Use eager mode (no CUDA graphs) |
|
||||
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
|
||||
| `max_num_seqs` | 256 | Max sequences per batch |
|
||||
| `max_num_seqs` | 128 | Max sequences per batch |
|
||||
| `block_size` | 16 | Token block size |
|
||||
| `num_gpu_blocks_override` | null | Override number of GPU blocks |
|
||||
| `max_model_len_per_seq` | null | Max length per sequence |
|
||||
@@ -157,8 +157,8 @@ curl http://localhost:8000/v1/chat/completions \
|
||||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `gpu_memory_utilization` | 0.9 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | 4 | CPU swap space in GiB |
|
||||
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | removed | Removed in current vLLM; use `--kv-offloading-size` |
|
||||
| `kv_cache_dtype` | auto | KV cache data type (auto, fp8, fp8_e5m2) |
|
||||
| `max_model_len` | auto | Maximum context length |
|
||||
| `num_gpu_blocks` | auto | Number of GPU blocks |
|
||||
@@ -204,18 +204,20 @@ VLLM supports over 200 model architectures including:
|
||||
|
||||
VLLM supports these quantization methods:
|
||||
|
||||
- FP8 for Hopper and Ada GPUs
|
||||
- FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
|
||||
- MXFP8/MXFP4 for Hopper GPUs
|
||||
- NVFP4 for Hopper GPUs
|
||||
- NVFP4 for Hopper and Blackwell GPUs
|
||||
- INT8 for most GPUs
|
||||
- INT4 for most GPUs
|
||||
- GPTQ for most GPUs
|
||||
- AWQ for Turing and newer GPUs
|
||||
- AWQ for Turing and newer GPUs (NVIDIA; native support varies on ROCm)
|
||||
- GGUF for CPU and GPU
|
||||
- Compressed-tensors for custom quantization
|
||||
- ModelOpt for NVIDIA Model Optimizer
|
||||
- TorchAO for PyTorch AO
|
||||
|
||||
AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not native tensor cores. AWQ and GPTQ support in vLLM on AMD ROCm is limited; use GGUF or AMD Quark.
|
||||
|
||||
## Parallelism
|
||||
|
||||
VLLM supports multiple parallelism strategies:
|
||||
|
||||
+26
-4
@@ -2,12 +2,34 @@
|
||||
|
||||
Documentation for running vLLM inference on two local machines.
|
||||
|
||||
## Deployment Topology
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B[vLLM 0.26.0 in Podman\nvllm-qwen36-dflash]
|
||||
C[Qwen3.6-35B-A3B-FP8\n+ DFlash draft, port 8000]
|
||||
A --> B --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
|
||||
E[ROCmFPX llama.cpp\n/ROCmFPX/build-strix-rocmfp4]
|
||||
F[Ling-3.0-flash ROCmFP4-MTP\ndraft-mtp, port 8080]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
G[LLM Clients / API] -->|HTTP :8000| B
|
||||
G -->|HTTP :8080| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
|
||||
| Server | GPU | VRAM / Memory | OS |
|
||||
| --------------------------------------- | ----------------------------- | --------------------- | ------------ |
|
||||
| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 24.04 |
|
||||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | 61 GB unified | Ubuntu 26.04 |
|
||||
| Server | GPU | VRAM / Memory | OS |
|
||||
| --------------------------------------- | ----------------------------- | ----------------------------------- | ------------ |
|
||||
| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 24.04 |
|
||||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | 30 GiB physical / 96 GiB GPU-visible | Ubuntu 26.04 |
|
||||
|
||||
## vLLM Reference
|
||||
|
||||
|
||||
Reference in New Issue
Block a user