serverAppConfigs/docs/index.md

vLLM Scripts — Knowledge Base

Documentation for the LLM inference fleet on two local machines (2026-08-17 redo): llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.

Deployment Topology

graph LR
    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
        A[RTX 3090 24GB + 62GB RAM]
        B[llama.cpp FastMTP
frank-default-large, port 7770]
        C[Qwen3.8-27B-Aggressive IQ4_XS
FastMTP 32K draft + ngram-mod, 128K ctx]
        A --> B --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        D[gfx1151 iGPU unified memory]
        E[vLLM ROCm
frank-default-small, port 52625]
        F[Avesed Qwen3.6-35B-A3B INT8 W8A8
full 262144 ctx, ngram spec, 4 seqs]
        D --> E --> F
    end

    I[LLM Clients / API] -->|HTTP :7770| B
    I -->|HTTP :52625| E

Servers

Server	GPU	VRAM / Memory	OS
Dankserver System Specs	NVIDIA RTX 3090	24 GB VRAM, 62 GB RAM	Ubuntu 24.04
Dankclaw System Specs	AMD Radeon 8060S (Strix Halo)	gfx1151 unified memory	Ubuntu 26.04

Active Deployment

• deployment — deploy and update process for both servers
• frank-redo-2026-08-17 — the full redo: topology change, model choices, context math, and the two install runbooks
• Dankserver — the llama.cpp FastMTP box (frank-default-large)
• Dankclaw — the vLLM ROCm box (frank-default-small)

vLLM Reference (Dankclaw)

• VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
• VLLM Index — quick reference with commands, endpoints, and parameter table
• Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes and sizing (the model family served by frank-default-small)

Memory Management

• RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
• Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
• Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration

Retired Topology (2026-08-17 redo)

• Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
• qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
• FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
• frank-default-large-benchmark — backend/spec benchmark behind the old head
• dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
• DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
• Quantization by Hardware Architecture — which quant format fits which GPU/CPU

Pipeline Parallelism (archive)

• Pipeline Parallelism — index of multi-node inference research. The PP pair is retired; these pages are research history.

Sibling projects

• serverAppConfigs — the Podman app stack on the same two hosts. Its OpenWebUI consumes the :7770 endpoint on Dankserver; its Gitea instance hosts this repository.