serverAppConfigs/docs/LLM/index.md

LLM Fleet — Knowledge Base

Documentation for the LLM inference fleet on two local machines (2026-09-01): upstream llama.cpp on Dankserver (frank-default-small, Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw (frank-default-large, kingjones Qwen3.8-27B ROCmFPX Q8 AGENT). Both services run the Qwen-Sharp chat template.

2026-08-22: the former vllmScripts repository is merged into the serverAppConfigs repository. This knowledge base moved from the old repo's docs/ root to docs/LLM/. The fleet scripts and units now live in the repo's scripts/ directory.

Deployment Topology

graph LR
    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
        A[RTX 3090 24GB + 62GB RAM]
        B[upstream llama.cpp b10121
frank-default-small, port 7770]
        C[Ornith-1.5-35B-A3B-CRACK Q6_K
ngram-mod, selected experts offloaded via override, 491520 ctx]
        A --> B --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        D[gfx1151 iGPU 64 GiB unified memory]
        E[CIRU ROCmFPX llama.cpp PromptForge v3
frank-default-large, port 7770]
        F[Qwen3.8-27B ROCmFPX Q8_0 AGENT
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
        D --> E --> F
    end

    I[LLM Clients / API] -->|HTTP :7770| B
    I -->|HTTP :7770| E

Servers

Server	GPU	VRAM / Memory	OS
Dankserver System Specs	NVIDIA RTX 3090	24 GB VRAM, 62 GB RAM	Ubuntu 26.04
Dankclaw System Specs	AMD Radeon 8060S (Strix Halo)	gfx1151 unified memory	Ubuntu 26.04

Active Deployment

• deployment — deploy and update process for both servers
• frank-redo-2026-08-17 — the 2026-08-17 topology change
• kingjones Qwen3.8-27B ROCmFPX Q8 AGENT — active frank-default-large model on Dankclaw; standalone MTP and Turbo4 V cache
• kingjones-q8agent-rocmfpx — the Q8_0-AGENT runbook: serving profile, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT
• eemin-rocmi4-mtp — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired)
• dealign-ornith-crack — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24)
• kingjones-strix-mtp — the retired kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25 by the ROCmI4 embedded-MTP model)
• dealign-ornith-crack — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24)
• ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
• kairic-edge — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22)
• Dankserver System Specs — the upstream llama.cpp box (frank-default-small)
• Dankclaw System Specs — the CIRU ROCmFPX box (frank-default-large)

CIRU Reference (Dankclaw)

• ciru-activefpx-promptforge — the RETIRED v3 model, ROCm 7.15 toolchain, and build history (on Dankclaw; the slot now serves the kingjones ROCmFP4 model, see kingjones-strix-mtp)
• kairic-edge — Kairic Edge IU4 runtime: build runbook, binary identity, and experiment summary (retired 2026-08-22)

DFlash2 Reference (archived)

• dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)

vLLM Reference (historical)

• VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
• VLLM Index — quick reference with commands, endpoints, and parameter table
• Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes and sizing (the historical family; frank-default-small now serves Ornith-1.5-35B-A3B, see dealign-ornith-crack)

Memory Management

• RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
• Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
• Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration

Retired Topology (2026-08-17 redo)

• Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
• qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
• FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
• frank-default-large-benchmark — backend/spec benchmark behind the old head
• dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
• DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
• Quantization by Hardware Architecture — which quant format fits which GPU/CPU

Pipeline Parallelism (archive)

• Pipeline Parallelism/index — index of multi-node inference research. The PP pair is retired. These pages are research history.

Related

• Project knowledge base — the Podman app stacks on the same two hosts. OpenWebUI consumes the :7770 endpoint on Dankserver. The Gitea instance on Dankclaw hosts this repository.
• Deployment Flow — the unified git-based deploy for both workloads