Frank
4144f0f0e8
ik_llama limits speculative decoding to a single server slot (-np 1) and hard-exits at load if a spec stage is configured with --parallel > 1: "Speculative decoding is currently limited to a single server slot (-np 1)." This service keeps PARALLEL=3 (one slot per user), so set SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B active of 35B), so throughput is fine without it. MTP stays disabled. KB and tests updated.
125 lines
5.9 KiB
Markdown
125 lines
5.9 KiB
Markdown
# LLM Fleet — Knowledge Base
|
|
|
|
Documentation for the LLM inference fleet on two local machines
|
|
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
|
|
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
|
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
|
services run the Qwen-Sharp chat template.
|
|
|
|
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
|
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
|
> `docs/` root to `docs/LLM/`. The fleet scripts and units now live in the
|
|
> repo's `scripts/` directory.
|
|
|
|
## Deployment Topology
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
|
A[RTX 3090 24GB + 62GB RAM]
|
|
B2[ik_llama.cpp build b166e269
|
|
frank-default-small, port 7770]
|
|
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
|
no speculation, q8_0 + Hadamard cache, 491520 ctx]
|
|
A --> B2 --> C
|
|
end
|
|
|
|
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
|
E2[gfx1151 iGPU 64 GiB unified memory]
|
|
E[charlie12345/ROCmFPX build-kairic (v81)
|
|
frank-default-large, port 7770]
|
|
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
|
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
|
E2 --> E --> F
|
|
end
|
|
|
|
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
|
CLI -->|HTTP :7770| E
|
|
```
|
|
|
|
## Servers
|
|
|
|
| Server | GPU | VRAM / Memory | OS |
|
|
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
|
| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 26.04 |
|
|
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
|
|
|
## Active Deployment
|
|
|
|
- [[deployment]] — deploy and update process for both servers
|
|
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
|
- [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP
|
|
- [[kingjones-strix-mtp]] — the active Q6 AGENT runbook: serving profile, standalone MTP, reasoning + preserve_thinking, 114 GiB TTM/GTT
|
|
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
|
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
|
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
|
on Dankclaw (retired)
|
|
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
|
Dankserver (frank-default-small, active 2026-08-24)
|
|
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
|
|
frank-default-small (2026-08-26): q8_0 + Hadamard KV cache,
|
|
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
|
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
|
model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
|
|
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
|
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
|
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
|
(deployed and reverted 2026-08-22)
|
|
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
|
|
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
|
|
|
## CIRU Reference (Dankclaw)
|
|
|
|
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
|
and build history (on Dankclaw; the slot now serves the kingjones
|
|
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
|
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
|
identity, and experiment summary (retired 2026-08-22)
|
|
|
|
## DFlash2 Reference (archived)
|
|
|
|
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
|
(2026-08-21)
|
|
|
|
## vLLM Reference (historical)
|
|
|
|
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
|
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
|
notes and sizing (the historical family; frank-default-small now serves
|
|
Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]])
|
|
|
|
## Memory Management
|
|
|
|
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
|
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
|
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
|
|
|
## Retired Topology (2026-08-17 redo)
|
|
|
|
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
|
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
|
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
|
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
|
|
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
|
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
|
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
|
|
|
## Pipeline Parallelism (archive)
|
|
|
|
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
|
pair is retired. These pages are research history.
|
|
|
|
## Runtime Research (llama.cpp family)
|
|
|
|
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
|
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
|
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (3090) vs Dankclaw (Strix Halo)
|
|
|
|
## Related
|
|
|
|
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
|
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
|
|
Gitea instance on Dankclaw hosts this repository.
|
|
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
|
git-based deploy with inference isolation |