Files

T

Frank 8212d4451a docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault

Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.

2026-09-03 16:17:14 -07:00

6.9 KiB

Raw Blame History

LLM Fleet — Knowledge Base

Documentation for the LLM inference fleet on two local machines (2026-08-30): FreeToken v0.1.2 on Dankserver (frank-small, Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX llama.cpp on Dankclaw (frank-large, kingjones777 Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template.

2026-08-22: the former vllmScripts repository is merged into the serverAppConfigs repository. This knowledge base moved from the old repo's serverAppConfigs-docs/ root to serverAppConfigs-docs/LLM/. The fleet scripts and units now live in the repo's scripts/ directory.

Deployment Topology

graph LR
    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
        A[RTX 3090 24GB + 60GB RAM]
        B2[FreeToken v0.1.2
frank-small, port 7770]
        C[Huihui-Qwen3.6-35B-A3B-abliterated-FP8
block-fp8, 20% MoE residency, 453k KV / 4 streams]
        A --> B2 --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[gfx1151 iGPU 64 GiB unified memory]
        E[kingjones30/ROCmFPX build
frank-large, port 7770]
        F[kingjones777 Qwen3.8-Flash-Next
ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
        E2 --> E --> F
    end

    CLI[LLM Clients / API] -->|HTTP :7770| B2
    CLI -->|HTTP :7770| E

Servers

Server	GPU	VRAM / Memory	OS
Dankserver System Specs	NVIDIA RTX 3090	24 GB VRAM, 62 GB RAM	Ubuntu 26.04
Dankclaw System Specs	AMD Radeon 8060S (Strix Halo)	gfx1151 unified memory	Ubuntu 26.04

Active Deployment

freetoken-frank-small — the active frank-small runbook: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B- abliterated-FP8 on Dankserver (swapped 2026-08-30; same format/pools)
freetoken-benchmark-sweep — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
deployment — deploy and update process for both servers
frank-redo-2026-08-17 — the 2026-08-17 topology change
kingjones-strix-rocmfp4 — ACTIVE frank-large runbook: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned kingjones30/ROCmFPX fork (swapped 2026-08-30)
agentionai-fast-imatrix — retired frank-large runbook: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38 (2026-08-29 → 2026-08-30); rollback binary + payload retained
ubatch-benchmark-large — active-model llama-bench ubatch sweep plan and results
kingjones-q8agent-rocmfpx — the retired Q8_0-AGENT investigation and set_rows failure
eemin-rocmi4-mtp — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired)
dealign-ornith-crack — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-small, retired 2026-09-02)
ikllama-ornith-dankserver — the retired ik_llama.cpp runtime for frank-small (2026-08-26): q8_0 + Hadamard KV cache, split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
kingjones-strix-mtp — the former Q6 AGENT runbook (restored 2026-09-01 after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
kairic-edge — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22)
Dankserver System Specs — the RTX 3090 box (frank-small)
Dankclaw System Specs — the Strix Halo gfx1151 box (frank-large, served on the pinned kingjones30/ROCmFPX llama.cpp fork since 2026-08-30)
frank-npu — ACTIVE FLM NPU sight+sound runbook on Dankclaw: Gemma4 E4B vision+audio pack on the XDNA2 NPU (FLM v1.0.1, port 7771, whole-registry serving, per-request model routing), incl. the IOMMU/SVA root cause and the iommu=pt fix behind its first-deploy crash

CIRU Reference (Dankclaw)

ciru-activefpx-promptforge — the RETIRED v3 model, ROCm 7.15 toolchain, and build history (on Dankclaw; the slot now serves the kingjones ROCmFP4 model, see kingjones-strix-mtp)
kairic-edge — Kairic Edge IU4 runtime: build runbook, binary identity, and experiment summary (retired 2026-08-22)

DFlash2 Reference (archived)

dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)

vLLM Reference (historical)

VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
VLLM Index — quick reference with commands, endpoints, and parameter table
Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes and sizing (this architecture family is what frank-small serves today in the Huihui abliterated FP8 variant — see freetoken-frank-small; the Ornith-1.5-35B-A3B CRACK checkpoint that preceded it is archived at dealign-ornith-crack)

Memory Management

RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration

Retired Topology (2026-08-17 redo)

Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
frank-large-benchmark — backend/spec benchmark behind the old head
dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
Quantization by Hardware Architecture — which quant format fits which GPU/CPU

Pipeline Parallelism (archive)

Pipeline Parallelism/index — index of multi-node inference research. The PP pair is retired. These pages are research history.

Runtime Research (llama.cpp family)

Llama Cpp Runtimes — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
Llama Cpp Code Diff — code-level quantization diff of the three runtimes + server recommendations
SGLang — the structured-generation serving engine; when it fits Dankserver (3090) vs Dankclaw (Strix Halo)

Project knowledge base — the Podman app stacks on the same two hosts. OpenWebUI (Dankclaw) consumes the on-host frank-large endpoint at :7770. The Gitea instance on Dankclaw hosts this repository.
Deployment Flow — the service-scoped git-based deploy with inference isolation

6.9 KiB Raw Blame History Unescape Escape