dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): one page per endpoint, forever -- 45 era pages fold into three

Browse Source 
Every payload swap a live endpoint has ever run is now one row in that
endpoint's Eras table, with its rollback path in the same row. No per-attempt
page, no per-swap ledger, nowhere else.

- Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md
  (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork
  on the 3090). Each carries Current shape, Eras, Laws, wire contract, files.
- Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the
  pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the
  benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras),
  the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus
  frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing.
- Rewired: every wikilink and markdown-path reference to a folded page points
  at the endpoint page instead; the three Research see-also lists and the
  ubatch sweep's runbook link included. No anchored links existed in the set.
- Regenerated both index pages as pure locators -- one row, one clause, no
  numbers -- so the docs_completeness locator law holds for all 153 pages.

tests/docs_completeness.bats green: index links every page, no unresolved
links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored,
and git history holds every byte).
This commit is contained in:
Frank committed 2026-10-19 00:49:03 -07:00
1 parent af5ddfc54c
commit 219a097b39
88 files changed
+1211 -11915
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/index.md
+205 -536
View File
@@ -1,566 +1,235 @@
# Server App Configs
# serverAppConfigs — knowledge base
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
One fact, one owner. A page owns exactly one thing and everywhere else links
to it with a wikilink. Shape, payload, eras, laws and rollback for an inference
endpoint live on its endpoint page; `scripts/<service>.env` is the authority for
every number, and a page that disagrees with it is the bug. Payload swaps are
rows in an endpoint page's **Eras** table, never pages of their own. This index
is a locator: one row, one clause, no numbers.
This repository is the single source of truth for both hosts. It tracks the
container stacks and the LLM inference fleet. Edits flow through this repo,
then deploy to the targets with `scripts/deploy.sh` (one script per target
covers both workloads).
## Inference endpoints
> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
> into this repo with its full git history. Its knowledge base now lives at
> [serverAppConfigs-docs/LLM/index.md](LLM/index.md).
| Page | Owns |
|---|---|
| [[Frank TTS]] | frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in… |
| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
## Hosts
## Retired endpoints
| Host | IP | Role | OS |
|------|----|------|----|
| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |
These services no longer exist, so no live page owns them; rollback provenance
is inside each page. Every payload swap of a LIVE endpoint is an Eras row on
that endpoint's page, not a page.
## Stacks
| Page | Owns |
|---|---|
| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
| [[frank-medium-model-swap]] | How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining… |
| [[frank-medium-radiance-mxfp4]] | ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
- [[Dankserver Stack]] — RomM, Copyparty, Calibre, Calibre-Web, Caddy
- [[Dankclaw Stack]] — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy
## Dankclaw services (192.168.0.143)
## Architecture
| Page | Owns |
|---|---|
| [[ComfyUI]] | ComfyUI is a node-based AI image generation engine. |
| [[ComfyUI Qwen-Image Workflows]] | Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which… |
| [[Dankclaw Stack]] | Service inventory and routing overview for the Dankclaw AI, development, and observability host. |
| [[Frank TTS Engine]] | frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker,… |
| [[Gitea]] | Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service. |
| [[Gluetun]] | Gluetun is a lightweight VPN client container. |
| [[Hermes]] | Hermes is the self-improving AI agent from Nous Research. |
| [[Hermes-Dashboard]] | The Hermes web dashboard is a browser UI for managing the agent. |
| [[Hermes-WebUI]] | The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH… |
| [[Marreta]] | The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet… |
| [[Open Notebook]] | open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out —… |
| [[OpenWebUI]] | OpenWebUI is a self-hosted chat interface for LLMs. |
| [[Postgres]] | PostgreSQL is the database backend for Gitea. |
| [[SearXNG]] | SearXNG is a privacy-respecting metasearch engine. |
| [[SearXNG Providers]] | How Google and other upstream search engines get blocked behind the Gluetun |
| [[Seerr]] | Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived… |
| [[Swap Configuration]] | The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit.… |
| [[Valkey]] | Valkey is the cache and limiter store for SearXNG. |
| [[Voice Packs]] | How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material… |
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
- [[Storage Policy]] — bind mounts, data locations, ownership
- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
- [[Testing Policy]] — the test seams and how to run them
- [[E2E Suite]] — the Playwright suite that drives the live guard chain
- [[Container Network Namespaces]] — which local services are addressable from each Podman network namespace on Dankclaw
## Dankserver services (192.168.0.235)
## Dankserver Storage
| Page | Owns |
|---|---|
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
| [[Dankserver Stack]] | Dankserver (192.168.0.235) runs the media and library stack. |
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
| [[Jellyfin]] | Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and… |
| [[mariadb]] | MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and… |
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
| [[redis]] | Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup… |
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
| [[RomM-Scraping]] | This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping… |
| [[Tautulli]] | Tautulli is the statistics, monitoring, and notification companion for |
- [[Entropy Volume]] — command-line unlock and mount helper
## FrankBook-Pro (192.168.0.112)
## Dankserver Services
| Page | Owns |
|---|---|
| [[ARR-Stack]] | The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every… |
| [[lidarr]] | Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
| [[prowlarr]] | Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and… |
| [[qbittorrent]] | qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns,… |
| [[radarr]] | Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
| [[readarr]] | Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB… |
| [[sonarr]] | Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
- [[RomM]] — retro game manager
- [[RomM-Scraping]] — metadata providers, cover art, naming conventions
- [[Copyparty]] — file sharing and streaming
- [[Jellyfin]] — media server (movies, TV, music)
- [[Calibre]] — e-book library engine
- [[Calibre-Web]] — e-book web frontend
- [[Plex]] — media server on Dankserver (lscr.io/linuxserver/plex container via
quadlet; CPU decode only — no GPU passthrough)
- [[Tautulli]] — Plex statistics and notifications (v2.18.1-ls245, quadlet unit)
- [[mariadb]] — RomM database backend
- [[redis]] — RomM cache/queue backend
## Workstations
## FrankBook-Pro Services
| Page | Owns |
|---|---|
| [[Noctalia and Niri]] | The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette… |
The *arr stack and qBittorrent run on **FrankBook-Pro** (192.168.0.112, the
macOS mini) as quadlets inside its podman machine — not on Dankserver.
## Cross-host infrastructure
- [[ARR-Stack]] — automated media download overview
- [[sonarr]] — TV show management
- [[radarr]] — movie management
- [[lidarr]] — music management
- [[readarr]] — ebook/audiobook management
- [[prowlarr]] — indexer management
- [[qbittorrent]] — torrent download client
| Page | Owns |
|---|---|
| [[Alertmanager]] | Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to… |
| [[Alloy]] | Alloy is Grafana's log and telemetry shipper. |
| [[Anubis]] | Anubis is the bot-challenge proxy that fronts the guarded public vhosts on |
| [[Authentik]] | Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the |
| [[authentik-postgres]] | The PostgreSQL 16 database for Authentik — the homelab's SSO / |
| [[authentik-server]] | The Authentik server — the core, the admin API, and the embedded Caddy outpost. |
| [[authentik-worker]] | The Authentik worker — background tasks: event processing, provider |
| [[Caddy]] | Caddy is the single reverse proxy for both hosts. |
| [[FrankBook-Pro]] | FrankBook-Pro — the Mac mini running the *arr stack + qBittorrent inside a podman machine VM, gluetun-gated |
| [[Grafana]] | Grafana visualizes the metrics that Prometheus collects from every service. |
| [[Homepage]] | A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link… |
| [[Homepage Design Grilling]] | The decision log for the homepage design. |
| [[Homepage Status Probes]] | The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage… |
| [[LAN-DNS]] | The homelab gives every service a name under `franksdank.cloud`. |
| [[Loki]] | Loki is the log aggregation backend of the observability stack. |
| [[Node-Exporter]] | Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. |
| [[Observability]] | The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal… |
| [[Observability Provisioning]] | How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the… |
| [[Pi-hole]] | Pi-hole is the LAN DNS server for the homelab. |
| [[Prometheus]] | Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the… |
## Dankclaw Services
## Repo architecture and policy
- [[SearXNG]] — privacy metasearch (VPN-routed)
- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
- [[Gluetun]] — the VPN tunnel container
- [[Valkey]] — SearXNG cache and limiter store
- [[ComfyUI]] — node-based AI image generation (ROCm GPU, gfx1151 build)
- [[ComfyUI Qwen-Image Workflows]] — the two proven Qwen-Image 2.1 workflows + what to add next (LoRAs, Edit, ControlNet, Wan video)
- [[OpenWebUI]] — chat UI for local LLMs
- [[Hermes]] — the Nous Research agent
- [[Hermes-Dashboard]] — the Hermes web management UI (config, keys, sessions)
- [[Hermes-WebUI]] — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
- [[Gitea]] — Git hosting (migrated)
- [[Postgres]] — Gitea database backend
- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
- [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
- [[Frank TTS Engine]] — the arm A TTS engine (`qwentts.cpp` resident `tts-server`, 1.7 B talker) behind frank-tts on a private bridge; deployed on Dankclaw, CPU-only, publishes no port by design
- [[Frank TTS]] — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; deployed, and it is the harness `speech` role
- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate
| Page | Owns |
|---|---|
| [[Container Network Namespaces]] | Which local services are addressable from which Podman network namespace on |
| [[Deploy State Triage]] | Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout,… |
| [[Deployment Flow]] | Service-scoped Git deployment for the two homelab hosts. |
| [[E2E Suite]] | `tests/e2e/` drives the **live fleet** through a real browser: fresh contexts |
| [[Network Architecture]] | One Caddy fronts both hosts. |
| [[Quadlet Lifecycle]] | The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop… |
| [[Stale Signal Audit]] | The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers,… |
| [[Storage Policy]] | All persistent data uses **bind mounts**: a host directory mapped directly into the container. |
| [[Testing Policy]] | This repository tests its configuration at defined seams. |
## Infrastructure
## Operations and incidents
- [[LAN-DNS]] — name resolution, Pi-hole, router DNS hand-off
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
- [[Caddy]] — reverse proxy and TLS
- [[Anubis]] — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
- [[Authentik]] — the homelab's SSO / OIDC provider (forward_auth)
- [[authentik-postgres]] — Authentik database service
- [[authentik-server]] — Authentik server + Caddy outpost service
- [[authentik-worker]] — Authentik background worker service
- [[Homepage]] — password-protected service index
- [[Homepage Design Grilling]] — homepage decision log and open questions
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
- [[Alertmanager]] — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
- [[Node-Exporter]] — host metrics
- [[Loki]] — log storage
- [[Alloy]] — log shipping (replaces Promtail)
- [[FrankBook-Pro]] — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)
| Page | Owns |
|---|---|
| [[Container Host Reachability]] | The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or… |
| [[Fleet Audit and Posture]] | Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health… |
| [[Fleet Ops Playbook]] | Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows… |
| [[Payload and Disk Reclaim]] | Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index,… |
| [[Remote Job Runner]] | Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally,… |
| [[SSH and Remote Access]] | Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a… |
## Research
## Agent / harness
- [[Service Research Brief]] — consolidated subagent research behind these pages
- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three llama.cpp runtimes
- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit)
- [[Backblaze Integration]] — off-site backup plan (future)
- [[Jellyfin]] — media server research brief
- [[ARR-Stack]] — automated download stack research brief
- [[Local TTS and Voice Cloning]] — creating a local voice from samples, the 2026 cloning landscape, the full audio stack, and a recommended homelab deployment
- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices
| Page | Owns |
|---|---|
| [[Agent Docs and Edits]] | Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline… |
| [[Command Center Dotfiles]] | The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite… |
| [[OMP Harness]] | The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via… |
## Skill Distillation (2026-10-05)
## Research and methodology
The managed skill library distilled into the vault, one reference page per
skill-cluster. See [log.md](log.md) for the wave plan and results.
| Page | Owns |
|---|---|
| [[AI Job Search and Resume Tools]] | Self-hosted AI job-search and resume-formatting applications: the 2026 landscape, local-LLM fit, and a recommended… |
| [[Backblaze Integration]] | Off-site backup plan for the bind-mounted data on both hosts. |
| [[ComfyUI ROCm and Vulkan]] | Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in… |
| [[Lemonade and sd.cpp]] | Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. |
| [[Llama Cpp Code Diff]] | Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml… |
| [[Llama Cpp Runtimes]] | Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level… |
| [[Local TTS and Voice Cloning]] | How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the… |
| [[Most Popular Self-Hosted AI Apps]] | The 15 most popular self-hosted AI applications of June–September 2026, ranked by GitHub stars, with homelab fit and… |
| [[PICO-8 Asset Standards]] | Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction,… |
| [[Service Research Brief]] | Consolidated research from the `frank-zealot` research subagents (2026-08-14) plus the pi agent knowledge base. |
| [[SGLang]] | What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to… |
**Operations:**
## LLM technique and hardware references
- [[Remote Job Runner]] — detached long-job execution with sentinel files
- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
stage, ask; plus purge-and-refresh
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
the pasta port gate
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
stop-first, pre-commit iteration, incident-window reconstruction, WoL
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
posture, whole-fleet audit, bulk redaction
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
GUI-hang diagnosis over ssh
| Page | Owns |
|---|---|
| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
| [[Quantization Production]] | — |
| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
| [[swift15-campaign-worklog]] | — |
| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
**Architecture:**
## Pipeline Parallelism (series)
- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
signals
- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
cutover
- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
interruptions, shared-checkout hygiene
| Page | Owns |
|---|---|
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
**Infrastructure:**
## Uncategorised
- [[Observability Provisioning]] — Prometheus/Grafana config as code
- [[Homepage Status Probes]] — the status page's checker, vantage rules,
and triage
**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**
- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
[[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
[[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
[[vLLM Boot Forensics and Config Interrogation]] ·
[[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
[[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
[[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
[[FreeToken Builds]] · [[Quantization Production]]
**Agent:**
- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
audits
- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment
**Desktop / Research:**
- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
rendering, rotation pinning, iCX3 RGB
- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets
## LLM Fleet
The LLM inference fleet on the two hosts. Full index:
[serverAppConfigs-docs/LLM/index.md](LLM/index.md).
### Deployment and Topology
- [[deployment]] — unified deploy and update process for both servers
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE frank-archon runbook
(2026-10-15)**: the PUBLISHER's container of the official
Qwen3.8-Flash-Next (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch)
swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab
25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries
verbatim; rollback = revert the swap commit (the w4dn8 `.rad` stays on
disk; the fp8-native `.rad` was deleted 2026-10-15, rebuildable)
- [[frank-archon-radiance-swift15-w4dn8]] — RETIRED 2026-10-08 → 10-15 frank-archon runbook:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` (radiance
1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s;
activates by flipping `MODEL_FILE` in `scripts/frank-archon.env`, rollback
= revert that line (the FP8-native `.rad` stays on disk)
- [[frank-archon-radiance-qwen38-native-fp8]] — the RETIRED 2026-10-05 → 10-08 frank-archon
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts
in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV
7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
FreeToken one-card era (tag `frank-archon-freetoken-pr587-20261005`),
then the tcclaviger shape-G era (tag `frank-archon-vllm-shapeG-20261003`)
- [[swift15-campaign-worklog]] — ACTIVE campaign log: the Swift1.5 →
w4dn8 radiance convert and frank-archon swap in flight (branch
`swift15-radiance-w4dn8`); the cutover runbook replaces it at completion.
- [[frank-archon-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
same day by the radiance cutover above; kept as the rollback runbook):
FreeToken built at PR #587 head `77a7201` on the host into
`frank-archon-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
`2dac17a2`, docker→podman shim) serving the unsloth
Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
LRU cache), PLE pinned 54.4 GB in host RAM, CED off, 131,072 ctx single
stream, Qwen-Sharp template, root unit with LimitMEMLOCK, the on-host
qualified state, the cutover, and the rollback
- [[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(rollback target): the R9V radiance-vLLM fork serving the abliterated
Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
vision — TP=2 tiered experts with a UVA pinned host copy, PLE on
ssd-mmap, CED on, 196,608 ctx; state dir + frozen config + payload
intact on disk
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
(rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
behind ngram-mod, BF16 vision, the experts and the 51B n-gram table
computed in host RAM, the placement law, the cutover, and the rollback
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
rollback target): the Lucebox runtime serving
DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
with the routed experts streamed from the mmapped page cache — held :7770
for hours
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
(320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
assessment — reusable expert tiering, four architecture gaps, one a
silent-wrongness trap on the Engram key
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
36% precision, wasting ~half the gather traffic (the LRU itself runs at
82.45% hit); the prefetch-on incumbent 26.8 survived every other
challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
slab size)
- [[frank-archon-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
frank-archon runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
(symmetric int4 g128) on the tcclaviger fork, and the reason the int4
AutoRound payload is the ONLY one that serves here (its `auto-round` quant
method routes to the W4A16 offload kernel; a compressed-tensors W4A16
checkpoint of the same model reaches TritonWNA16Experts, which has no
offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
35 tok/s target is NOT met** — the step is bounded by the fork's gather
read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
`glm47` tool/reasoning parser wiring
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
entrypoint, the three-arm battery that chose TP=2, verified capabilities,
and the record of the 2026-09-16 two-replica era
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
quadlet edit, the deploy, the capability battery, and the rollback path
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
163,840-token slots); holds the batch/ubatch optimum curve and the
per-layer split arithmetic the rollback recipe points back to
- [[Dual-R9700-Cooperation]] — whether the two R9700s can run ONE model: the
RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor
parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed
— both chains `Routing+`, both endpoints `ReqEn+` — and the measured
three-way verdict: radiance TP=2 won and is the deployed shape
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
charlie12345/ROCmFPX serving kingjones777/
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
replaced
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
- [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
- [[agentionai-fast-imatrix]] — the retired frank-dragoon runbook:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record)
- [[frank-dragoon-halogen27b-dflash2]] — RETIRED 2026-10-05 → 10-06 frank-dragoon
runbook (root systemd unit + podman, port 7770): the digest-pinned
ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b
(qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other
engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1
disables speculation — engine law); TEXT-ONLY and censored base, both
accepted tradeoffs; no server-side sampler, so the thinking recipe rides
the wire
- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook
since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2
abliterated** conversion of orcarouter's pinned BF16
(`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a
**401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, 11.0 GiB
reserved — one pool behind both slots, raised to 2 × ctx and deployed
2026-10-18 so TWO maximum-length
requests coexist) and the stock peonist n-gram/vision companions **hardlinked**
in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP
always-on, vision ON, no server-side sampler change
- [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17
frank-dragoon runbook (rollback target — payload + fetch script on disk):
halogen-flash-server 0.16.2 serving the abliterated **Swift1.5 finetune**
of the Qwen3.8-Flash-Next v2 `.hgn`, two slots over a 524,288-position
pool; owns the 400-zone law and the sampler/prompt-cache contract
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05
(rollback target; payload, fork build and build script on disk): the pinned
kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
**Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of
GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the
host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1
t/s measured, 86.8 % acceptance; the ngram-mod chain measured and
rejected) and its BF16 projector, one session at the native 262,144 window
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback
target, aligned Q8 payload still on disk): the same fork serving the
ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
- [[frank-dragoon-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history;
the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh`
remains): gufo v0.1.1
built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
drafter on the gfx1151 iGPU — the two from-source build workarounds, the
template-hash allowlist, the measured prefill/decode trade
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
replaced by the gufo rollback, then by
[[frank-dragoon-rocmfpx-q8agent-mtp]]): the digest-pinned
kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
the `reasoning` field change, troubleshooting
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
(depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
(the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
114 GiB GTT memory model, client contract
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (a 27B
dense model, main GGUF + FFN/GDN/GDN-Output .pfs sidecars) on the gfx1151
iGPU — one 262,144-token conversation, native MTP depth 4 + n-gram
drafting, compatibility mode, the 114 GiB GTT memory model, client contract
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-kairic-edge-qwen38-27b]]; same-day swap): the unsloth
llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (a 125B MoE / 6B active) on the
gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the
explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target, 2026-09-19 →
09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
gate, client contract
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-dragoon-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
conversations over one 400,384-position pool, deployed flags, memory model,
client contract
- [[frank-dragoon-ik-uncensored]] — the RETIRED 2026-09-16 frank-dragoon runbook
and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
Strix Halo hybrid — its measured envelope is why the fleet went back
- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: `iommu=pt`
since 2026-10-18 (measured cost of identity DMA: nothing — +2.0 % prefill), the
`amd_iommu=off` era it replaced, and the tracked file + installer that own it
- [[kingjones-strix-rocmfp4]] — the PRIOR frank-dragoon runbook
(2026-08-30 → 2026-09-11) and the rollback target:
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
pinned kingjones30/ROCmFPX build
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-zealot-llama-prism-bonsai9-mtp]] — the **ACTIVE** frank-zealot
runbook since 2026-10-18: the PrismML vendor llama.cpp build 10777 (the only
loader for the custom PQ2_0 tensor type / `prism.hadamard.*`, and the floor
for MTP beside a quantized V cache) serving
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head** on the RTX 3090
— TWO 262,144-token conversations (`--ctx-size 524288 --parallel 2`), exact
f16 K + f16 V = **16,384 MiB** over 8 full-attention layers (32,768 B/token;
the per-slot window is UNCHANGED at 262,144 — what halved is the POOL,
because exact f16 over the OLD 786,432 pool bills **24,576 MiB** against the
**24,123** the driver leaves addressable; the old "26,112 vs 13,056 MiB" note
doubled the q8_0 figure with its 1/32 scale overhead included, so f16/q8_0 is
1.882×, not 2×), MTP **off by measurement** (the draft context bills 2,048
MiB of f16 KV over the whole pool, and exact KV leaves no room for one),
vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free (the q8_0 era)
- [[frank-zealot-llama-prism-bonsai2-pq20]] — RETIRED 2026-10-17 → 10-18
(rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-**27B**
PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP **impossible**
(no head in that export), vision through the Q8_0 mmproj; boot 22,664 / 1,461
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — RETIRED 2026-10-17 (rollback
target, payload on disk): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
- [[frank-zealot-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day;
payload on disk as rollback): llama.cpp master (CUDA sm_86) serving
HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the
`gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
with it, 21,890/2,235 without), vision through the publisher's mmproj
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
(superseded by gemma4; payload on disk as the rollback): the
frank-zealot runbook 2026-10-05 (evening) → 2026-10-07
(**UNCECERSED RCO-IQ3_XXS
with MTP back on since 2026-10-06 evening; renamed from frank-small and
reshaped to TWO 114,688-token conversations at Frank's 112k cap with
the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA
sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
(the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free
at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision
through the publisher's own projector, ~52–58 t/s recipe decode with MTP
- [[frank-zealot-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
superseded by swift15): llama.cpp master serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
200,704 tokens each with fully unquantized f16 KV and mmproj vision;
94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M —
weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
images answer "video")
- [[frank-zealot-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
target, payload on disk; owns the measured boot-pair table): same runtime
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
- [[frank-zealot-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
[[frank-zealot-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
SGLang in a self-contained python3.12 venv serving
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
2 x 262,144 slots, bf16 mamba states, eager prefill
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
(MTP assistant sidecar + ngram-mod, q8_0 KV) on
Dankclaw's RTX 3090, port 7771
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
token pool, 2 concurrent full-context sessions, native text+image+audio
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
the prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-zealot (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon)
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize
ordering that drains the old file into fresh swap instead of into a host
whose `MemAvailable` lies by the size of the model, and the swappiness caveat
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
retirement record, including the IOMMU/SVA err=-19 root cause and
where every dependent route moved
### Benchmarks and Sizing
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-zealot-benchmark]] — the small-model benchmark suite
- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x
R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the
logical batch buys nothing measurable, decode never moves, and the ubatch's
uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved
the tensor split from 36:29 to 35:30
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot
do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that
decides a split, the one flag (`-ot`) that kills per-tensor placement across
devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor
CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the
first round of measurements say otherwise
### Memory and Hardware
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
### Retired Topology and History
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
### vLLM Reference (historical)
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[vllm_research_findings]] — research findings from the vLLM era
### Pipeline Parallelism (archive)
- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
- [[Expected Performance]] — expected PP throughput
- [[llama.cpp RPC]] — the RPC transport between head and worker
- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
- [[Sample Script Set]] — reference scripts for the PP pair
- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation
## Update Log
- [[log]] — chronological record of knowledge-base changes
## See Also
- [AGENTS.md](../AGENTS.md) — repository rules
- [serverAppConfigs-docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
- [dankware Configuration Knowledgebase](~/.pi/agent/docs/index.md) — the "root" level system knowlegebase
| Page | Owns |
|---|---|
| [[log]] | — |