Frank
54b2d84613
frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
7.1 KiB
7.1 KiB
Server App Configs
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
This repository is the single source of truth for both hosts. It tracks the
container stacks and the LLM inference fleet. Edits flow through this repo,
then deploy to the targets with scripts/deploy.sh (one script per target
covers both workloads).
2026-08-22: the former
vllmScriptsrepository (LLM fleet) is merged into this repo with its full git history. Its knowledge base now lives at docs/LLM/index.md.
Hosts
| Host | IP | Role | OS |
|---|---|---|---|
| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |
Stacks
- Dankserver Stack — RomM, Copyparty, Calibre, Calibre-Web, Caddy
- Dankclaw Stack — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy
Architecture
- Network Architecture — Caddy reverse proxy, hostname routing, Gluetun VPN path
- Storage Policy — bind mounts, data locations, ownership
- Deployment Flow — git-based deploy, target map, health checks
- Testing Policy — the test seams and how to run them
Dankserver Services
- RomM — retro game manager
- RomM-Scraping — metadata providers, cover art, naming conventions
- Copyparty — file sharing and streaming
- Jellyfin — media server (movies, TV, music)
- Calibre — e-book library engine
- Calibre-Web — e-book web frontend
- ARR-Stack — automated media download overview
- sonarr — TV show management
- radarr — movie management
- lidarr — music management
- readarr — ebook/audiobook management
- prowlarr — indexer management
- qbittorrent — torrent download client
- mariadb — RomM database backend
- redis — RomM cache/queue backend
Dankclaw Services
- SearXNG — privacy metasearch (VPN-routed)
- SearXNG Providers — unblocking Google & other upstream engines behind the VPN
- Gluetun — the VPN tunnel container
- Valkey — SearXNG cache and limiter store
- ComfyUI — node-based AI image generation (CPU)
- OpenWebUI — chat UI for local LLMs
- Hermes — the Nous Research agent
- Hermes-Dashboard — the Hermes web management UI (config, keys, sessions)
- Gitea — Git hosting (migrated)
- Postgres — Gitea database backend
Infrastructure
- LAN-DNS — name resolution, Pi-hole, router DNS hand-off
- Pi-hole — the LAN DNS server (quadlet, records, web UI)
- Caddy — reverse proxy and TLS
- Authentik — the homelab's SSO / OIDC provider (forward_auth)
- authentik-postgres — Authentik database service
- authentik-server — Authentik server + Caddy outpost service
- authentik-worker — Authentik background worker service
- Homepage — password-protected service index
- Homepage Design Grilling — homepage decision log and open questions
- Observability — Grafana, Prometheus, Loki, Alloy
- Grafana — dashboards
- Prometheus — metrics collection
- Node-Exporter — host metrics
- Loki — log storage
- Alloy — log shipping (replaces Promtail)
Research
- Service Research Brief — consolidated subagent research behind these pages
- ComfyUI ROCm and Vulkan — live-verified GPU path for ComfyUI on Strix Halo
- Lemonade and sd.cpp — the ggml-family image-generation alternative
- Backblaze Integration — off-site backup plan (future)
- Jellyfin — media server research brief
- ARR-Stack — automated download stack research brief
LLM Fleet
The LLM inference fleet on the two hosts. Full index: docs/LLM/index.md.
Deployment and Topology
- deployment — unified deploy and update process for both servers
- frank-redo-2026-08-17 — the 2026-08-17 topology change
- eemin-rocmi4-mtp — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active)
- kingjones-strix-mtp — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25)
- kingjones-q8agent-rocmfpx — the Q8_0-AGENT runbook for frank-default-large (active): serving, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT
- ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- kingjones-strix-mtp — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- dealign-ornith-crack — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
- kairic-edge — the retired Kairic Edge IU4 experiment (2026-08-22)
- dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)
- Dankserver System Specs — the upstream llama.cpp box hardware
- Dankclaw System Specs — the CIRU ROCmFPX box hardware
- Dankclaw Improvements — host-level changes on the CIRU box
Benchmarks and Sizing
- frank-default-large-benchmark — backend/spec benchmark behind the old head
- frank-default-small-benchmark — the small-model benchmark suite
- Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes
- Running Qwen3.6 27B FP8 on Dankclaw — Qwen3.6-27B sizing on the iGPU
- qwen3.8-strix-optimizations — Strix Halo tuning notes
Memory and Hardware
- RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
- Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
- Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration
- dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
- Quantization by Hardware Architecture — which quant format fits which GPU/CPU
Retired Topology and History
- Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
- qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
- FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
- DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
- ling-3.0-flash-trial — the Ling-3.0-Flash trial runbook
- ling-3.0-flash-vllm-plan — the Ling-3.0-Flash vLLM plan (retired)
vLLM Reference (historical)
- VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
- VLLM Index — quick reference with commands, endpoints, and parameter table
- vllm_research_findings — research findings from the vLLM era
Pipeline Parallelism (archive)
- Ethernet Bandwidth Math — the bandwidth budget for the PP pair
- Expected Performance — expected PP throughput
- llama.cpp RPC — the RPC transport between head and worker
- Mixed-Vendor Experiments — cross-vendor PP experiments
- Qwen3.6 27B Spec Methods — spec-decoding methods for the 27B
- Sample Script Set — reference scripts for the PP pair
- vLLM Pipeline Parallelism — vLLM's PP implementation
See Also
- AGENTS.md — repository rules
- docs/LLM/index.md — the LLM fleet knowledge base