Frank
54b2d84613
frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
157 lines
7.1 KiB
Markdown
157 lines
7.1 KiB
Markdown
# Server App Configs
|
|
|
|
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
|
|
|
|
This repository is the single source of truth for both hosts. It tracks the
|
|
container stacks and the LLM inference fleet. Edits flow through this repo,
|
|
then deploy to the targets with `scripts/deploy.sh` (one script per target
|
|
covers both workloads).
|
|
|
|
> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
|
|
> into this repo with its full git history. Its knowledge base now lives at
|
|
> [docs/LLM/index.md](LLM/index.md).
|
|
|
|
## Hosts
|
|
|
|
| Host | IP | Role | OS |
|
|
|------|----|------|----|
|
|
| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
|
|
| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |
|
|
|
|
## Stacks
|
|
|
|
- [[Dankserver Stack]] — RomM, Copyparty, Calibre, Calibre-Web, Caddy
|
|
- [[Dankclaw Stack]] — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy
|
|
|
|
## Architecture
|
|
|
|
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
|
|
- [[Storage Policy]] — bind mounts, data locations, ownership
|
|
- [[Deployment Flow]] — git-based deploy, target map, health checks
|
|
- [[Testing Policy]] — the test seams and how to run them
|
|
|
|
## Dankserver Services
|
|
|
|
- [[RomM]] — retro game manager
|
|
- [[RomM-Scraping]] — metadata providers, cover art, naming conventions
|
|
- [[Copyparty]] — file sharing and streaming
|
|
- [[Jellyfin]] — media server (movies, TV, music)
|
|
- [[Calibre]] — e-book library engine
|
|
- [[Calibre-Web]] — e-book web frontend
|
|
- [[ARR-Stack]] — automated media download overview
|
|
- [[sonarr]] — TV show management
|
|
- [[radarr]] — movie management
|
|
- [[lidarr]] — music management
|
|
- [[readarr]] — ebook/audiobook management
|
|
- [[prowlarr]] — indexer management
|
|
- [[qbittorrent]] — torrent download client
|
|
- [[mariadb]] — RomM database backend
|
|
- [[redis]] — RomM cache/queue backend
|
|
|
|
## Dankclaw Services
|
|
|
|
- [[SearXNG]] — privacy metasearch (VPN-routed)
|
|
- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
|
|
- [[Gluetun]] — the VPN tunnel container
|
|
- [[Valkey]] — SearXNG cache and limiter store
|
|
- [[ComfyUI]] — node-based AI image generation (CPU)
|
|
- [[OpenWebUI]] — chat UI for local LLMs
|
|
- [[Hermes]] — the Nous Research agent
|
|
- [[Hermes-Dashboard]] — the Hermes web management UI (config, keys, sessions)
|
|
- [[Gitea]] — Git hosting (migrated)
|
|
- [[Postgres]] — Gitea database backend
|
|
|
|
## Infrastructure
|
|
|
|
- [[LAN-DNS]] — name resolution, Pi-hole, router DNS hand-off
|
|
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
|
|
- [[Caddy]] — reverse proxy and TLS
|
|
- [[Authentik]] — the homelab's SSO / OIDC provider (forward_auth)
|
|
- [[authentik-postgres]] — Authentik database service
|
|
- [[authentik-server]] — Authentik server + Caddy outpost service
|
|
- [[authentik-worker]] — Authentik background worker service
|
|
- [[Homepage]] — password-protected service index
|
|
- [[Homepage Design Grilling]] — homepage decision log and open questions
|
|
- [[Observability]] — Grafana, Prometheus, Loki, Alloy
|
|
- [[Grafana]] — dashboards
|
|
- [[Prometheus]] — metrics collection
|
|
- [[Node-Exporter]] — host metrics
|
|
- [[Loki]] — log storage
|
|
- [[Alloy]] — log shipping (replaces Promtail)
|
|
|
|
## Research
|
|
|
|
- [[Service Research Brief]] — consolidated subagent research behind these pages
|
|
- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
|
|
- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
|
|
- [[Backblaze Integration]] — off-site backup plan (future)
|
|
- [[Jellyfin]] — media server research brief
|
|
- [[ARR-Stack]] — automated download stack research brief
|
|
|
|
## LLM Fleet
|
|
|
|
The LLM inference fleet on the two hosts. Full index:
|
|
[docs/LLM/index.md](LLM/index.md).
|
|
|
|
### Deployment and Topology
|
|
|
|
- [[deployment]] — unified deploy and update process for both servers
|
|
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
|
- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active)
|
|
- [[kingjones-strix-mtp]] — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25)
|
|
- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook for frank-default-large (active): serving, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT
|
|
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
|
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
|
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
|
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
|
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
|
- [[Dankserver System Specs]] — the upstream llama.cpp box hardware
|
|
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box hardware
|
|
- [[Dankclaw Improvements]] — host-level changes on the CIRU box
|
|
|
|
### Benchmarks and Sizing
|
|
|
|
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
|
|
- [[frank-default-small-benchmark]] — the small-model benchmark suite
|
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
|
|
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
|
|
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
|
|
|
|
### Memory and Hardware
|
|
|
|
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
|
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
|
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
|
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
|
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
|
|
|
### Retired Topology and History
|
|
|
|
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
|
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
|
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
|
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
|
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
|
|
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
|
|
|
|
### vLLM Reference (historical)
|
|
|
|
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
|
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
|
- [[vllm_research_findings]] — research findings from the vLLM era
|
|
|
|
### Pipeline Parallelism (archive)
|
|
|
|
- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
|
|
- [[Expected Performance]] — expected PP throughput
|
|
- [[llama.cpp RPC]] — the RPC transport between head and worker
|
|
- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
|
|
- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
|
|
- [[Sample Script Set]] — reference scripts for the PP pair
|
|
- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation
|
|
|
|
## See Also
|
|
|
|
- [AGENTS.md](../AGENTS.md) — repository rules
|
|
- [docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
|