Frank
cc29875842
New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with the measured evidence rather than the theory. **The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise `AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1 mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈ 1.4 ms at Gen4 x4. **New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all three, and `send`/`recv` fails too — which kills pipeline parallelism as an escape, since vLLM's PP handoff is exactly that primitive. **Correction.** I previously said no software path exists. That overshot. One is open and unverified: recompiling RCCL so its kernels stop declaring a hostcall buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's device source contains no live printf/assert/__assertfail, so the trigger is unexplained, and the claim traces to search summaries, not a primary source. The page carries the two-command test that would settle it. Also corrected: 0.9.3 is radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory path cannot run. **Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just benchmarks — including a bare 2-rank probe. Running one against the live service on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle required); a hung AQL dispatch wedging the driver is the plausible mechanism. The policy also now says how to tell off from wedged: `ip neigh` from a peer.
227 lines
12 KiB
Markdown
227 lines
12 KiB
Markdown
# Server App Configs
|
||
|
||
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
|
||
|
||
This repository is the single source of truth for both hosts. It tracks the
|
||
container stacks and the LLM inference fleet. Edits flow through this repo,
|
||
then deploy to the targets with `scripts/deploy.sh` (one script per target
|
||
covers both workloads).
|
||
|
||
> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
|
||
> into this repo with its full git history. Its knowledge base now lives at
|
||
> [serverAppConfigs-docs/LLM/index.md](LLM/index.md).
|
||
|
||
## Hosts
|
||
|
||
| Host | IP | Role | OS |
|
||
|------|----|------|----|
|
||
| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
|
||
| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |
|
||
|
||
## Stacks
|
||
|
||
- [[Dankserver Stack]] — RomM, Copyparty, Calibre, Calibre-Web, Caddy
|
||
- [[Dankclaw Stack]] — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy
|
||
|
||
## Architecture
|
||
|
||
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
|
||
- [[Storage Policy]] — bind mounts, data locations, ownership
|
||
- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
|
||
- [[Testing Policy]] — the test seams and how to run them
|
||
- [[E2E Suite]] — the Playwright suite that drives the live guard chain
|
||
- [[Container Network Namespaces]] — which local services are addressable from each Podman network namespace on Dankclaw
|
||
|
||
## Dankserver Storage
|
||
|
||
- [[Entropy Volume]] — command-line unlock and mount helper
|
||
|
||
## Dankserver Services
|
||
|
||
- [[RomM]] — retro game manager
|
||
- [[RomM-Scraping]] — metadata providers, cover art, naming conventions
|
||
- [[Copyparty]] — file sharing and streaming
|
||
- [[Jellyfin]] — media server (movies, TV, music)
|
||
- [[Calibre]] — e-book library engine
|
||
- [[Calibre-Web]] — e-book web frontend
|
||
- [[ARR-Stack]] — automated media download overview
|
||
- [[sonarr]] — TV show management
|
||
- [[radarr]] — movie management
|
||
- [[lidarr]] — music management
|
||
- [[readarr]] — ebook/audiobook management
|
||
- [[prowlarr]] — indexer management
|
||
- [[qbittorrent]] — torrent download client
|
||
- [[Plex]] — media server on Dankserver (lscr.io/linuxserver/plex container via
|
||
quadlet; CPU decode only — no GPU passthrough)
|
||
- [[Tautulli]] — Plex statistics and notifications (v2.18.1-ls242, quadlet unit)
|
||
- [[mariadb]] — RomM database backend
|
||
- [[redis]] — RomM cache/queue backend
|
||
|
||
## Dankclaw Services
|
||
|
||
- [[SearXNG]] — privacy metasearch (VPN-routed)
|
||
- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
|
||
- [[Gluetun]] — the VPN tunnel container
|
||
- [[Valkey]] — SearXNG cache and limiter store
|
||
- [[ComfyUI]] — node-based AI image generation (ROCm GPU, gfx1151 build)
|
||
- [[OpenWebUI]] — chat UI for local LLMs
|
||
- [[Hermes]] — the Nous Research agent
|
||
- [[Hermes-Dashboard]] — the Hermes web management UI (config, keys, sessions)
|
||
- [[Hermes-WebUI]] — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
|
||
- [[Gitea]] — Git hosting (migrated)
|
||
- [[Postgres]] — Gitea database backend
|
||
- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
|
||
- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
|
||
|
||
## Infrastructure
|
||
|
||
- [[LAN-DNS]] — name resolution, Pi-hole, router DNS hand-off
|
||
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
|
||
- [[Caddy]] — reverse proxy and TLS
|
||
- [[Anubis]] — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
|
||
- [[Authentik]] — the homelab's SSO / OIDC provider (forward_auth)
|
||
- [[authentik-postgres]] — Authentik database service
|
||
- [[authentik-server]] — Authentik server + Caddy outpost service
|
||
- [[authentik-worker]] — Authentik background worker service
|
||
- [[Homepage]] — password-protected service index
|
||
- [[Homepage Design Grilling]] — homepage decision log and open questions
|
||
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and what is still missing (alert delivery)
|
||
- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
|
||
- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule; evaluated but not delivered — no Alertmanager)
|
||
- [[Node-Exporter]] — host metrics
|
||
- [[Loki]] — log storage
|
||
- [[Alloy]] — log shipping (replaces Promtail)
|
||
- [[FrankBook-Pro]] — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)
|
||
|
||
## Research
|
||
|
||
- [[Service Research Brief]] — consolidated subagent research behind these pages
|
||
- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
|
||
- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
|
||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit
|
||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three llama.cpp runtimes
|
||
- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit)
|
||
- [[Backblaze Integration]] — off-site backup plan (future)
|
||
- [[Jellyfin]] — media server research brief
|
||
- [[ARR-Stack]] — automated download stack research brief
|
||
|
||
## LLM Fleet
|
||
|
||
The LLM inference fleet on the two hosts. Full index:
|
||
[serverAppConfigs-docs/LLM/index.md](LLM/index.md).
|
||
|
||
### Deployment and Topology
|
||
|
||
- [[deployment]] — unified deploy and update process for both servers
|
||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the
|
||
vllm-radiance MXFP4 stack serving
|
||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon
|
||
AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, plus
|
||
the PCIe-atomics evidence that rules out TP=2 on this host (since
|
||
2026-09-15/16)
|
||
- [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel
|
||
together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
|
||
(collectives *and* P2P `send`/`recv` both fail on card #2, which rules out
|
||
pipeline parallelism too), what is closed and what is still open, and the two
|
||
hardware paths — a ~$30 M.2→PCIe x4 riser into the free, CPU-attached M.2_2
|
||
slot, or a board whose CPU x16 bifurcates to x8/x8
|
||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
|
||
charlie12345/ROCmFPX serving kingjones777/
|
||
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
|
||
replaced
|
||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
|
||
- [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
|
||
standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
|
||
- [[agentionai-fast-imatrix]] — the retired frank-large runbook:
|
||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||
rollback record)
|
||
- [[frank-large-ik-uncensored]] — the ACTIVE frank-large runbook since
|
||
2026-09-15: ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-
|
||
Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid (root systemd
|
||
unit, port 7770) — measured performance envelope and traps
|
||
- [[halogen-flash-server]] — the RETIRED 2026-09-15 halogen runbook and
|
||
rollback target: peonist-ai/halogen-flash-server 0.5.8 serving
|
||
peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet
|
||
container on Dankclaw, port 7770 — flags, memory model, client contract
|
||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost
|
||
(the NPU endpoint), and how to revert
|
||
- [[kingjones-strix-rocmfp4]] — the PRIOR frank-large runbook
|
||
(2026-08-30 → 2026-09-11) and the rollback target:
|
||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
|
||
pinned kingjones30/ROCmFPX build
|
||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
|
||
- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
|
||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
|
||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
|
||
- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium)
|
||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large)
|
||
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
|
||
- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
|
||
8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize
|
||
ordering that drains the old file into fresh swap instead of into a host
|
||
whose `MemAvailable` lies by the size of the model, and the swappiness caveat
|
||
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
|
||
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
|
||
retirement record, including the IOMMU/SVA err=-19 root cause and
|
||
where every dependent route moved
|
||
|
||
### Benchmarks and Sizing
|
||
|
||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||
- [[frank-small-benchmark]] — the small-model benchmark suite
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
|
||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
|
||
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
|
||
|
||
### Memory and Hardware
|
||
|
||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||
|
||
### Retired Topology and History
|
||
|
||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
|
||
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
|
||
|
||
### vLLM Reference (historical)
|
||
|
||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||
- [[vllm_research_findings]] — research findings from the vLLM era
|
||
|
||
### Pipeline Parallelism (archive)
|
||
|
||
- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
|
||
- [[Expected Performance]] — expected PP throughput
|
||
- [[llama.cpp RPC]] — the RPC transport between head and worker
|
||
- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
|
||
- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
|
||
- [[Sample Script Set]] — reference scripts for the PP pair
|
||
- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation
|
||
|
||
## Update Log
|
||
|
||
- [[log]] — chronological record of knowledge-base changes
|
||
|
||
## See Also
|
||
|
||
- [AGENTS.md](../AGENTS.md) — repository rules
|
||
- [serverAppConfigs-docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
|
||
- [dankware Configuration Knowledgebase](~/.pi/agent/docs/index.md) — the "root" level system knowlegebase
|