Frank
cc29875842
New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with the measured evidence rather than the theory. **The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise `AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1 mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈ 1.4 ms at Gen4 x4. **New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all three, and `send`/`recv` fails too — which kills pipeline parallelism as an escape, since vLLM's PP handoff is exactly that primitive. **Correction.** I previously said no software path exists. That overshot. One is open and unverified: recompiling RCCL so its kernels stop declaring a hostcall buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's device source contains no live printf/assert/__assertfail, so the trigger is unexplained, and the claim traces to search summaries, not a primary source. The page carries the two-command test that would settle it. Also corrected: 0.9.3 is radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory path cannot run. **Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just benchmarks — including a bare 2-rank probe. Running one against the live service on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle required); a hung AQL dispatch wedging the driver is the plausible mechanism. The policy also now says how to tell off from wedged: `ip neigh` from a peer.
Server App Configs
The single repository for the two homelab servers: the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script.
2026-08-22: the former
vllmScriptsrepository (LLM fleet) is merged into this repo with its full git history. See AGENTS.md for the merge note.
Targets
| Host | User | IP | Podman stack | LLM service | LLM port |
|---|---|---|---|---|---|
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
Active Services
| Service | Host | Hostname | Port |
|---|---|---|---|
| RomM | dankserver | romm.dankserver.local | 8085 |
| Copyparty | dankserver | files.dankserver.local | 3923 |
| Calibre-Web | dankserver | books.dankserver.local | 8083 |
| frank-medium | dankserver | — (bare endpoint) | 7770 |
| SearXNG | dankclaw | search.dankclaw.local | 8081 |
| ComfyUI | dankclaw | comfyui.dankclaw.local | 8188 |
| OpenWebUI | dankclaw | chat.dankclaw.local | 3000 |
| Hermes | dankclaw | hermes.dankclaw.local | 8090 |
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
| frank-large | dankclaw | — (bare endpoint) | 7770 |
Structure
dankserver/— compose.yaml + Caddyfile + .env.example for the Dankserver stackdankclaw/— compose.yaml + Caddyfile + .env.example for the Dankclaw stackshared/— homepage, prometheus config, deploy helpersscripts/—deploy.shanddeploy-remote.sh(service-scoped deployment), the planner, LLM serve scripts, systemd units, env tunables, and bench harnessesold/— retired PP/NPU-era LLM scripts (reference only)serverAppConfigs-docs/— project knowledge base (Obsidian vault, wikilinks).serverAppConfigs-docs/LLM/is the LLM fleettests/— bats test suite (see Testing Policy)
Deployment
See serverAppConfigs-docs/Architecture/Deployment Flow.md
and serverAppConfigs-docs/LLM/deployment.md. Run
./scripts/deploy.sh <dankclaw|dankserver> after committing changes. It
plans and deploys only the changed services. Use --service NAME for an
explicit single-service retry; inference is restarted only when its own files
(or the shared chat template) changed.
Knowledge Base
The serverAppConfigs-docs/ directory is an Obsidian vault. Start at
serverAppConfigs-docs/index.md. The LLM fleet knowledge base lives at
serverAppConfigs-docs/LLM/index.md.
Testing
Run ./tests/run.sh before every push. See
serverAppConfigs-docs/Architecture/Testing Policy.md.