Frank
0bf26b0741
The KV pool at the fp8 shape backed 386,904 tokens — 1.93 sessions of the served 200,000 window, so a second full-height conversation preempted to the host/disk prefix tiers. Moving the cache to bf16 (the engine's only unquantized rung; `--kv-cache-dtype` takes bf16|fp8 and has no per-tensor K/V split) doubles what a token costs: 7.3125 -> 14.625 KiB per rank (kv_attn 26.00 -> 52.00 KiB per 4-token page + kv_qsa_bkey 3.25 -> 6.50). Pool 3543 -> 6912 MiB (437k tokens, >= 2 x 200,000), and the weights slab pays for all of it: 26071 -> 22208 MiB, the pair pinned to the measured card ceiling 29,120 MiB (card 31,856 - 1,710 consumed outside both budgets - 1,024 free). The resident set that leaves VRAM spills into the pinned host pool, so 40960 -> 44928 MiB, and the prefix host tier doubles with the entry size (58.50 -> 117 KiB): 4096 -> 8192 MiB. Pinned ledger ~147 GiB, ttm floor 140000 -> 150000 (cap 200 GiB). --max-num-seqs 3 -> 2: two slots x the window is the capacity contract, and the "seqs 2 cannot boot" law is the retired tcclaviger fork's torch.compile query_start_loc constraint, not radiance's (radiance booted seqs 2 in the fp8-native era; the 2026-10-06 sweep had 2 and 3 tied inside +-3%). Bonus: the fused all-reduce is now declared over 2 x 3 = 6 rows, under the 25-row bound at which the wht6 wire is bit-exact. Cost, paid twice on this engine: idle KV VRAM is also the expert cache's spillover (lends 2.69 GiB + 740 MiB of arena), so the pool steals weight residency from the slab AND the lendable cache. Priors: 2.14 GiB of stolen weights cost 2.5% decode (MAX_MODEL_LEN 262144), the EVC ladder's other direction cost 8.9%. Prediction -3..-8% vs the fp8 control (136 t/s at 44.5k, acceptance 0.7246); the deploy boot ledger and live probes are the referee. Seams: MAX_SEQS/HOST_POOL/KV_DTYPE/PREFIX_HOST literals follow the shape, and two new invariants guard it — the KV pool must back seqs x max-model-len at the declared dtype (an engine hard limit: an over-long request is accepted then refused as too large to ever fit; the formula reproduces the fp8 boot's 386,904 to within 0.1%), and the VRAM pair must stay inside the ceiling.
Server App Configs
The single repository for the two homelab servers: the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script.
2026-08-22: the former
vllmScriptsrepository (LLM fleet) is merged into this repo with its full git history. See AGENTS.md for the merge note.
Targets
| Host | User | IP | Podman stack | LLM service | LLM port |
|---|---|---|---|---|---|
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-dragoon (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
Active Services
| Service | Host | Hostname | Port |
|---|---|---|---|
| RomM | dankserver | romm.dankserver.local | 8085 |
| Copyparty | dankserver | files.dankserver.local | 3923 |
| Calibre-Web | dankserver | books.dankserver.local | 8083 |
| frank-medium | dankserver | — (bare endpoint) | 7770 |
| SearXNG | dankclaw | search.dankclaw.local | 8081 |
| ComfyUI | dankclaw | comfyui.dankclaw.local | 8188 |
| OpenWebUI | dankclaw | chat.dankclaw.local | 3000 |
| Hermes | dankclaw | hermes.dankclaw.local | 8090 |
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
| frank-dragoon | dankclaw | — (bare endpoint) | 7770 |
Structure
dankserver/— compose.yaml + Caddyfile + .env.example for the Dankserver stackdankclaw/— compose.yaml + Caddyfile + .env.example for the Dankclaw stackshared/— homepage, prometheus config, deploy helpersscripts/—deploy.shanddeploy-remote.sh(service-scoped deployment), the planner, LLM serve scripts, systemd units, env tunables, and bench harnessesold/— retired PP/NPU-era LLM scripts (reference only)serverAppConfigs-docs/— project knowledge base (Obsidian vault, wikilinks).serverAppConfigs-docs/LLM/is the LLM fleettests/— bats test suite (see Testing Policy)
Deployment
See serverAppConfigs-docs/Architecture/Deployment Flow.md
and serverAppConfigs-docs/LLM/deployment.md. Run
./scripts/deploy.sh <dankclaw|dankserver> after committing changes. It
plans and deploys only the changed services. Use --service NAME for an
explicit single-service retry; inference is restarted only when its own files
(or the shared chat template) changed.
Knowledge Base
The serverAppConfigs-docs/ directory is an Obsidian vault. Start at
serverAppConfigs-docs/index.md. The LLM fleet knowledge base lives at
serverAppConfigs-docs/LLM/index.md.
Testing
Run ./tests/run.sh before every push. See
serverAppConfigs-docs/Architecture/Testing Policy.md.