Files

T

Frank 08f1c7ad09 feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.

2026-09-20 19:35:56 -07:00

14 KiB

Raw Blame History

Server App Configs

Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.

This repository is the single source of truth for both hosts. It tracks the container stacks and the LLM inference fleet. Edits flow through this repo, then deploy to the targets with scripts/deploy.sh (one script per target covers both workloads).

2026-08-22: the former vllmScripts repository (LLM fleet) is merged into this repo with its full git history. Its knowledge base now lives at serverAppConfigs-docs/LLM/index.md.

Hosts

Host	IP	Role	OS
Dankserver	192.168.0.235	Media & library stack	Ubuntu 24.04
Dankclaw	192.168.0.143	AI, search & development stack	Ubuntu 26.04

Stacks

Dankserver Stack — RomM, Copyparty, Calibre, Calibre-Web, Caddy
Dankclaw Stack — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy

Architecture

Network Architecture — Caddy reverse proxy, hostname routing, Gluetun VPN path
Storage Policy — bind mounts, data locations, ownership
Deployment Flow — service-scoped git deploy, target map, and health checks
Testing Policy — the test seams and how to run them
E2E Suite — the Playwright suite that drives the live guard chain
Container Network Namespaces — which local services are addressable from each Podman network namespace on Dankclaw

Dankserver Storage

Entropy Volume — command-line unlock and mount helper

Dankserver Services

RomM — retro game manager
RomM-Scraping — metadata providers, cover art, naming conventions
Copyparty — file sharing and streaming
Jellyfin — media server (movies, TV, music)
Calibre — e-book library engine
Calibre-Web — e-book web frontend
ARR-Stack — automated media download overview
sonarr — TV show management
radarr — movie management
lidarr — music management
readarr — ebook/audiobook management
prowlarr — indexer management
qbittorrent — torrent download client
Plex — media server on Dankserver (lscr.io/linuxserver/plex container via quadlet; CPU decode only — no GPU passthrough)
Tautulli — Plex statistics and notifications (v2.18.1-ls242, quadlet unit)
mariadb — RomM database backend
redis — RomM cache/queue backend

Dankclaw Services

SearXNG — privacy metasearch (VPN-routed)
SearXNG Providers — unblocking Google & other upstream engines behind the VPN
Gluetun — the VPN tunnel container
Valkey — SearXNG cache and limiter store
ComfyUI — node-based AI image generation (ROCm GPU, gfx1151 build)
OpenWebUI — chat UI for local LLMs
Hermes — the Nous Research agent
Hermes-Dashboard — the Hermes web management UI (config, keys, sessions)
Hermes-WebUI — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
Gitea — Git hosting (migrated)
Postgres — Gitea database backend
Seerr — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
Marreta — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)

Infrastructure

LAN-DNS — name resolution, Pi-hole, router DNS hand-off
Pi-hole — the LAN DNS server (quadlet, records, web UI)
Caddy — reverse proxy and TLS
Anubis — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
Authentik — the homelab's SSO / OIDC provider (forward_auth)
authentik-postgres — Authentik database service
authentik-server — Authentik server + Caddy outpost service
authentik-worker — Authentik background worker service
Homepage — password-protected service index
Homepage Design Grilling — homepage decision log and open questions
Observability — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
Grafana — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
Prometheus — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
Alertmanager — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
Node-Exporter — host metrics
Loki — log storage
Alloy — log shipping (replaces Promtail)
FrankBook-Pro — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)

Research

Service Research Brief — consolidated subagent research behind these pages
ComfyUI ROCm and Vulkan — live-verified GPU path for ComfyUI on Strix Halo
Lemonade and sd.cpp — the ggml-family image-generation alternative
Llama Cpp Runtimes — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit
Llama Cpp Code Diff — code-level quantization diff of the three llama.cpp runtimes
SGLang — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit)
Backblaze Integration — off-site backup plan (future)
Jellyfin — media server research brief
ARR-Stack — automated download stack research brief

LLM Fleet

The LLM inference fleet on the two hosts. Full index: serverAppConfigs-docs/LLM/index.md.

Deployment and Topology

deployment — unified deploy and update process for both servers
frank-redo-2026-08-17 — the 2026-08-17 topology change
frank-medium-radiance-mxfp4 — the ACTIVE frank-medium runbook since 2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700 behind the :7770 endpoint — deployed flags, the three-arm battery that chose it, verified capabilities, and the record of the 2026-09-16 two-replica era
frank-medium-llamacpp-dflash2 — RETIRED 2026-09-16 → 2026-09-20: mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two 163,840-token slots); holds the batch/ubatch optimum curve and the per-layer split arithmetic the rollback recipe points back to
Dual-R9700-Cooperation — whether the two R9700s can run ONE model: the RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed — both chains Routing+, both endpoints ReqEn+ — and the measured three-way verdict: radiance TP=2 won and is the deployed shape
frank-medium-rocmfpx — PRIOR frank-medium deployment (llama.cpp era): charlie12345/ROCmFPX serving kingjones777/ Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it replaced
freetoken-benchmark-sweep — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
kingjones-strix-mtp — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
agentionai-fast-imatrix — the retired frank-large runbook: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record)
frank-large-ciru-orca — the ACTIVE frank-large runbook since 2026-09-19 (root systemd unit, port 7770): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root gate, client contract
halogen-flash-server — RETIRED 2026-09-19 (replaced by frank-large-ciru-orca; rollback target, 2026-09-16 → 09-19): peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8- flash-next (w4b .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract
frank-large-ik-uncensored — the RETIRED 2026-09-16 frank-large runbook and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid — its measured envelope is why the fleet went back
dankclaw-iommu-translation-off — why Dankclaw boots with amd_iommu=off since 2026-09-11 (13-16 % prefill), what it cost (the NPU endpoint), and how to revert
kingjones-strix-rocmfp4 — the PRIOR frank-large runbook (2026-08-30 → 2026-09-11) and the rollback target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's pinned kingjones30/ROCmFPX build
kingjones-q8agent-rocmfpx — the retired Q8_0-AGENT investigation and set_rows failure
eemin-rocmi4-mtp — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
kingjones-strix-mtp — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
dealign-ornith-crack — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
frank-small-gemma4-12b — the ACTIVE frank-small runbook since 2026-09-17: mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on Dankclaw's RTX 3090, port 7771
frank-small-vllm-gemma4 — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- token pool, 2 concurrent full-context sessions, native text+image+audio
frank-small-qwen38-gsq-rco — RETIRED 2026-09-17 (rollback target): the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
ikllama-ornith-dankserver — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
freetoken-frank-small — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
kairic-edge — the retired Kairic Edge IU4 experiment (2026-08-22)
dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)
Dankserver System Specs — the 2x R9700 box hardware (frank-medium)
Dankclaw System Specs — the Strix Halo gfx1151 box (frank-large)
Dankclaw Improvements — host-level changes on the Strix Halo box
Swap Configuration — the 64 GiB /swap.img host swapfile (raised from 8 GiB 2026-09-12): fstab-driven swap.img.swap unit, the safe resize ordering that drains the old file into fresh swap instead of into a host whose MemAvailable lies by the size of the model, and the swappiness caveat
frank-npu — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11): Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the retirement record, including the IOMMU/SVA err=-19 root cause and where every dependent route moved

Benchmarks and Sizing

frank-large-benchmark — backend/spec benchmark behind the old head
ubatch-benchmark-large — active-model llama-bench ubatch sweep plan and results
frank-small-benchmark — the small-model benchmark suite
frank-medium-batch-ubatch — the 2026-09-16 batch/ubatch sweep on the 2x R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the logical batch buys nothing measurable, decode never moves, and the ubatch's uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved the tensor split from 36:29 to 35:30
Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes
Running Qwen3.6 27B FP8 on Dankclaw — Qwen3.6-27B sizing on the iGPU
qwen3.8-strix-optimizations — Strix Halo tuning notes
x4-pair-experiment — what a 3090 + Strix Halo combination can and cannot do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that decides a split, the one flag (-ot) that kills per-tensor placement across devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the first round of measurements say otherwise

Memory and Hardware

RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration
dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
Quantization by Hardware Architecture — which quant format fits which GPU/CPU

Retired Topology and History

Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
ling-3.0-flash-trial — the Ling-3.0-Flash trial runbook
ling-3.0-flash-vllm-plan — the Ling-3.0-Flash vLLM plan (retired)

vLLM Reference (historical)

VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
VLLM Index — quick reference with commands, endpoints, and parameter table
vllm_research_findings — research findings from the vLLM era

Pipeline Parallelism (archive)

Ethernet Bandwidth Math — the bandwidth budget for the PP pair
Expected Performance — expected PP throughput
llama.cpp RPC — the RPC transport between head and worker
Mixed-Vendor Experiments — cross-vendor PP experiments
Qwen3.6 27B Spec Methods — spec-decoding methods for the 27B
Sample Script Set — reference scripts for the PP pair
vLLM Pipeline Parallelism — vLLM's PP implementation

Update Log

log — chronological record of knowledge-base changes

14 KiB Raw Blame History Unescape Escape