# Server App Configs

Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.

This repository is the single source of truth for both hosts. It tracks the
container stacks and the LLM inference fleet. Edits flow through this repo,
then deploy to the targets with `scripts/deploy.sh` (one script per target
covers both workloads).

> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
> into this repo with its full git history. Its knowledge base now lives at
> [serverAppConfigs-docs/LLM/index.md](LLM/index.md).

## Hosts

| Host | IP | Role | OS |
|------|----|------|----|
| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |

## Stacks

- [[Dankserver Stack]] — RomM, Copyparty, Calibre, Calibre-Web, Caddy
- [[Dankclaw Stack]] — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy

## Architecture

- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
- [[Storage Policy]] — bind mounts, data locations, ownership
- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
- [[Testing Policy]] — the test seams and how to run them
- [[E2E Suite]] — the Playwright suite that drives the live guard chain
- [[Container Network Namespaces]] — which local services are addressable from each Podman network namespace on Dankclaw

## Dankserver Storage

- [[Entropy Volume]] — command-line unlock and mount helper

## Dankserver Services

- [[RomM]] — retro game manager
- [[RomM-Scraping]] — metadata providers, cover art, naming conventions
- [[Copyparty]] — file sharing and streaming
- [[Jellyfin]] — media server (movies, TV, music)
- [[Calibre]] — e-book library engine
- [[Calibre-Web]] — e-book web frontend
- [[Plex]] — media server on Dankserver (lscr.io/linuxserver/plex container via
  quadlet; CPU decode only — no GPU passthrough)
- [[Tautulli]] — Plex statistics and notifications (v2.18.1-ls245, quadlet unit)
- [[mariadb]] — RomM database backend
- [[redis]] — RomM cache/queue backend

## FrankBook-Pro Services

The *arr stack and qBittorrent run on **FrankBook-Pro** (192.168.0.112, the
macOS mini) as quadlets inside its podman machine — not on Dankserver.

- [[ARR-Stack]] — automated media download overview
- [[sonarr]] — TV show management
- [[radarr]] — movie management
- [[lidarr]] — music management
- [[readarr]] — ebook/audiobook management
- [[prowlarr]] — indexer management
- [[qbittorrent]] — torrent download client

## Dankclaw Services

- [[SearXNG]] — privacy metasearch (VPN-routed)
- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
- [[Gluetun]] — the VPN tunnel container
- [[Valkey]] — SearXNG cache and limiter store
- [[ComfyUI]] — node-based AI image generation (ROCm GPU, gfx1151 build)
- [[ComfyUI Qwen-Image Workflows]] — the two proven Qwen-Image 2.1 workflows + what to add next (LoRAs, Edit, ControlNet, Wan video)
- [[OpenWebUI]] — chat UI for local LLMs
- [[Hermes]] — the Nous Research agent
- [[Hermes-Dashboard]] — the Hermes web management UI (config, keys, sessions)
- [[Hermes-WebUI]] — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
- [[Gitea]] — Git hosting (migrated)
- [[Postgres]] — Gitea database backend
- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)

## Infrastructure

- [[LAN-DNS]] — name resolution, Pi-hole, router DNS hand-off
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
- [[Caddy]] — reverse proxy and TLS
- [[Anubis]] — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
- [[Authentik]] — the homelab's SSO / OIDC provider (forward_auth)
- [[authentik-postgres]] — Authentik database service
- [[authentik-server]] — Authentik server + Caddy outpost service
- [[authentik-worker]] — Authentik background worker service
- [[Homepage]] — password-protected service index
- [[Homepage Design Grilling]] — homepage decision log and open questions
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
- [[Alertmanager]] — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
- [[Node-Exporter]] — host metrics
- [[Loki]] — log storage
- [[Alloy]] — log shipping (replaces Promtail)
- [[FrankBook-Pro]] — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)

## Research

- [[Service Research Brief]] — consolidated subagent research behind these pages
- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three llama.cpp runtimes
- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit)
- [[Backblaze Integration]] — off-site backup plan (future)
- [[Jellyfin]] — media server research brief
- [[ARR-Stack]] — automated download stack research brief
- [[Local TTS and Voice Cloning]] — creating a local voice from samples, the 2026 cloning landscape, the full audio stack, and a recommended homelab deployment
- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices

## Skill Distillation (2026-10-05)

The managed skill library distilled into the vault, one reference page per
skill-cluster. See [log.md](log.md) for the wave plan and results.

**Operations:**

- [[Remote Job Runner]] — detached long-job execution with sentinel files
- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
  stage, ask; plus purge-and-refresh
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
  the pasta port gate
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
  stop-first, pre-commit iteration, incident-window reconstruction, WoL
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
  posture, whole-fleet audit, bulk redaction
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
  GUI-hang diagnosis over ssh

**Architecture:**

- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
  signals
- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
  cutover
- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
  interruptions, shared-checkout hygiene

**Infrastructure:**

- [[Observability Provisioning]] — Prometheus/Grafana config as code
- [[Homepage Status Probes]] — the status page's checker, vantage rules,
  and triage

**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**

- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
  [[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
  [[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
  [[vLLM Boot Forensics and Config Interrogation]] ·
  [[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
  [[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
  [[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
  [[FreeToken Builds]]

**Agent:**

- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
  audits
- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment

**Desktop / Research:**

- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
  rendering, rotation pinning, iCX3 RGB
- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets

## LLM Fleet

The LLM inference fleet on the two hosts. Full index:
[serverAppConfigs-docs/LLM/index.md](LLM/index.md).

### Deployment and Topology

- [[deployment]] — unified deploy and update process for both servers
- [[frank-primary-radiance-qwen38-native-fp8]] — the ACTIVE frank-primary
  runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
  287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
  `Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
  after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts
  in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV
  7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
  FreeToken one-card era (tag `frank-primary-freetoken-pr587-20261005`),
  then the tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`)
- [[swift15-campaign-worklog]] — ACTIVE campaign log: the Swift1.5 →
  w4dn8 radiance convert and frank-primary swap in flight (branch
  `swift15-radiance-w4dn8`); the cutover runbook replaces it at completion.
- [[frank-primary-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
  same day by the radiance cutover above; kept as the rollback runbook):
  FreeToken built at PR #587 head `77a7201` on the host into
  `frank-primary-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
  host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
  back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
  `2dac17a2`, docker→podman shim) serving the unsloth
  Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
  BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
  LRU cache), PLE pinned 54.4 GB in host RAM, CED off, 131,072 ctx single
  stream, Qwen-Sharp template, root unit with LimitMEMLOCK, the on-host
  qualified state, the cutover, and the rollback
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
  (rollback target): the R9V radiance-vLLM fork serving the abliterated
  Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
  vision — TP=2 tiered experts with a UVA pinned host copy, PLE on
  ssd-mmap, CED on, 196,608 ctx; state dir + frozen config + payload
  intact on disk
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
  (rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
  gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
  GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
  behind ngram-mod, BF16 vision, the experts and the 51B n-gram table
  computed in host RAM, the placement law, the cutover, and the rollback
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
  rollback target): the Lucebox runtime serving
  DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
  routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
  with the routed experts streamed from the mmapped page cache — held :7770
  for hours
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
  (320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
  FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
  payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
  across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
  assessment — reusable expert tiering, four architecture gaps, one a
  silent-wrongness trap on the Engram key
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
  the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
  (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
  TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
  `--expert-prefetch` DELETED** — telemetry measured the prefetcher at
  36% precision, wasting ~half the gather traffic (the LRU itself runs at
  82.45% hit); the prefetch-on incumbent 26.8 survived every other
  challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
  concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
  fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
  slab size)
- [[frank-primary-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
  frank-primary runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
  checkpoint of the same model reaches TritonWNA16Experts, which has no
  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
  35 tok/s target is NOT met** — the step is bounded by the fork's gather
  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
  `glm47` tool/reasoning parser wiring
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
  W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
  2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
  entrypoint, the three-arm battery that chose TP=2, verified capabilities,
  and the record of the 2026-09-16 two-replica era
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
  the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
  swap-frank-medium.sh), the quantization-aware entrypoint contract, the
  quadlet edit, the deploy, the capability battery, and the rollback path
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
  mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
  Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
  163,840-token slots); holds the batch/ubatch optimum curve and the
  per-layer split arithmetic the rollback recipe points back to
- [[Dual-R9700-Cooperation]] — whether the two R9700s can run ONE model: the
  RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor
  parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed
  — both chains `Routing+`, both endpoints `ReqEn+` — and the measured
  three-way verdict: radiance TP=2 won and is the deployed shape
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
  charlie12345/ROCmFPX serving kingjones777/
  Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
  replaced
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
- [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
  standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
- [[agentionai-fast-imatrix]] — the retired frank-large runbook:
  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
  Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
  rollback record)
- [[frank-large-halogen27b-dflash2]] — the ACTIVE frank-large runbook since
  2026-10-05 (root systemd unit + podman, port 7770): the digest-pinned
  ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b
  (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other
  engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1
  disables speculation — engine law); TEXT-ONLY and censored base, both
  accepted tradeoffs; no server-side sampler, so the thinking recipe rides
  the wire
- [[frank-large-flash-server-v2-swift15]] — the ACTIVE frank-large runbook
  since 2026-10-06 (supersedes halogen 0.1.4): halogen-flash-server 0.16.2
  serving the abliterated Swift1.5 Qwen3.8-Flash-Next v2 (.hgn)
- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05
  (rollback target; payload, fork build and build script on disk): the pinned
  kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
  Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
  **Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of
  GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the
  host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1
  t/s measured, 86.8 % acceptance; the ngram-mod chain measured and
  rejected) and its BF16 projector, one session at the native 262,144 window
- [[frank-large-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback
  target, aligned Q8 payload still on disk): the same fork serving the
  ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history;
  the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh`
  remains): gufo v0.1.1
  built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
  drafter on the gfx1151 iGPU — the two from-source build workarounds, the
  template-hash allowlist, the measured prefill/decode trade
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
  replaced by the gufo rollback, then by
  [[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
  kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
  4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
  the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
  MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
  the `reasoning` field change, troubleshooting
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
  [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
  (unsloth b11030, ROCm gfx1151) serving
  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
  (depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
  (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
  114 GiB GTT memory model, client contract
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
  [[frank-large-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
  runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
  Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (a 27B
  dense model, main GGUF + FFN/GDN/GDN-Output .pfs sidecars) on the gfx1151
  iGPU — one 262,144-token conversation, native MTP depth 4 + n-gram
  drafting, compatibility mode, the 114 GiB GTT memory model, client contract
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (replaced by
  [[frank-large-kairic-edge-qwen38-27b]]; same-day swap): the unsloth
  llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (a 125B MoE / 6B active) on the
  gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the
  explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
  [[frank-large-unsloth-qwen38-flash-next]]; rollback target, 2026-09-19 →
  09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving
  jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
  262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
  gate, client contract
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
  [[frank-large-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
  peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
  flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
  conversations over one 400,384-position pool, deployed flags, memory model,
  client contract
- [[frank-large-ik-uncensored]] — the RETIRED 2026-09-16 frank-large runbook
  and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
  mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
  Strix Halo hybrid — its measured envelope is why the fleet went back
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
  `amd_iommu=off` since 2026-09-11 (13-16 % prefill), what it cost
  (the NPU endpoint), and how to revert
- [[kingjones-strix-rocmfp4]] — the PRIOR frank-large runbook
  (2026-08-30 → 2026-09-11) and the rollback target:
  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
  pinned kingjones30/ROCmFPX build
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the ACTIVE
  frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
  with MTP back on since 2026-10-06 evening; renamed from frank-small and
  reshaped to TWO 114,688-token conversations at Frank's 112k cap with
  the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA
  sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
  (the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free
  at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision
  through the publisher's own projector, ~52–58 t/s recipe decode with MTP
- [[frank-secondary-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
  superseded by swift15): llama.cpp master serving
  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
  200,704 tokens each with fully unquantized f16 KV and mmproj vision;
  94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M —
  weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
  2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
  images answer "video")
- [[frank-secondary-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
  target, payload on disk; owns the measured boot-pair table): same runtime
  and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
- [[frank-secondary-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
  Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
  [[frank-secondary-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
  for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
  SGLang in a self-contained python3.12 venv serving
  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
  2 x 262,144 slots, bf16 mamba states, eager prefill
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
  Dankclaw's RTX 3090, port 7771
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
  Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
  token pool, 2 concurrent full-context sessions, native text+image+audio
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
  the prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
  Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-secondary (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large)
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
  8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize
  ordering that drains the old file into fresh swap instead of into a host
  whose `MemAvailable` lies by the size of the model, and the swappiness caveat
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
  Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
  retirement record, including the IOMMU/SVA err=-19 root cause and
  where every dependent route moved

### Benchmarks and Sizing

- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-secondary-benchmark]] — the small-model benchmark suite
- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x
  R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the
  logical batch buys nothing measurable, decode never moves, and the ubatch's
  uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved
  the tensor split from 36:29 to 35:30
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot
  do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that
  decides a split, the one flag (`-ot`) that kills per-tensor placement across
  devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor
  CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the
  first round of measurements say otherwise

### Memory and Hardware

- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU

### Retired Topology and History

- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)

### vLLM Reference (historical)

- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[vllm_research_findings]] — research findings from the vLLM era

### Pipeline Parallelism (archive)

- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
- [[Expected Performance]] — expected PP throughput
- [[llama.cpp RPC]] — the RPC transport between head and worker
- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
- [[Sample Script Set]] — reference scripts for the PP pair
- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation

## Update Log

- [[log]] — chronological record of knowledge-base changes

## See Also

- [AGENTS.md](../AGENTS.md) — repository rules
- [serverAppConfigs-docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
- [dankware Configuration Knowledgebase](~/.pi/agent/docs/index.md) — the "root" level system knowlegebase