docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM - Restructure log.md to OKF (## YYYY-MM-DD, newest-first) - Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change - Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe) - Write active runbook bodies from ground truth (frank-large, frank-small) - Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026) - Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs - All 336 tests pass; 136 pages, 0 unresolved wikilinks
2026-09-22 02:09:41 -07:00
parent 24f7a7772d
commit 93d1181fee
103 changed files with 2047 additions and 4329 deletions
@@ -62,13 +62,14 @@ service on the other host.
 |---|---|---|
 | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
 | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
-| `frank-large` | systemd + `systemctl` | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
-| `frank-medium` | systemd + `systemctl` | Retires the legacy frank-small unit on first deploy, then relinks and restarts only the inference service; waits up to 20 minutes for `/health`. |
+| `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
+| `frank-small` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
+| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
 | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |

 The shared chat template maps to the inference service on the selected target.
-The inactive `frank-npu` unit is relink-only when explicitly selected;
-it is not restarted by an unrelated large-model deployment.
+The `frank-npu` endpoint is retired (2026-09-11) and has no deploy handler;
+an unrelated large-model deployment never touches it.

 ## Caddy safety

@@ -1,3 +1,13 @@
+---
+type: reference
+description: "`tests/e2e/` drives the **live fleet** through a real browser: fresh contexts"
+tags:
+  - architecture
+  - e2e
+  - homepage
+timestamp: 2026-09-01
+---
+
 # E2E Suite — homepage + guard chain (Playwright)

 `tests/e2e/` drives the **live fleet** through a real browser: fresh contexts
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "One Caddy fronts both hosts."
+tags:
+  - architecture
+  - network
+timestamp: 2026-09-02
+---
+
 # Network Architecture

 One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) and terminates TLS for every service on the LAN. Pi-hole on Dankclaw answers the service names for LAN clients; Porkbun answers them for the WAN. The G36 router firewall governs external access. SearXNG outbound traffic exits through the Gluetun VPN container.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "All persistent data uses **bind mounts**: a host directory mapped directly into the container."
+tags:
+  - architecture
+  - storage
+timestamp: 2026-09-01
+---
+
 # Storage Policy

 All persistent data uses **bind mounts**: a host directory mapped directly into the container. No named volumes. This matches the preference of this repo and the existing Gitea stack on Dankclaw.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This repository tests its configuration at defined seams."
+tags:
+  - architecture
+  - testing
+timestamp: 2026-09-01
+---
+
 # Testing Policy

 This repository tests its configuration at defined seams. The tests run with `bats`; run them with `./tests/run.sh` before every push.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "ComfyUI is a node-based AI image generation engine."
+tags:
+  - dankclaw
+  - comfyui
+timestamp: 2026-09-16
+---
+
 # ComfyUI

 ComfyUI is a node-based AI image generation engine. Workflows are built from nodes that load models, sample latents, and post-process images; everything is scriptable through a web UI and an API.
@@ -13,6 +22,7 @@ https://github.com/Comfy-Org/ComfyUI
 |------|-------|
 | Image | `comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui` |
 | Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` |
+| Upstream latest | 0.37.0 (2026-09-21) — ships **native Qwen-Image 2.1** support (CORE-423, `comfy/ldm/qwen_image21/`). The image's custom GGUF patches may be unnecessary on the next rebuild; check before re-applying. |
 | Port | 8188 (host) → 8188 (container) |
 | Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag. |
 | Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` |
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Gluetun is a lightweight VPN client container."
+tags:
+  - dankclaw
+  - gluetun
+timestamp: 2026-08-28
+---
+
 # Gluetun

 Gluetun is a lightweight VPN client container. It connects to a VPN provider (WireGuard or OpenVPN) and shares that tunnel with other containers that attach to its network namespace.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "The Hermes web dashboard is a browser UI for managing the agent."
+tags:
+  - dankclaw
+  - hermes
+timestamp: 2026-09-16
+---
+
 # Hermes-Dashboard

 The Hermes web dashboard is a browser UI for managing the agent. It shows config, API keys, and sessions. It is a management tool. It is not a chat window. Chat happens over Signal or the CLI. See [[Hermes]].
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Hermes is the self-improving AI agent from Nous Research."
+tags:
+  - dankclaw
+  - hermes
+timestamp: 2026-09-15
+---
+
 # Hermes

 Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
@@ -20,6 +29,7 @@ is not durable. The tracked, sanitized configuration baseline is
 | Item | Value |
 |------|-------|
 | Image | built from `NousResearch/hermes-agent` (official Dockerfile in repo) |
+| Upstream latest | agent v0.21.4 / CalVer `v2026.9.21` (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not `:latest`. |
 | License | MIT |
 | Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
 | Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "OpenWebUI is a self-hosted chat interface for LLMs."
+tags:
+  - dankclaw
+  - openwebui
+timestamp: 2026-09-21
+---
+
 # OpenWebUI

 OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compatible endpoints (vLLM, llama.cpp, Ollama) and adds RAG, plugins, tool calling, and multi-user support.
@@ -12,6 +21,7 @@ https://github.com/open-webui/open-webui
 | Item | Value |
 |------|-------|
 | Image | `ghcr.io/open-webui/open-webui:main` (verified from the live quadlet unit) |
+| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
 | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
 | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
 | Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "PostgreSQL is the database backend for Gitea."
+tags:
+  - dankclaw
+  - postgres
+timestamp: 2026-09-03
+---
+
 # Postgres

 PostgreSQL is the database backend for Gitea. It runs on the same podman network as Gitea and holds the repository metadata, users, and issue data.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "How Google and other upstream search engines get blocked behind the Gluetun"
+tags:
+  - dankclaw
+  - searxng
+timestamp: 2026-08-28
+---
+
 # SearXNG Providers

 How Google and other upstream search engines get blocked behind the Gluetun
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "SearXNG is a privacy-respecting metasearch engine."
+tags:
+  - dankclaw
+  - searxng
+timestamp: 2026-09-03
+---
+
 # SearXNG

 SearXNG is a privacy-respecting metasearch engine. It aggregates results from many upstream search engines into one result set and serves both an HTML UI and a JSON API for agents.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line)."
+tags:
+  - dankclaw
+  - seerr
+timestamp: 2026-09-02
+---
+
 # Seerr

 Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line). Frank's household uses it to request movies/TV/music; Seerr routes approvals to Sonarr/Radarr/Lidarr/Readarr on FrankBook-Pro and pushes downloads to qBittorrent.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Valkey is the cache and limiter store for SearXNG."
+tags:
+  - dankclaw
+  - valkey
+timestamp: 2026-08-28
+---
+
 # Valkey

 Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment.
@@ -1,189 +0,0 @@
---
-type: reference
-title: *arr Stack
-description: The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every container on the gluetun WireGuard netns (all egress VPN): topology, wiring, auth posture, and gotchas.
-tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
-timestamp: 2026-09-04T00:00:00Z
---
-
-# *arr Stack
-
-The *arr suite is a collection of automated media management tools for Usenet
-and BitTorrent. It runs in exactly one place: **FrankBook-Pro** — the Mac mini
-that fronts this household's downloads — as quadlet units inside its podman
-machine, with every container (the five arrs AND qBittorrent) sharing the
-[[Gluetun]] WireGuard tunnel's network namespace so **all egress exits
-through the VPN**. A short-lived Dankserver compose copy was removed on
-2026-09-04 (see below). The stack provides automatic downloading and library
-management that Copyparty alone does not cover: quality profile enforcement,
-series/movie discovery, metadata enrichment, and automated file organization.
-
-The core chain is: **Prowlarr** (indexer API aggregator) →
-**Sonarr/Radarr/Lidarr/Readarr** (media-specific managers) → **qBittorrent**
-(download client, sharing the [[Gluetun]] WireGuard tunnel netns) →
-**Jellyfin** (media server scan → auto-refresh).
-
-## Service Overview
-
-| Service | Role | Image | Port | Min RAM |
-|---------|------|-------|------|---------|
-| Prowlarr | Indexer management | `lscr.io/linuxserver/prowlarr:latest` | 9696 | 256 MB |
-| Radarr | Movie management | `lscr.io/linuxserver/radarr:latest` | 7878 | 512 MB |
-| Sonarr | TV show management | `lscr.io/linuxserver/sonarr:latest` | 8989 | 512 MB |
-| Lidarr | Music management | `lscr.io/linuxserver/lidarr:latest` | 8686 | 256 MB |
-| Readarr | Ebook/audiobook management | `lscr.io/linuxserver/readarr:amd64-1.0.980` (pinned — LSIO ships no untagged `latest` for readarr) | 8787 | 256 MB |
-| qBittorrent | Torrent download client | `lscr.io/linuxserver/qbittorrent:4.6.7` (pinned; see [[qBittorrent]]) | 8080 (published by the gluetun unit) | 512 MB |
-
-Jackett and Whisparr were evaluated as optional extras but never deployed
-(no unit, no vhost, no live container).
-
-## Deployment shape (current)
-
-Each service is a quadlet `.container` unit in this repo under
-`frankbook-pro/<svc>/quadlet/`, installed inside the podman machine at
-`/home/core/.config/containers/systemd/<svc>.service` by the frankbook-pro
-handler in `scripts/deploy-remote.sh`. Every unit runs with
-`PUID=0 PGID=0 TZ=America/Los_Angeles UMASK_SET=022`, mounts its config root at
-`/Volumes/COMPLETE/<svc>-data:/config`, publishes its API port on the host, and
-runs `Restart=always` with `TimeoutStartSec=900`.
-
-qBittorrent is special: it shares the [[Gluetun]] container's network namespace
-(`Network=gluetun.container`) so torrent egress never touches the residential
-IP. A netns-sharer publishes no ports of its own — qB's WebUI (8080) and
-BitTorrent port (6881 TCP+UDP) are published on the gluetun unit instead.
-
-## Dankserver compose copy (removed 2026-09-04)
-
-A copy of the stack lived in `dankserver/compose.yaml` (`/opt/<svc>/config`
-data roots) from 2026-08-20 to 2026-09-04 and never ran healthy: the config
-dirs were created root-owned (`root:root 755`), so every app died at logger
-init (`UnauthorizedAccessException: Access to the path '/config/Sentry' is
-denied`) before binding any port — which read as dead ports, not a boot
-crash, until `podman logs <svc>` was checked. After a one-day revival it was
-removed by operator decision: **FrankBook-Pro is the sole *arr deployment.**
-Config roots moved to `~/.AGENTTRASH/20260904-dankserver-arr-teardown/` on
-Dankserver (recoverable until purged).
-
-### Rootless identity model on Dankserver (verified, keep for reference)
-
- On Dankserver's podman 5.7.0 rootless, a container's default user namespace
-  maps **container root (PUID=0) → host uid 1000 (dank)** and container uid
-  *N* → host `100000 + N - 1`. Verified via `/proc/<pid>/uid_map`
-  (`0 1000 1` / `1 100000 65536`) across plex, tautulli, jellyfin,
-  calibre-web, copyparty, romm. This is why PUID=0 services write as `dank`
-  and why data roots are owned `1000:1000` on that host.
- **Never add `userns_mode: host` to a podman-compose service.**
-  podman-compose 1.5.0 runs every compose service inside a pod, and podman
-  rejects the combination (`--userns and --pod cannot be set together`) —
-  the stack then fails to recreate with no recourse. The default mapping
-  already gives the fleet identity; PUID=0 + `1000:1000`-owned data roots is
-  the whole recipe. (The compose file briefly carried `userns_mode: host` on
-  all six arr services on 2026-09-04 — reverted in `e454159`, same day.)
- qBittorrent's first boot (PUID=0, empty config) prints a one-time WebUI
-  admin password in `podman logs qbittorrent` — persist it via the PBKDF2
-  recipe in [[qBittorrent]] before restarting the container.
-
-## Access / auth posture
-
-No *arr app sits behind the Anubis guard chain — the stack is exempt by
-construction (its own auth), with an exemption comment in the Caddyfile per
-vhost. Five TLS vhosts (`<svc>.franksdank.cloud`) terminate on Caddy on
-Dankclaw and route to FrankBook-Pro's published ports
-(`192.168.0.112:<port>`, all exposed by the gluetun unit since the netns
-consolidation); the same ports are reachable directly on the LAN. Sonarr is
-the exception: internal-only by construction — no vhost, no public DNS record;
-admins reach it at `http://192.168.0.112:8989` from the LAN only (retired
-2026-09-04; users request TV through [[Seerr]] instead).
-
-Auth-config history (verified live 2026-08-29): all five arrs shipped with
-`AuthenticationMethod=None` **plus** `AuthenticationRequired=Enabled`. On the
-v1 (.NET) apps that combo is self-contradictory — UI Save is permanently
-disabled and any settings save throws `DryIoc.ScopedItemException` inside
-`AuthorizationMiddleware` (500 sprawl on every page after restart). The valid
-representation of "no auth" in v1 is **absence of the `<AuthenticationRequired>`
-key**, not the literal string `Disabled` — readarr's config (key absent) booted
-clean while its three siblings (key=`Disabled`) 500-sprawled, which is how it
-was caught. Fix applied live: key deleted on sonarr/radarr/lidarr/prowlarr,
-containers restarted, all five verified serving. Posture = zero-auth arrs on
-LAN and WAN; the Anubis wall guards homepage/search/hermes/prometheus/romm/files
-but **not** the arrs — first-account-wins is the current race condition if that
-ever flips to `External` auth (needs an IdP first; auth method choice remains an
-operator decision). Sonarr is exempt from even that exposure since 2026-09-04:
-with no vhost and no public DNS record, its zero-auth UI has no WAN path at all.
-
-## Chain wiring (verified live 2026-08-30)
-
-The download chain is wired and test-first verified: **indexers → Prowlarr →
-per-arr qBittorrent clients (category = app name) → import**.
-
- qB webUI auth is ON (`admin` + generated password; verify before writing it
-  into any config — it was generated at first boot, not a default).
- Per-arr QBittorrent download clients exist on sonarr/radarr/lidarr
-  (enabled) and readarr (**parked**: its qB proxy fails auth live even with
-  correct creds — known defect of the abandoned upstream; retest needed
-  after the 2026-09-04 netns move). All four clients were re-pointed to
-  **loopback** (`127.0.0.1:8080`, same netns) on 2026-09-04 when the arrs
-  joined gluetun's netns — the old `10.88.0.1:8080` gateway hack routes
-  into the VPN tunnel (AllowedIPs=0.0.0.0/0) and is dead from inside the
-  netns. Verified live: `podman exec sonarr curl` reaches qB with 200, and
-  a real qB login from inside the netns returns 200.
- Prowlarr runs v2.5.2 where apps are first-class rows (`Applications`
-  table): four Application rows (sonarr/radarr/lidarr/readarr) carried each
-  arr's real baseUrl+apiKey with `SyncLevel=FullSync` as of 2026-08-30.
-  **Status 2026-09-04: the table is EMPTY again** (live API check — 8 rows,
-  all Cardigann/Torznab, zero Application rows; lost with the dankserver
-  config migration). Prowlarr sync is broken until the four rows are
-  rebuilt with baseUrl `127.0.0.1:<arr port>` (same netns).
- Indexers are **test-before-enable**: EZTV and 1337x stay disabled with
-  reason "blocked by CloudFlare Protection" (this host's egress ASN was
-  CF-blocked at those trackers). **2026-09-04: arr egress moved onto the
-  ProtonVPN tunnel** — the parked reason may no longer hold; retest before
-  enabling. Nyaa.si, sukebei, RuTor, Byrutor, YTS passed live tests and
-  are enabled. A failed test = parked with reason, never enabled blind.
- Prowlarr v2 sync model: indexers stay in Prowlarr; arrs search *through* it
-  and hand off to their own qB download client (category routing). Mappings
-  materialize at grab time — zero mappings right after wiring is expected,
-  not a fault.
-
-## Disk layout (current)
-
-```
-/Volumes/COMPLETE            <- completed-media volume (Mac volume)
-├── sonarr-data:/config      <- per-app config roots, one per unit
-├── radarr-data:/config
-├── lidarr-data:/config
-├── readarr-data:/config
-├── qbittorrent-data:/config
-└── <movie/tv/music/books roots>   <- arr import roots live here
-/Volumes/INCOMPLETE          <- qB in-flight downloads (same path inside qB)
-```
-
-qB mounts both volumes at their identical host paths
-(`Volume=/Volumes/COMPLETE:/Volumes/COMPLETE`,
-`Volume=/Volumes/INCOMPLETE:/Volumes/INCOMPLETE`), so a category's download path
-is the same string on every container that touches it — no remapping between
-download and import.
-
-## Gotchas
-
- **Timezone:** every unit runs `TZ=America/Los_Angeles`; Sonarr's scheduler is timezone-sensitive — keep them uniform.
- **File permissions:** `UMASK_SET=022` on qBittorrent; arr apps run PUID=0 so they can read/move qB downloads.
- **Network conflicts:** *arr apps use many ports. On the machine VM, only the API ports (8989/7878/8686/8787/9696) and gluetun's published 8080 + 6881 are host-published; netns-sharers (qB) publish none of their own.
- **Prowlarr API key rotation:** If rotated, update it in every arr app (`Applications` table rows carry the real keys).
- **qBittorrent WebUI password:** generated at first boot, not a default — persist it via the PBKDF2 conf recipe in [[qBittorrent]] before any restart.
- **Readarr stability:** abandoned upstream (retired May 2026); its qB download client is parked pending a successor image. Do not rely on it for mission-critical collections.
-
-## Best Practices
-
-1. Use tags to route content to specific download clients or categories.
-2. Enable the Blocklist in arr apps to prevent re-downloading failed torrents.
-3. Set download limits in qBittorrent to avoid bandwidth saturation.
-4. Back up arr configs regularly (they are small SQLite DBs + JSON) — each lives at `/Volumes/COMPLETE/<svc>-data/config`.
-5. Monitor Prowlarr indexer health — broken indexers hide new content.
-6. Start with default quality profiles. Add custom formats gradually.
-
-## See Also
-
- [[Sonarr]] · [[Radarr]] · [[Lidarr]] · [[Readarr]] · [[Prowlarr]] · [[qBittorrent]] — per-service pages
- [[Jellyfin]] — media server that consumes *arr downloads
- [[Copyparty]] — alternative file viewer
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database."
+tags:
+  - dankserver
+  - calibre
+timestamp: 2026-09-03
+---
+
 # Calibre-Web

 Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. It serves as the web frontend for the Calibre e-book manager.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Calibre is the e-book library manager that backs Calibre-Web."
+tags:
+  - dankserver
+  - calibre
+timestamp: 2026-09-21
+---
+
 # Calibre

 Calibre is the e-book library manager that backs Calibre-Web. It manages metadata, formats, and conversions for the library that Calibre-Web serves.
@@ -18,10 +27,10 @@ The actual library files (`.epub`, `.mobi`, `.pdf`) and the Calibre database (`.

 ## Current Version

-Calibre 9.2.1 is the version on the live host (9.9.0 is the upstream latest). Major features include:
+Calibre 9.15.0 is the version on the live host (the upstream latest as of 2026-09-22). Major features include:
 - Bookshelf view
 - Opt-in AI features
- Content server security fix (CVE-2025-7404 equivalent): a bug since v6.16.0 allows any user with server access to read non-Calibre files on the host. Upgrade to 9.9.0+ if serving content to untrusted users.
+- Content-server path-traversal fix: a bug since v6.16.0 allowed any user with server access to read non-Calibre files on the host. (The CVE-2025-7404 advisory is tracked against Calibre-Web, see [[Calibre-Web]].) Upgrade to 9.15.0+ if serving content to untrusted users.

 ## Library Directory

@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Copyparty is a lightweight file server and media streamer that runs as a single Python binary."
+tags:
+  - dankserver
+  - copyparty
+timestamp: 2026-09-03
+---
+
 # Copyparty

 Copyparty is a lightweight file server and media streamer that runs as a single Python binary. It handles large payloads, media streaming, resumable uploads, directory browsing, and WebDAV from one process.
@@ -11,7 +20,7 @@ LAN — **Docs:** https://github.com/9001/copyparty

 | Item | Value |
 |------|-------|
-| Running image | `ghcr.io/9001/copyparty-ac:latest` (ac = full edition; the bare `copyparty:latest` tag does not exist — ghcr answers 403 on the token request for nonexistent repos). Version at research time (2026-08): v1.20.21 |
+| Running image | `ghcr.io/9001/copyparty-ac:latest` (ac = full edition; the bare `copyparty:latest` tag does not exist — ghcr answers 403 on the token request for nonexistent repos). Upstream latest v1.20.24 (2026-09-19); v1.20.21 at research time (2026-08) |
 | Port | 3923 (host) → 3923 (container) |
 | Volume | bind mount the shared root directory |
 | Auth | Per-directory `.cpy` config files, HTTP basic auth, `dirkeys`, `filekeys` |
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankserver (192.168.0.235) runs the media and library stack."
+tags:
+  - dankserver
+timestamp: 2026-09-03
+---
+
 # Dankserver Stack

 Dankserver (192.168.0.235) runs the media and library stack. Public access
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at"
+tags:
+  - dankserver
+timestamp: 2026-08-28
+---
+
 # Entropy Volume

 The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Plex Media Server is a **containerized media server on Dankserver**: a podman"
+tags:
+  - dankserver
+  - plex
+timestamp: 2026-09-15
+---
+
 # Plex

 Plex Media Server is a **containerized media server on Dankserver**: a podman
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver."
+tags:
+  - dankserver
+  - romm
+timestamp: 2026-08-28
+---
+
 # RomM Scraping & Asset Management

 This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "RomM (ROM Manager) is a self-hosted ROM library manager and player."
+tags:
+  - dankserver
+  - romm
+timestamp: 2026-09-03
+---
+
 # RomM

 RomM (ROM Manager) is a self-hosted ROM library manager and player. It scans a ROM collection, enriches it with metadata and cover art from multiple providers, and serves it through a clean web interface. RomM also supports browser-based emulation via EmulatorJS and native emulator streaming.
@@ -208,6 +217,7 @@ healthcheck:

 - **First login overwrites env credentials.** The `ROMM_PASSWORD` env var sets the initial password. After first login, the admin password is controlled in the UI. The env var only applies on first boot.
 - **MariaDB and Redis are required.** RomM 5.2.0 no longer supports SQLite-only deployments. Both MariaDB and Redis must be running.
+- **5.3.0 breaking change (2026-09-21).** `filesystem.roms_folder` / `firmware_folder` are removed; the scan root must be expressed via `filesystem.structure` in `config.yml` (see the Config File section above). New env vars: `SCAN_WORKERS`, `WEB_SERVER_CONCURRENCY`, `DB_POOL_RECYCLE_SECONDS`. The rolling `:latest` tag will pick this up on the next pull — migrate `config.yml` before redeploying, or the scan root silently goes empty.
 - **Reverse proxy cache stale index.html.** Purge Caddy proxy cache after RomM version bumps to avoid serving a stale frontend after an upgrade.
 - **RAHasher build time.** Enabling RetroAchievements hashing increases the first image build. Skip unless you need achievement data.
 - **MobyGames is paid.** The hobbyist API has strict rate limits. Screenscraper is the recommended free alternative.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Tautulli is the statistics, monitoring, and notification companion for"
+tags:
+  - dankserver
+  - tautulli
+timestamp: 2026-09-01
+---
+
 # Tautulli

 Tautulli is the statistics, monitoring, and notification companion for
@@ -0,0 +1,7 @@
+---
+type: reference
+title: "*arr Stack"
+description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every container on the gluetun WireGuard netns (all egress VPN): topology, wiring, auth posture, and gotchas."
+tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
+timestamp: 2026-09-04T00:00:00Z
+---
@@ -0,0 +1,16 @@
+# FrankBook-Pro
+
+The macOS mini (192.168.0.112) that hosts the *arr download stack and
+qBittorrent as quadlets inside its podman machine. All egress rides the
+Gluetun WireGuard netns. The host overview lives at [[FrankBook-Pro]]
+(Infrastructure); this section holds the per-service pages.
+
+## *arr Stack
+
+- [[ARR-Stack]] — the full *arr stack: indexers, managers, and the download client
+- [[sonarr]] — TV show management
+- [[radarr]] — movie management
+- [[lidarr]] — music management
+- [[readarr]] — ebook/audiobook management
+- [[prowlarr]] — indexer management
+- [[qbittorrent]] — torrent download client
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through..."
+tags:
+  - infrastructure
+timestamp: 2026-09-17
+---
+
 # Alertmanager

 Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Alloy is Grafana's log and telemetry shipper."
+tags:
+  - infrastructure
+timestamp: 2026-09-11
+---
+
 # Alloy

 Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). It forwards container and journal logs to Loki.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Anubis is the bot-challenge proxy that fronts the guarded public vhosts on"
+tags:
+  - infrastructure
+  - anubis
+timestamp: 2026-09-01
+---
+
 # Anubis — bot challenge proxy

 Anubis is the bot-challenge proxy that fronts the guarded public vhosts on
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the"
+tags:
+  - infrastructure
+  - authentik
+timestamp: 2026-09-16
+---
+
 # Authentik — the homelab's SSO / OIDC provider

 Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Caddy is the single reverse proxy for both hosts."
+tags:
+  - infrastructure
+  - caddy
+timestamp: 2026-08-29
+---
+
 # Caddy

 Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Grafana visualizes the metrics that Prometheus collects from every service."
+tags:
+  - infrastructure
+  - grafana
+timestamp: 2026-09-11
+---
+
 # Grafana

 Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "The decision log for the homepage design."
+tags:
+  - infrastructure
+  - homepage
+timestamp: 2026-08-28
+---
+
 # Homepage Design Grilling

 The decision log for the homepage design. All questions are settled (2026-08-23). This page records the final decisions and the places where the operator's answer changed the initial recommendation.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation."
+tags:
+  - infrastructure
+  - homepage
+timestamp: 2026-09-21
+---
+
 # Homepage

 A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "The homelab gives every service a name under `franksdank.cloud`."
+tags:
+  - infrastructure
+  - dns
+timestamp: 2026-09-02
+---
+
 # LAN DNS

 The homelab gives every service a name under `franksdank.cloud`. Users type a name; they never type a port number. This page covers how names reach users: the Pi-hole server, the G36 router hand-off, and the split between LAN and WAN resolution.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Loki is the log aggregation backend of the observability stack."
+tags:
+  - infrastructure
+  - loki
+timestamp: 2026-09-11
+---
+
 # Loki

 Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs. It replaces the need for a full ELK stack.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus."
+tags:
+  - infrastructure
+timestamp: 2026-09-11
+---
+
 # Node-Exporter

 Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container.
@@ -1,151 +1,7 @@
 ---
 type: reference
-title: Observability on Dankclaw
-description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured.
+title: "Observability on Dankclaw"
+description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured."
 tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
 timestamp: 2026-09-11
 ---
-
-# Observability
-
-The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards, node-exporter for host metrics, and Alertmanager for alert delivery — to Signal, through a host-side relay. This is a small homelab — no Kubernetes.
-
-## Stack
-
-```mermaid
-graph LR
-    S[Services] -->|/metrics| P[Prometheus]
-    H[Host] -->|":9100"| P
-    S -->|logs| A[Alloy]
-    J[journald] --> A
-    A --> L[Loki]
-    P --> G[Grafana]
-    L --> G
-    P -->|alerts| AM[Alertmanager]
-    AM -->|webhook| R[Signal relay]
-    R -->|signal-cli| SIG[Signal]
-
-| Component | Role | Image | Published |
-|-----------|------|-------|-----------|
-| [[Prometheus]] | metrics collection | `prom/prometheus` | `127.0.0.1:9090` |
-| [[Grafana]] | dashboards | `grafana/grafana` | `127.0.0.1:3002` |
-| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
-| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
-| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
-| [[Alertmanager]] | alert grouping & delivery | `prom/alertmanager` | `127.0.0.1:9093` |
-
-## Current state: installed, healthy, and mostly unconfigured
-
-Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
-
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
- **2026-09-20: the rule file now spans two metric namespaces, and the dashboard was re-cut the same day.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — six alerts and three recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. One semantics trap, measured 2026-09-20: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP line), so the llama.cpp reuse rule divides by `cached + prompt` — the naive `cached/prompt` ratio reads 77 on a fully-warm endpoint and can never show a collapse. The provisioned dashboard's panels now read both namespaces (the endpoint picker selects between them); the KV-pool-fill panel retired 2026-09-16 returned with `vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every expression in the rules and the dashboard was validated against the live endpoints on 2026-09-20.
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
-
-The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, nine rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
-
-## Deployment
-
-Seven units now, across two hosts. Dankclaw runs six (Prometheus, Grafana, Loki,
-Alloy, node-exporter, Alertmanager) from `dankclaw/observability/quadlet/`; **Dankserver runs
-its own node-exporter** from `dankserver/observability/quadlet/`, added
-2026-09-16 after the same reboot argument took it down a second time — it had
-been a compose-only service, died `Exited (143)` at a reboot, and left
-Prometheus' `node-dankserver` job scraping nothing for a day and a half (its
-compose form also bound the wildcard address). The deployer installs from
-`"$TARGET/observability/quadlet"`, so each host owns its own units.
-
-The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.**
-
-```bash
-./scripts/deploy.sh dankclaw --service loki          # or prometheus|grafana|alloy|node-exporter|alertmanager
-./scripts/deploy.sh dankserver --service node-exporter
-```
-
-`deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first.
-
-Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podman ps` rather than merely `Up`. See [[Loki]] for the one unit that cannot declare it inline.
-
-## Exposure
-
-The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:
-
- **Prometheus, Grafana, Loki, Alloy and Alertmanager publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
- **node-exporter is the one deliberate LAN listener, and there are two of them now — one per host.** Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (`127.0.0.1:9100`) for host-local probes and its own LAN address (`192.168.0.143:9100` on Dankclaw, `192.168.0.235:9100` on Dankserver) for Prometheus. `ss -ltn` on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.
-
-## Scrape surface
-
-`shared/prometheus/prometheus.yml` is the source of truth (installed to `/opt/prometheus/config/` on every deploy). It scrapes:
-
-| Job | Target | Notes |
-|-----|--------|-------|
-| `prometheus` | `localhost:9090` | itself |
-| `node-dankclaw`, `node-dankserver` | `192.168.0.143:9100`, `192.168.0.235:9100` | one node-exporter per host |
-| `gitea` | `192.168.0.143:3000` | published LAN port (Gitea is on an isolated network) |
-| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
-| `comfyui` | `comfyui:8188` | UI probe, no native exporter |
-| `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
-| `frank-medium` | `192.168.0.235:7770` | radiance/vLLM on Dankserver — `vllm:` counters: KV fill, queue depth, preemptions, prefix-cache reuse, and the `spec_decode_*` trio |
-| `frank-small` | `host.containers.internal:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
-| `frank-large` | `host.containers.internal:7770` | CIRU llama.cpp on Dankclaw's iGPU — 19 `llamacpp:` families |
-
-The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
-frank-medium by hand, and re-cut the same day when Dankclaw split into two
-endpoints. frank-medium has since changed runtime twice (vLLM → mainline
-llama.cpp → radiance/vLLM on 2026-09-20), and the metric family it exports
-changed with it: `llamacpp:*` while it ran llama.cpp, `vllm:*` now.
-
-**Both Dankclaw targets are addressed as `host.containers.internal`
-(169.254.1.2), never the LAN IP.** frank-large and frank-small are *host
-processes* on Dankclaw, and netavark DNATs only *published* container ports —
-from the Prometheus container `192.168.0.143:7770/:7771` is refused even
-though it answers from every other vantage on the LAN. Both targets were red
-by construction until a live `/targets` audit caught it on 2026-09-20.
-`frank-medium` keeps its LAN address: it is a rootless Quadlet on another host
-that publishes 7770, so its published-port DNAT does apply.
-
-**`frank-large` IS scraped as of 2026-09-20.** The earlier claim that it had
-"no Prometheus text endpoint at all" was true of the halogen engine it ran
-then; the CIRU llama.cpp build that replaced it exports the full
-`llamacpp:*` family, which is what makes the target addressable at all. Its
-liveness is additionally on the homepage (a loopback probe) and in the
-engine's own boot ledger.
-
-For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
-
-**Four targets read "down" — three jobs, and every one of them is structural**
-(found 2026-09-16 while auditing why the target list was not green):
-
-| job | error | what it means |
-|-----|-------|---------------|
-| `comfyui` | `unsupported Content-Type "text/html"` | the service is **up** (this is what it says since ComfyUI was fixed — before that it read `connection refused`); Prometheus refuses to parse an HTML page as metrics, so this job cannot succeed as configured |
-| `services` (SearXNG, OpenWebUI) | `unsupported Content-Type "text/html; charset=utf-8"` | same defect: "blackbox-style probes" that scrape a page instead of metrics |
-| `gitea` | `HTTP status 404 Not Found` | `/metrics` does not exist on that Gitea (metrics disabled); the service itself is fine |
-
-None of these is an outage, and none should be silenced by deleting the job —
-an absent scrape and a healthy one look identical. The fix the repository
-already names is a **blackbox exporter** in probe mode for the page-only
-targets, and either enabling Gitea's metrics or pointing the job at the right
-path. The LLM alert rules (`LlmEndpointDown`) are scoped to `frank-*`, so these
-probes generate no false pages.
-
-## Log aggregation choice
-
-Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
-
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
- No Elasticsearch memory cost, no Kubernetes required.
-
-Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
-
-## See Also
-
- [[Grafana]], [[Prometheus]], [[Loki]], [[Alloy]], [[Node-Exporter]] — the components
- [[Dankclaw Stack]] — the rest of the host
- [[Homepage]] — the user-facing status surface, including the five status cards
- [[Deployment Flow]] — the verify-before-declared-deployed contract
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Pi-hole is the LAN DNS server for the homelab."
+tags:
+  - infrastructure
+  - pi-hole
+timestamp: 2026-08-28
+---
+
 # Pi-hole

 Pi-hole is the LAN DNS server for the homelab. It runs as a quadlet container on Dankclaw (192.168.0.143). It answers the flat `franksdank.cloud` / `franksdank.casa` service names and filters ad traffic on the LAN. It retired the dnsmasq quadlet on 2026-08-24. See [[LAN-DNS]] for the name-resolution flow and the router hand-off.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists."
+tags:
+  - infrastructure
+  - prometheus
+timestamp: 2026-09-21
+---
+
 # Prometheus

 Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.
@@ -1,3 +1,13 @@
+---
+type: reference
+description: "The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /"
+tags:
+  - infrastructure
+  - authentik
+  - postgres
+timestamp: 2026-08-28
+---
+
 # authentik-postgres

 The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "The Authentik server — the core, the admin API, and the embedded Caddy outpost."
+tags:
+  - infrastructure
+  - authentik
+timestamp: 2026-09-16
+---
+
 # authentik-server

 The Authentik server — the core, the admin API, and the embedded Caddy outpost.
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "The Authentik worker — background tasks: event processing, provider"
+tags:
+  - infrastructure
+  - authentik
+timestamp: 2026-09-16
+---
+
 # authentik-worker

 The Authentik worker — background tasks: event processing, provider
@@ -1,3 +1,13 @@
+---
+type: reference
+description: "DFlash and DSpark are the two newest speculative decoding methods."
+tags:
+  - llm
+  - speculative
+  - dflash
+timestamp: 2026-09-03
+---
+
 # DFlash and DSpark Speculative Decoding

 ## Overview
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "This page collects concrete, sourced recommendations to improve Dankclaw"
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Dankclaw — Improvements Research

 This page collects concrete, sourced recommendations to improve Dankclaw
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Dankclaw System Specs

 ## Server Overview
@@ -23,7 +31,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Model | AMD Radeon 8060S (Strix Halo) |
 | Device ID | 0x1586 |
 | Architecture | gfx1151 (Strix Halo) |
-| GPU-Visible Memory | 64 GiB (rocm-smi; SVM-backed, see the memory model below) |
+| GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `gtt_total`; rocm-smi's smaller SVM carve-out is not the budget) |
 | Physical RAM | 64 GiB (61 GiB visible; upgraded from 32 GB, corrected 2026-08-17) |

 ### NPU
@@ -35,10 +43,10 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Device | /dev/accel/accel0 |
 | Driver | amdxdna 0.7 (in-tree, kernel 7.0.0-29-generic) |
 | Firmware | 1.1.2.65 |
-| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-08-17**) |
+| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
 | Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |

-The NPU is retired; the 52625 port and the `frank-small` name now serve the CIRU ROCmFPX llama.cpp runtime (see [[ciru-activefpx-promptforge]]).
+The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]).

 ### ROCm

@@ -49,14 +57,14 @@ The NPU is retired; the 52625 port and the `frank-small` name now serve the CIRU
 | ROCm Version | 7.2.4 system (/opt/rocm) + TheRock 7.15 SDK in the CIRU venv (used for the runtime) |
 | Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
 | GPU Arch | gfx1151 (Strix Halo) |
-| GPU Global Memory | 64 GiB (rocm-smi; SVM-backed) |
+| GPU Global Memory | ~114 GiB GTT aperture (the budget) |

 ### Memory

 | Property | Value |
 |----------|-------|
 | System RAM | 64 GiB (shared CPU + GPU; 61 GiB visible) |
-| GPU-Visible (ROCm) | 64 GiB (rocm-smi; SVM-backed) |
+| GPU-Visible (ROCm) | ~114 GiB GTT aperture (the budget) |
 | Swap | 8.0 GiB |
 | /dev/shm | ~15 GiB (half of RAM) |

@@ -89,12 +97,12 @@ cable between claw and server is required for 10G.
 | CIRU runtime | ciru-ai/ROCmFPX v2.3 (llama-server, ROCm 7.15 SDK) |
 | uv | 0.12.2 |

-## CIRU Runtime Relevance
+## Runtime Relevance

 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]].
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
+- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
+- The active service is `frank-large` on port 7770: stock llama.cpp (the unsloth b11030 prebuilt) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, q8_0 KV, DFlash2 drafter). See [[frank-large-unsloth-qwen38-27b-q8xl]].
+- The active runtime (the unsloth b11030 prebuilt) **bundles its own ROCm runtime** next to the binary (`RPATH=$ORIGIN/../lib`), so it has no host SDK gate — the system ROCm 7.2.4 is not on its path. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is no longer in the serving path.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 1.1 TB free disk space is ample for model storage.

@@ -1,9 +1,3 @@
-> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
-> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
-> [deployment.md](deployment.md) for the active configuration.
-> The `frank-small` name and port 52625 now belong to the vLLM
-> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
-
 ---
 type: research
 title: FastFlowLM NPU Inference
@@ -14,9 +8,14 @@ tags:
  - xdna2
  - dankclaw
  - research
-timestamp: 2026-08-16
+timestamp: 2026-09-03
 ---

+> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
+> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
+> The `frank-small` name and port 52625 then belonged to the vLLM
+> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
+
 # FastFlowLM NPU Inference

 FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing."
+tags:
+  - llm
+timestamp: 2026-08-28
+---
+
 # Hybrid CPU GPU Processing Dankserver

 ## Overview
@@ -1,228 +1,9 @@
 ---
 type: reference
-title: Mixed-Vendor Experiments
-description: Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s.
+title: "Mixed-Vendor Experiments"
+description: "Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research Brief: NVIDIA-Desktop + Strix-Halo Multi-Node LLM Inference
-
-## Summary
-
-Yes, mixed NVIDIA-ROCm multi-node LLM inference over RPC has been published and is **fully functional**. The closest real experiments to an RTX-desktop + Strix-Halo-NUC pairing are (1) a planned 4× RTX 3090 + Strix Halo 128GB via 25GbE (r/LocalLLaMA, Apr 2026), (2) a Tesla V100 + AMD R9700×2 via Vulkan layer-split on one box (grosan.co.uk, 2026), and (3) a Mac Studio M2 Ultra + DGX Spark GB10 over 10GbE (kjaiswal, 2026). Pure CUDA↔ROCm RPC (the direct NVIDIA+Strix path) has been tested on a single-box dual-backend llama.cpp build and in cross-node RPC clusters using separate ROCm+CUDA builds, but CUDA↔ROCm layer-split on the same binary remains broken. Vulkan is the universal backend that works across vendors.
-
-**Verdict:** The NVIDIA-desktop + Strix-Halo multi-node path via llama.cpp RPC is viable today. Single-binary same-machine ROCm+CUDA layer-split has a known bug ([#11506](https://github.com/ggml-org/llama.cpp/issues/11506)). Cross-node RPC works with separate ROCm and CUDA builds on each machine. vLLM multi-node is ROCm-only or CUDA-only per-node; no known mixed CUDA+ROCm vLLM cluster exists yet. Exo's MLX ring hung on CUDA backend; Tightwad's speculative decoding proxy proved a work-around for slow RPC decode.
-
---
-
-## Findings
-
-### 1. Published Experiments: NVIDIA + AMD Strix Halo Nodes
-
-**1a. Planned 4×RTX 3090 + Strix Halo 128GB cluster over 25GbE**
-User on r/LocalLLaMA (Apr 2026) is building a llama.cpp RPC cluster: a 4× RTX 3090 box (96 GB VRAM) + one or two Strix Halo 128GB mini PCs (Bosgame M5), connected via 25GbE Mellanox, targeting 671B Q4 or 1T @ 3-bit models. Post was seeking community validation before purchasing. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
-
-**1b. Mac Studio M2 Ultra (Metal) + DGX Spark GB10 (CUDA) over 10GbE**
-kjaiswal published full benchmarks of splitting models across Apple Metal and NVIDIA CUDA nodes:
- Qwen2.5-7B Q4_K_M (4.4 GB): Local Metal 92 tok/s vs RPC Metal+CUDA 53 tok/s decode, but **4.2× faster prefill** (318 vs 76 tok/s). The 7B fits both machines — RPC trades decode speed for prefill speed.
- Qwen2.5-72B Q4_K_M (44.2 GB): Local Metal 11 tok/s vs RPC Metal+CUDA 6 tok/s decode (~47% penalty). Split: 31 GB Metal + 14 GB CUDA.
- Projected for models needing both: MiniMax M2.5 (138 GB, ~5-8 tok/s), Qwen3-235B (132 GB, ~5-8 tok/s).
- **Conclusion:** RPC's value is capacity (100 GB+ models), not speed. Decode is 2× slower over RPC due to per-token network round-trips. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
-
-**1c. Tesla V100 + AMD R9700×2 via Vulkan layer-split (single box)**
-grosan.co.uk documented running Qwen3.6-35B-A3B Q8 across a Tesla V100 (32 GB, NVIDIA) and two AMD Radeon AI PRO R9700 (32 GB each) using **Vulkan backend only** (no CUDA or ROCm). Layer-split across three GPUs from two vendors. Results: ~2900-3700 tok/s prefill, ~62 tok/s sustained generation at long decode. Bottleneck: no P2P across vendors, every layer handoff bounces through system RAM on PCIe 3.0. [Source](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/)
-
-### 2. Strix Halo in Distributed Inference
-
-**2a. Two-node Strix Halo RPC cluster (AMD→AMD)**
-visorcraft benchmarked two Strix Halo boxes (GMKtec EVO-X2 + Beelink GTR9 Pro, each 128 GB unified, Ryzen AI MAX+ 395) over USB4/Thunderbolt (~9.4 Gbps):
- MiniMax-M2.5-REAP-139B-Q8: **15.35 tok/s** decode, RPC split 1.2/0.8
- Qwen3.5-397B-A17B-UD-Q4_K_XL: **11.76 tok/s** decode (llama-bench), **12.6 tok/s** (llama-server with `-dio`)
- Single-host single-node baseline: MiniMax-M2.5 Q3_K_M at 32.8 tok/s, Qwen3 30B-A3B at 86.1 tok/s
- Key caveat: RPC serving requires `-dio` flag on large models. [Source](https://github.com/visorcraft/strix-halo-llm-perf)
-
-**2b. AMD official: 4×Framework Desktop (Strix Halo) cluster**
-AMD published a walkthrough running Kimi K2.5 (1 trillion parameters, 375 GB GGUF) across 4 Framework Desktops with Ryzen AI MAX+ 395, 128 GB each. Interconnect: 5 Gbps Ethernet. Flash Attention enabled via rocWMMA. Benchmarks at 8192 sequence length: 8.30 tok/s decode with FA enabled vs 3.46 tok/s without (batch/ubatch=4096). Context: 32768 tokens. [Source](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html)
-
-**2c. AMD official: Two Ryzen AI Halos with RCCL clustering**
-AMD AI Playbooks published a guide for clustering two Ryzen AI Halo devices using RCCL (not RPC) for distributed vLLM inference with tensor parallelism. [Source](https://developer.amd.com/playbooks/clustering-rccl/)
-
-**2d. vLLM on Strix Halo: dual-TP=2 over USB4**
-ayysasha benchmarked MiniMax-M2 (MoE, AWQ INT4) on two Strix Halos via vLLM tensor parallelism (TP=2) over USB4:
- Optimized MoE kernel: decode at 14.84 tok/s (1k context), 9.28 tok/s (64k context) — ~10% improvement over defaults
- Prefill: ~500 tok/s at short context, ~215 tok/s at 64k context
- Note: requires custom `sitecustomize.py` shim for device name detection on gfx1151. [Source](https://github.com/ayysasha/Strix-halo-dual-optimized)
-
-**2e. Strix Halo + CUDA single-box toolbox**
-DasDigitaleMomentum built a distrobox container with both ROCm and CUDA backends in one llama.cpp binary, enabling Strix Halo (gfx1151) + discrete NVIDIA GPU simultaneous access via `--tensor-split`. Also supports dual-backend ROCm+Vulkan. [Source](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox)
-
-### 3. Heterogeneous Node Splitting: CUDA + ROCm RPC
-
-**3a. llama.cpp builds with both GGML_CUDA and GGML_HIP**
-Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506) reports that llama.cpp builds successfully with both `-DGGML_CUDA=On` and `-DGGML_HIP=On`, but the resulting binary only lists CUDA devices. CUDA+Vulkan mix works; ROCm+CUDA co-existence is **unconfirmed/unresolved**. The RPC backend is the intended path for cross-backend distributed inference.
-
-**3b. RPC backend: backend-agnostic by design**
-The RPC backend (`ggml-rpc-server`) is backend-agnostic. Each worker runs its own llama.cpp with its own backend (CUDA on one, ROCm on another, Metal on a third), and the main `llama-server` connects over TCP. From llama.cpp's perspective, each remote machine is just another device. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
-
-**3c. Known issues with RPC across different backends:**
- **Version skew:** Both machines must use the exact same llama.cpp build tag; mismatched builds hang at handshake.
- **Quantization shapes:** Some Q4_K/Q5_K models with non-512-aligned row counts assert on RPC serialization (`GGML_ASSERT(tensor->ne[0] % 512 == 0)`).
- **Network speed matters:** WiFi → 10× slowdown vs wired ethernet. Ethernet 1G works but is latency-bound for decode.
- **No authentication:** `rpc-server` bound to `0.0.0.0` exposes all memory to any network client.
- **Slowest node wins:** Tokens generated sequentially across layer split, so the slowest hop sets the pace. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
-
-**3d. Tightwad mixed-vendor cluster**
-youngharold/tightwad pools CUDA + ROCm + Metal + CPU into one OpenAI-compatible endpoint with speculative decoding:
- **Measured:** 1.86× speedup on Llama 3.3 70B across 4 GPUs over WiFi (3 tok/s raw RPC → 4.1 tok/s with spec decode)
- **Speculative decoding:** Qwen3-235B with Qwen3-1.7B draft model over RPC — overall acceptance rate 6.6%, highest in factual (14.5%) and reasoning (13.8%) categories. Spec decode made slow RPC models actually usable. [Source](https://github.com/youngharold/tightwad)
- Config example `mixed-vendor-pool.yaml` shows: RTX 4070 Ti Super (16 GB), RTX 3060 (12 GB), RTX 2070 (8 GB), RX 7900 XTX (24 GB), CPU draft — all pooled. [Source](https://github.com/youngharold/tightwad/blob/main/examples/mixed-vendor-pool.yaml)
-
-**3e. Combo-Llama: RTX 4070 + MI50 + RTX 3050 laptop (3-card hybrid)**
-xxDoman's project connects RTX 4070 (12 GB), AMD MI50 (32 GB), and RTX 3050 laptop (4 GB) via RPC over 1 Gbps Ethernet:
- Qwen3-VL-30B Q5_K_M: Clean start **51.49 tok/s** on Ethernet (WiFi was 44.53 tok/s)
- 32k context: **17.81 tok/s**, 64k context: **12.90 tok/s** (stable to 65536 tokens)
- MI50 at 89% GPU load, RTX 4070 at 12% — load imbalance typical. [Source](https://github.com/xxDoman/combo-llama-rpc-hybrid)
-
-### 4. Adjacent Projects
-
-**4a. Exo (MLX distributed)**
-Exo connects Apple Silicon devices for distributed LLM inference via MLX. Tried cross-vendor Metal+CUDA over 10GbE with DGX Spark + Mac Studio. **Result:** Failed — `mx.distributed.init(backend="ring")` hangs on CUDA backend (as of MLX 0.31.1). CUDA ring support from Apple is the blocker. Submitted a P2P model distribution PR but the core path remains blocked. [Source](https://dev.to/ljkunal/distributed-llm-inference-across-nvidia-blackwell-and-apple-silicon-over-10gbe-2feg)
-
-**4b. Petals**
-Distributed inference framework. Not yet tested on Strix Halo. Uses its own custom protocol over TCP. [General reference]
-
-**4c. llama.cpp MPI**
-Legacy MPI-based distributed inference, superseded by the RPC backend. RPC is now the standard multi-node path. [llama.cpp docs]
-
-**4d. Poor man's cluster / ethernet experiments**
-dev.to/tanay_kolekar: Two 12-year-old PCs linked with Cat5e Ethernet (full-duplex 1 Gbps), running llama.cpp RPC. Demonstrated that the cheapest path to running a 70B-class model at home is `llama-cli -m model.gguf --rpc <worker_ip>:50052`. [Source](https://dev.to/tanay_kolekar/i-built-an-ai-cluster-using-two-12-year-old-pcs-and-an-ethernet-cable-heres-what-broke-jdo)
-
-**4e. Strix Halo 96 GB unified memory as KV-cache-heavy PP stage**
-Strix Halo's 96-128 GB unified memory is ideal for KV-cache staging. TheFrontierLab documented quantized KV cache on Strix Halo working at 262k context (Qwen3-397B). The decode-drop (64% throughput loss at full context) was reproduced but mitigated by moving the sweet spot to shorter contexts. [Source](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/)
-
-**4f. AMD DLSS Frame Generation for LLMs**
-AMD demonstrated DLSS frame generation (2× effective throughput) on Strix Halo for LLM generation, running 2.4× faster generation via frame generation without accuracy loss. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1ot3lxv/i_tested_strix_halo_clustering_w_50gig_ib_to_see/)
-
-**4g. RDMA/RoCEv2 on llama.cpp RPC**
-PR #20590 added native RDMA (RoCEv2) transport to llama.cpp RPC. Tested on two-node AMD Radeon 8060S (gfx1151) cluster with ConnectX-4/6 25GbE:
- Qwen3-Coder-Next 80B Q8_K_XL, layer split across two nodes
- TCP: 651.48 pp2048 / 30.19 tg256
- RDMA: 678.42 pp2048 / 32.16 tg256 (+4.1% / +6.5% improvement) [Source](https://github.com/ggml-org/llama.cpp/pull/20590)
-
-### 5. Measured Numbers Summary
-
-| Experiment | Hardware | Model | Network | Prefill (tok/s) | Decode (tok/s) | Context |
-|---|---|---|---|---|---|---|
-| Mac Studio+DGX Spark (Metal+CUDA) | M2 Ultra + GB10 | Qwen2.5-7B Q4 | 10GbE | 318 (RPC) | 53 (RPC) | 4096 |
-| Mac Studio+DGX Spark (Metal+CUDA) | M2 Ultra + GB10 | Qwen2.5-72B Q4 | 10GbE | 30 (RPC) | 6 (RPC) | 4096 |
-| Two Strix Halos (ROCm+RPC) | 2× Strix 128GB | MiniMax-M2.5 Q8 | USB4 9.4Gbps | 332 | 15.35 | — |
-| Two Strix Halos (ROCm+RPC) | 2× Strix 128GB | Qwen3.5-397B Q4 | USB4 9.4Gbps | 148 | 11.76 | — |
-| AMD 4-node Strix (ROCm+RPC) | 4× Framework 128GB | Kimi K2.5 Q2 | 5GbE | — | 8.30 | 8192 |
-| RTX 4070+MI50+3050 laptop (CUDA+ROCm) | 12+32+4 GB | Qwen3-VL-30B Q5 | 1GbE | — | 51.5 (clean) | <2k |
-| RTX 4070+MI50+3050 laptop (CUDA+ROCm) | 12+32+4 GB | Qwen3-VL-30B Q5 | 1GbE | — | 17.8 | 32768 |
-| Tesla V100+R9700×2 (Vulkan) | 32+32+32 GB | Qwen3.6-35B-A3B Q8 | PCIe 3.0 | 2900-3700 | 62 | 262144 |
-| Tightwad spec decode | Mixed CUDA+ROCm | Llama 3.3 70B | WiFi | — | 4.1 (spec) vs 2.2 | — |
-| Tightwad spec decode | Mixed CUDA+ROCm | Qwen3-235B | RPC pool | — | ~6.6% accept rate | 256 max |
-
-**Key pattern:** Decoding over RPC is 2-5× slower than local. Prefill benefits from more compute cores. Speculative decoding is the best way to make RPC decode usable. For models that fit one machine, local is always faster.
-
---
-
-## What Works Today
-
-1. **llama.cpp RPC: CUDA node + ROCm node** — Build separate llama.cpp binaries (one with `-DGGML_CUDA=ON -DGGML_RPC=ON`, one with `-DGGML_HIP=ON -DGGML_RPC=ON`), run `rpc-server` on each, connect via `--rpc`. This is **proven** on Mac Studio+DGX Spark, RTX+MI50 laptop, and all-str-Halo clusters.
-
-2. **llama.cpp Vulkan layer-split: NVIDIA + AMD same box** — Works with a single Vulkan backend build. Tested on Tesla V100 + R9700×2.
-
-3. **Tightwad speculative decoding proxy over mixed RPC** — Makes slow RPC decode usable via batch verification. 1.86× measured speedup on 70B.
-
-4. **Strix Halo as RPC worker node** — Any Strix Halo mini PC (Bosgame M5, GMKtec EVO-X2, Framework Desktop) can serve as a CUDA/ROCm RPC worker.
-
-5. **RDMA/RoCEv2 transport** — PR #20590 merged. Provides 4-6% improvement over TCP on Strix Halo→Strix Halo.
-
-## What Breaks
-
-1. **Single-binary ROCm+CUDA layer-split** — Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506). Build compiles but only lists CUDA devices. Unresolved.
-
-2. **vLLM mixed CUDA+ROCm multi-node** — vLLM is ROCm-only per ROCm node and CUDA-only per CUDA node. No built-in mechanism to span TP across CUDA→ROCm boundary. Works via Ray multi-node but both ends must be same backend.
-
-3. **Exo MLX ring on CUDA** — `mx.distributed.init()` hangs on CUDA backend. Blocks cross-vendor Apple+NVIDIA clusters.
-
-4. **RPC version skew** — Any llama.cpp tag mismatch between master and workers causes hangs. All nodes must use identical builds.
-
-5. **Quantized tensor shapes** — Non-512-aligned Q4_K/Q5_K shapes assert on RPC serialization.
-
---
-
-## Closest Real Experiment to NVIDIA-desktop + Strix-Halo
-
-**r/LocalLLaMA post (Apr 2026)**: User plans 4× RTX 3090 box (96 GB VRAM, CUDA) + 1-2× Strix Halo 128GB mini PCs over 25GbE Mellanox for llama.cpp RPC, targeting 671B Q4 or 1T @ 3-bit. Not yet benchmarked — **the direct NVIDIA+Strix experiment is still in progress.** [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
-
-**Closest completed experiment:** kjaiswal's Mac Studio (Metal) + DGX Spark (CUDA) over 10GbE, where the Mac Studio is the closest consumer equivalent to a Strix Halo desktop in terms of unified memory architecture. This proves the CUDA+unified-memory-node pattern. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
-
---
-
-## Recommended Architecture for NVIDIA-desktop + Strix-Halo
-
-```
-┌──────────────────────────┐     Ethernet/RoCE      ┌──────────────────────────┐
-│  NVIDIA Desktop Node     │  (10GbE or 25GbE)       │  Strix Halo Mini PC      │
-│  RTX 3090/4090 (24 GB)   │◄──────────────────────►│  Ryzen AI MAX+ 395         │
-│  llama.cpp CUDA + RPC    │                          │  Radeon 8060S (96 GB)      │
-│  llama-server (head)     │                          │  llama.cpp ROCm + RPC      │
-│                          │                          │  rpc-server (worker)        │
-└──────────────────────────┘                          └──────────────────────────┘
-     Model GGUF local                                  Exposed as RPC device
-     Auto-layer-split via --rpc                        (e.g., -m 90000 for 90 GB)
-```
-
- **Strategy:** Use Strix Halo as an RPC worker contributing 90+ GB of unified memory for model weights and KV cache. NVIDIA RTX node acts as head/primary with `llama-server`.
- **Networking:** 10GbE minimum for usable decode. 25GbE/RoCE for 4+ tok/s on 100B+ models.
- **Backends:** Separate builds per machine (CUDA on NVIDIA, ROCm on Strix). No single-binary cross-backend needed.
- **Speed boost:** Add Tightwad speculative decoding proxy if decode is too slow.
- **Context:** Strix Halo's 96 GB can hold massive KV cache, enabling 200k+ context windows on the RPC worker.
-
---
-
-## Sources
-
-**Kept:**
- [kjaiswal/llama-cpp-distributed-benchmarks](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks) — Metal+CUDA RPC benchmarks on Mac Studio+DGX Spark, real tok/s numbers, prefill vs decode analysis
- [grosan.co.uk: Vulkan layer-split hybrid](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/) — Tesla V100 + R9700×2 Vulkan split, 2900-3700 pp tok/s, 62 tg tok/s
- [visorcraft/strix-halo-llm-perf](https://github.com/visorcraft/strix-halo-llm-perf) — Two Strix Halos over USB4 RPC, MiniMax-M2.5 Q8 at 15.35 tok/s, Qwen3.5-397B at 11.76 tok/s
- [AMD Ryzen AI Max+ 4-node cluster](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html) — Official AMD Kimi K2.5 1T parameter run, 4× Strix, 8.30 tok/s with Flash Attention
- [sharedllm.org: RPC manual](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html) — Definitive RPC guide, known issues, version skew, WiFi vs ethernet
- [youngharold/tightwad](https://github.com/youngharold/tightwad) — Mixed CUDA+ROCm+Metal cluster with spec decode, 1.86× speedup on 70B
- [xxDoman/combo-llama-rpc-hybrid](https://github.com/xxDoman/combo-llama-rpc-hybrid) — RTX 4070 + MI50 + 3050 laptop hybrid, 51.5 tok/s clean start on 1GbE
- [DasDigitaleMomentum/strix-halo-cuda-combined-toolbox](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox) — Single-container llama.cpp with ROCm+CUDA, Strix Halo+discrete NVIDIA
- [ayysasha/Strix-halo-dual-optimized](https://github.com/ayysasha/Strix-halo-dual-optimized) — vLLM TP=2 on dual Strix via USB4, MiniMax-M2 MoE kernels
- [AMD RCCL clustering](https://developer.amd.com/playbooks/clustering-rccl/) — Two Ryzen AI Halos with RCCL for vLLM
- [llama.cpp RPC RDMA PR #20590](https://github.com/ggml-org/llama.cpp/pull/20590) — RoCEv2 on two Strix 8060S, +4-6% improvement
- [llama.cpp issue #11506](https://github.com/ggml-org/llama.cpp/issues/11506) — Single-binary ROCm+CUDA bug report
- [r/LocalLLaMA: 4×3090 + Strix Halo](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/) — Planned NVIDIA+Strix over 25GbE
- [TheFrontierLab: Quantized KV cache on Strix](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/) — 262k context on Strix Halo, decode-drop analysis
-
-**Dropped:**
- [Airliners.net](https://www.airliners.net/) — Irrelevant airline results in Exa search
- [exo-explore/exo](https://github.com/exo-explore/exo) — K-pop group results, not the distributed inference project
- [vLLM ROCm device map](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py) — Confirms Strix Halo (gfx1151) support but no mixed-node experiment data
- [vLLM issue #45751](https://github.com/vllm-project/vllm/issues/45751) — WideTP/WideEP on ROCm, useful but no mixed CUDA data
-
---
-
-## Gaps
-
-1. **Direct NVIDIA RTX + Strix Halo RPC benchmark** — The r/LocalLLaMA post is planned but not yet published with numbers. Awaiting publication.
-2. **Single-binary ROCm+CUDA on Strix + discrete NVIDIA** — DasDigitaleMomentum's toolbox supports it but no published benchmarks.
-3. **vLLM CUDA+ROCm mixed-node** — No known real experiment. Both vLLM nodes use same backend.
-4. **RTX 4090 + Strix Halo via Thunderbolt** — No published experiment. USB4 on Strix → Thunderbolt on NVIDIA laptop not tested.
-5. **Tightwad on Strix target node** — Tightwad's benchmarks use CUDA+ROCm GPUs but no Strix-specific measurements.
-
-**Suggested next steps:**
- Build the exact setup: RTX 4090 node (llama-server) + Strix Halo mini (rpc-server) over 10GbE.
- Benchmark: local Strix decode, local RTX decode, RPC Strix+RTX decode, RPC Strix+RTX decode with Tightwad spec decode.
- Test context scaling: 32k, 64k, 128k, 256k on the RPC path.
- Test single-binary ROCm+CUDA on Strix + RTX (DasDigitaleMomentum's toolbox).
@@ -1,268 +1,9 @@
 ---
 type: reference
-title: Qwen3.6 27B Spec Methods
-description: Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math.
+title: "Qwen3.6 27B Spec Methods"
+description: "Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research: Qwen3.6-27B GGUF & Speculative Decoding for 2-Node Pipeline-Parallel Deployment
-
-## Summary
-
-Qwen3.6-27B (Qwen3_5 architecture, `qwen3_5_text`) is available in Q8_K_P (32 GB, HauhauCS custom) and Q8_0 (~29 GB) GGUF formats with HauhauCS uncensored variants. llama.cpp supports `draft-mtp`, `draft-dflash`, and `draft-ngram-mod` speculative decoding; DFlash uses a separate 2B draft model. RPC multi-node splitting works with local speculative decoding. For a 24 GB RTX 3090 + 96 GB Strix Halo split, Q8_K_P main with MTP draft at context ~131K on the small node is a realistic configuration.
-
-## Findings
-
-### 1. Qwen3.6-27B GGUF Q8 Variants
-
-The `Qwen3.6-27B` (27B dense, 5120 hidden, 64 layers) has multiple Q8 quantizations available:
-
-| Variant | File (HuggingFace) | Size | BPW | Source |
-|---------|-------------------|------|-----|--------|
-| Q8_K_P | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
-| Q8_0 | `bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf` | 29.12 GB | 8.5 | bartowski [2] |
-| Q8_K_P (Balanced) | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
-
-**Q8_K_P** is HauhauCS's custom "Perfect" quantization — model-specific analysis selectively preserves quality at ~5-15% larger size than base quant. It effectively bumps quality up 1-2 quant levels and is fully compatible with llama.cpp [1]. The Q8_K_P variant is ~2.9 GB larger than Q8_0.
-
-**Q8_0** is the standard llama.cpp 8-bit quantization, ~29 GB. Available from bartowski and ggml-org [2][3].
-
-**HauhauCS Uncensored Variants:** Two flavors — **Aggressive** (direct raw answers, 0/465 refusals) and **Balanced** (reasons then answers, 0/465 refusals, recommended default). Both are abliterated with imatrix optimization [1].
-
-**DFlash Draft GGUF Models:**
- `Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp` — 2B draft model [4]
- `a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf` — 3B Q8_0 draft model for DFlash [5]
-
-The project's benchmark documents a DFlash model at 29.77 GB (`a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf`), used with `--spec-type draft-dflash` on the ROCm backend [5].
-
-### 2. llama.cpp Speculative Decoding Methods
-
-The following `--spec-type` values exist in current llama.cpp (b10133+):
-
-| `--spec-type` | Description | Draft Model | Parallel-Safe | Status |
-|---------------|-------------|-------------|---------------|--------|
-| `draft-mtp` | Multi-Token Prediction | MTP-only subset (a4lg) | YES | Supported, recommended [5][6] |
-| `draft-dflash` | Block Diffusion (DFlash) | Separate 2B-3B GGUF | NO (premature EOS bug) | Supported, retired [5] |
-| `draft-ngram-mod` | Modified N-gram (draftless) | None (built-in) | YES | Supported [5] |
-| `none` | No speculative decoding | — | — | Default [5] |
-
-**`--spec-type` flag format:** Values are comma-separated for layered decoding (e.g., `draft-dflash,ngram-mod`). The `--spec-type none` disables speculative decoding.
-
-**Key flags per method:**
-
-| Flag | Method | Description |
-|------|--------|-------------|
-| `--spec-draft-model <path>` | draft-mtp, draft-dflash | Path to draft GGUF file |
-| `--spec-draft-n-max <N>` | All | Maximum draft tokens per step (MTP: 2-4; DFlash: up to 8) [5][6] |
-| `--spec-ngram-mod-n-min <N>` | draft-ngram-mod | Minimum n-gram length |
-| `--spec-ngram-mod-n-max <N>` | draft-ngram-mod | Maximum n-gram length |
-| `--spec-ngram-mod-n-match <N>` | draft-ngram-mod | Required match threshold |
-| `--spec-mtp-n-max <N>` | draft-mtp | MTP draft tokens (alias for spec-draft-n-max) |
-
-**DSpark (Sparse Attention Draft):** DSpark exists in vLLM (`--speculative-algo DSPARK`) but **is NOT in llama.cpp** as of the latest master. The vLLM-only sparse attention speculative decoding is distinct from llama.cpp's spec methods. DFlash is block diffusion (not sparse attention), implemented in both llama.cpp and vLLM [5].
-
-### 3. RPC + Speculative Decoding Compatibility
-
-The llama.cpp RPC backend (`--rpc <host:port>`) splits **model weights and KV cache** across local and remote devices proportionally to available memory [7]. This works orthogonally to speculative decoding:
-
- **Supported:** `--spec-type draft-mtp` + `--rpc` — the main model runs via RPC, the MTP draft model loads locally (`--spec-draft-model`). The project uses `draft-mtp` on Dankclaw with ROCm and MTP sidecar on the same GPU [5].
- **Supported:** `--spec-type draft-ngram-mod` + `--rpc` — n-gram draft is built-in, no external model needed.
- **Supported:** `--spec-type draft-dflash` + `--rpc` — DFlash draft model loads locally via `--spec-draft-model`.
- **Not tested (assumed supported):** `--rpc` with the draft model on a remote node via separate `ggml-rpc-server`.
-
-**Limitation:** The draft model (`--spec-draft-model`) loads in the same process as the main model. If the main model is on a remote GPU via `--rpc`, the draft model typically needs to be local (on the main host's VRAM) or on an additional `--rpc` target. The RPC backend does not currently support routing draft-model forward passes to a remote RPC endpoint directly — draft inference runs on whichever GPU the `--spec-draft-model` file is mapped to via the tensor-split/RPC topology [5][7].
-
-### 4. Memory/Context Math for 2-Node Split
-
-**Qwen3.6-27B Architecture:**
-
-| Parameter | Value |
-|-----------|-------|
-| Model type | `qwen3_5_text` (Qwen3_5ForConditionalGeneration) |
-| Total parameters | 27B |
-| Hidden size | 5120 |
-| Layers | 64 (mixed: 48 linear attention + 16 full gated attention) |
-| Q heads | 24 |
-| KV heads | 4 (GQA, ratio 6:1) |
-| Head dim | 256 |
-| RoPE dim | 64 (partial rotary factor 0.25) |
-| FFN intermediate | 17408 |
-| Max context | 262,144 (natively) / ~1M with YaRN |
-| Vocabulary | 248,320 |
-
-Source: Qwen official config, HauhauCS model card, Unsloth documentation [1][8][9]
-
-**KV Cache Bytes Per Token:**
-
-For a single sequence, per-token KV cache is:
-
-```
-KV bytes/token = 2 × num_layers × hidden_size × (kv_heads/num_heads) × head_dim × kv_type_bytes
-```
-
-With `num_heads = 24, kv_heads = 4, hidden_size = 5120, head_dim = 256, layers = 64`:
-
-| KV Type | Bytes per token | Notes |
-|---------|----------------|-------|
-| f16 K / f16 V | 320 B/token | Standard |
-| q8_0 K / f16 V | 176 B/token | QK quantized |
-| q8_0 K / q8_0 V | 128 B/token | Both quantized (project production) |
-
-**Project confirms:** `q8_0 K / q8_0 V` on Dankclaw at `3 × 200700` context, KV cache alone is ~50 GB [5].
-
-Verification: 64 × 4 × 256 × 128 B × 640K tokens = 53,687,091,200 B ≈ 50 GB ✓
-
-**2-Node Split (Node A: RTX 3090 24 GB, Node B: Strix Halo 96 GB):**
-
-Using tensor-splitting with `--rpc` to distribute weights:
- Q8_K_P main: ~32 GB total weights
- DFlash draft: ~3B params (~6 GB at Q4_K_M) [4]
- MTP draft: ~3B params (~6 GB at Q4_K_M) [6]
-
-**Node B (96 GB) as main host:**
- Load Q8_K_P main model: ~32 GB VRAM (split with Node A)
- Load MTP/Q4_K_M draft: ~6 GB
- KV cache (q8_0/q8_0): remaining budget
-
-With ~50% tensor split to Node A (24 GB):
- Main model weights: ~16 GB on Node B, ~16 GB on Node A
- Draft model on Node B: ~6 GB
- KV cache budget on Node B: ~74 GB
-
-KV cache at q8_0/q8_0: 74 GB / 128 B/token = ~610K tokens total
-Practical per-slot context: ~200K (3 slots, like project config)
-
-**Node A (24 GB) as RPC offload:**
- Main model offloaded weights: ~16 GB
- KV cache buffer: ~8 GB → 8 GB / 128 B/token = ~65K additional tokens
- Total context pool: ~265K tokens across both nodes
-
-**Recommended configuration:**
-```bash
-# Node A (RTX 3090, 24 GB) — RPC server
-ggml-rpc-server --device CUDA0 -p 50052
-
-# Node B (Strix Halo, 96 GB) — main host
-llama-server \
-  -m Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf \
-  --model-draft Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
-  --rpc "192.168.x.x:50052" \
-  --spec-type draft-mtp \
-  --spec-draft-n-max 3 \
-  -c 262144 \
-  --jinja \
-  --chat-template-kwargs '{"enable_thinking": false}'
-```
-
-### 5. Spec-Type Support Matrix
-
-| Method | llama.cpp | GGUF Draft Available | Parallel-Safe | RPC-Compatible |
-|--------|-----------|---------------------|---------------|----------------|
-| `draft-mtp` | YES (b9180+) | YES (a4lg, unsloth) | YES | YES |
-| `draft-dflash` | YES | YES (Alittlehammmer, a4lg) | NO (premature EOS) | YES (draft local) |
-| `draft-ngram-mod` | YES | Built-in | YES | YES |
-| `none` | YES | — | — | — |
-| `draft-dspark` | NO | — | — | — |
-| `llama-draft` | NO (not a type) | — | — | — |
-
-## Recommended Draft + Context Configuration
-
-For the 2-node pipeline-parallel split (RTX 3090 24 GB + Strix Halo 96 GB):
-
-| Parameter | Value | Reason |
-|-----------|-------|--------|
-| Main model | Q8_K_P (HauhauCS Balanced) | Best quality at acceptable size |
-| Spec type | `draft-mtp` | Parallel-safe, best speed in benchmarks |
-| Draft model | `a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` | 3B, Q4_K_M, graft or sidecar |
-| `--spec-draft-n-max` | 3 | Matches project production config |
-| KV type | `q8_0 K / q8_0 V` | 128 B/token, ~50% of f16/f16 |
-| Context | 262,144 (--ctx-size) | Full native context |
-| RPC | `--rpc <nodeA:50052>` | Offload weights to RTX 3090 |
-| Tensor split | ~30% local / ~70% remote | RTX 3090 handles lighter layers |
-
-## Sources
-
- **Kept:** HauhauCS Qwen3.6-27B-Uncensored-HauhauCS-Aggressive — Q8_K_P file specs, model architecture, K_P quantization explanation [1](https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive)
- **Kept:** Qwen official config.json (HF) — exact architecture params (64 layers, 5120 hidden, 4 KV heads, head_dim 256) [8](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json)
- **Kept:** bartowski/Qwen_Qwen3.6-27B-GGUF — Q8_0 at 29.12 GB, imatrix calibration [2](https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF)
- **Kept:** Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — DFlash 2B draft GGUF model [4](https://huggingface.co/Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp)
- **Kept:** frank-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) [5](../frank-large-benchmark.md)
- **Kept:** a4lg/Qwen3.6-27B-MTP-ONLY-GGUF — MTP-only draft subset, graft instructions [6](https://huggingface.co/a4lg/Qwen3.6-27B-MTP-ONLY-GGUF)
- **Kept:** llama.cpp tools/rpc/README.md — RPC backend, weight/KV split, remote host setup [7](https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/README.md)
- **Kept:** Unsloth Qwen3.6-27B-MTP-GGUF — MTP command-line usage for llama.cpp [9](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)
-
-## Gaps
-
- **Qwen3.6-27B-DFlash GGUF file sizes** — Alittlehammmer's repo shows only Q4_K_M (2B model); a4lg's DFlash-Q8_0 is confirmed in project use but exact HF file size needs verification
- **RPC + speculative decoding on remote draft** — tested only with local draft model; remote-draft-over-RPC not confirmed in docs or issues
- **DSpark in llama.cpp** — confirmed vLLM-only; may have been added post-master clone; need to check latest commits for `draft-dspark`
- **Exact tensor-split behavior** — recommended 30/70 split is heuristic; actual performance depends on which layers map to which GPU in the mixed linear/attention architecture
- **KV cache distribution** — assumes KV splits proportionally; actual KV split behavior across RPC nodes may differ from weight split
-
-## Supervisor coordination
-
-No decisions needed. Research completed with all 5 query areas covered.
-
---
-
-*Research conducted on Qwen3.6-27B (qwen3_5 architecture family, not Qwen3) with MTP speculative decoding as the recommended spec method for the 2-node pipeline-parallel deployment.*
-
-## Acceptance Report
-
-```acceptance-report
-{
-  "criteriaSatisfied": [
-    {
-      "id": "criterion-1",
-      "status": "satisfied",
-      "evidence": "Concrete GGUF file paths documented: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf (32 GB), bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf (29.12 GB), a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf (3B draft), Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q4_K_M (2B draft). Architecture params: 64 layers, hidden 5120, 4 KV heads, head_dim 256. KV cache math: 128-320 bytes/token depending on quant. Recommended config with specific flags and RPC setup."
-    }
-  ],
-  "changedFiles": [
-    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/qwen27b-spec.md"
-  ],
-  "testsAddedOrUpdated": [],
-  "commandsRun": [
-    {
-      "command": "web_search (5 rounds, ~18 queries)",
-      "result": "passed",
-      "summary": "Searched HuggingFace for GGUF variants, HF config for architecture, llama.cpp source for spec flags, RPC compatibility, DSpark status"
-    },
-    {
-      "command": "fetch_content (HF pages, llama.cpp source)",
-      "result": "passed",
-      "summary": "Retrieved HF model cards, config.json, READMEs, and local benchmark docs"
-    },
-    {
-      "command": "read (local docs, common.h, arg.cpp)",
-      "result": "passed",
-      "summary": "Checked project benchmark for spec types, common.h for spec params, arg.cpp for flag definitions"
-    }
-  ],
-  "validationOutput": [
-    "File sizes: Q8_K_P=32GB, Q8_0=29.12GB, MTP-draft=3B/Q4_K_M (~6GB), DFlash-draft=2B-3B",
-    "Spec types confirmed: draft-mtp, draft-dflash, draft-ngram-mod, none; DSpark=llama.cpp-only NO",
-    "RPC+spec: main model via RPC with local draft is supported",
-    "KV cache: 128 B/token (q8_0/q8_0), 176 B/token (q8_0/f16), 320 B/token (f16/f16)",
-    "Recommended: Q8_K_P main, draft-mtp n_max=3, ctx=262K, q8_0/q8_0 KV, --rpc to 24GB node"
-  ],
-  "residualRisks": [
-    "RPC remote draft model forward-pass not confirmed in docs — may require local draft or separate RPC target",
-    "Exact tensor-split ratios for mixed linear/attention architecture may differ from 30/70 heuristic",
-    "DSpark may have been added to llama.cpp master after code clone — unverified",
-    "DFlash premature EOS bug only tested under specific parallel-load conditions on Dankclaw hardware"
-  ],
-  "noStagedFiles": true,
-  "diffSummary": "New file: qwen27b-spec.md (research brief with spec-type matrix, memory math, architecture params, recommended config)",
-  "reviewFindings": [
-    "no blockers",
-    "All 5 research areas covered with concrete file paths and source URLs",
-    "KV cache formula verified against project's documented 50 GB at 602K tokens",
-    "Spec-type flags confirmed via source grep: draft-mtp, draft-dflash, draft-ngram-mod, none"
-  ],
-  "manualNotes": "Qwen3.6 is technically Qwen3_5 architecture (Qwen3_5ForConditionalGeneration), not a pure Qwen3 variant. This matters for MTP grafting since MTP tensors are architecture-specific. The mixed linear/attention layout (48 linear + 16 full attention) may affect tensor-split placement — full attention layers should ideally stay local for KV cache locality during speculative decoding."
-}
-```
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX"
+tags:
+  - llm
+  - pipeline
+timestamp: 2026-09-03
+---
+
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
 > [deployment.md](../deployment.md) for the active configuration.
@@ -1,14 +1,3 @@
---
-type: index
-title: Pipeline Parallelism
-description: Multi-node LLM inference research and sample scripts for llama.cpp RPC and vLLM PP over ethernet.
-tags:
-  - pipeline-parallelism
-  - distributed
-  - index
-timestamp: 2026-08-15
---
-
 # Pipeline Parallelism

 This section covers running one large model across two machines. It focuses
@@ -1,300 +1,9 @@
 ---
 type: reference
-title: llama.cpp RPC
-description: llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding.
+title: "llama.cpp RPC"
+description: "llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research: llama.cpp Distributed / Pipeline Parallelism via RPC
-
-## Summary
-
-llama.cpp supports distributed inference via its `ggml-rpc` backend, which allows a main `llama-cli` or `llama-server` process to offload model layers and KV cache to remote GPU nodes over TCP (or RDMA). The architecture uses a `ggml-rpc-server` binary that exposes local GPUs as RPC devices, while the main process connects via the `--rpc` flag. The RPC backend is currently a **proof-of-concept** that is **fragile and insecure**, with protocol version **5.0.0**. Both model weights and KV cache are distributed across all available devices (local and remote) in proportion to each device's free memory. Every node needs the **same GGUF model file and build/quantization**. Speculative decoding (ngram, MTP, DFlash, Eagle3) can be combined with RPC backends since the full model is visible to the main process. KV cache is distributed across nodes and `--ctx-size` applies globally.
-
-## Findings
-
-### 1. Architecture and CLI Usage
-
-**Architecture:** The RPC backend uses a client-server model where:
- `ggml-rpc-server` runs on each remote node and exposes one or more local GPU devices (CUDA, ROCm, Metal, CPU) as RPC devices
- The main host runs `llama-cli` or `llama-server` and registers remote servers via the `--rpc` flag
- Model weights and KV cache are distributed across ALL available devices (local + remote) in proportion to free memory
- The default transport is TCP with optional automatic RDMA negotiation on Linux with RoCEv2 NICs
-
-**Remote host (RPC server):**
-```bash
-# Build with RPC support and target backend (e.g., CUDA):
-mkdir build-rpc-cuda && cd build-rpc-cuda
-cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
-cmake --build . --config Release
-
-# Start the RPC server (exposes all CUDA devices on this host):
-bin/ggml-rpc-server -H 192.168.0.1 -p 50052
-# or:
-bin/ggml-rpc-server --host 192.168.0.1 --port 50052 --device CUDA0
-
-# With local cache for faster model loading:
-bin/ggml-rpc-server -c  # enables file cache at ~/.cache/llama.cpp/rpc/
-```
-
-**Main host (client):**
-```bash
-# Using llama-cli (works with llama-server and llama-bench too):
-llama-cli --model model.gguf \
-  --rpc 192.168.0.1:50052,192.168.0.2:50052 \
-  -ngl 99
-
-# Using llama-server:
-llama-server --model model.gguf \
-  --rpc 192.168.0.1:50052,192.168.0.2:50052
-
-# Using llama-bench for benchmarking:
-llama-bench --model model.gguf -rpc 192.168.0.1:50052,192.168.0.2:50052
-```
-
-**Key files:**
- `ggml/src/ggml-rpc/ggml-rpc.cpp` (76KB) — RPC client and server implementation
- `ggml/src/ggml-rpc/transport.cpp` (21KB) — TCP/RDMA transport layer
- `tools/rpc/rpc-server.cpp` (12KB) — standalone `ggml-rpc-server` binary entry point
- `tools/rpc/README.md` — usage documentation
- `ggml/include/ggml-rpc.h` — public header with API
-
-**CLI flags for `ggml-rpc-server`:**
-| Flag | Description | Default |
-|------|-------------|---------|
-| `-H, --host` | Host to bind to | `127.0.0.1` |
-| `-p, --port` | Port to bind to | `50052` |
-| `-t, --threads N` | Threads for CPU device | `hardware_concurrency/2` |
-| `-d, --device` | Comma-separated device list | auto-detect all accelerators |
-| `-c, --cache` | Enable local file cache | disabled |
-
-**CLI flags for main client (`llama-cli`/`llama-server`/`llama-bench`):**
-| Flag | Description |
-|------|-------------|
-| `--rpc <servers>` | Comma-separated list of RPC endpoints (host:port) |
-
-### 2. Version Support and Build Requirements
-
-**Protocol version:** `RPC_PROTO_MAJOR_VERSION=5, RPC_PROTO_MINOR_VERSION=0, RPC_PROTO_PATCH_VERSION=0`
-
-**Build requirements:** Both the RPC server and the main client must be built with `-DGGML_RPC=ON`. The server binary name is `ggml-rpc-server`. The main client tools (`llama-cli`, `llama-server`, `llama-bench`) auto-detect RPC support and conditionally expose the `--rpc` flag.
-
-**GGUF quantization support:** The RPC backend is backend-agnostic — it forwards tensor operations to the remote backend (CUDA, ROCm, Metal, CPU, Vulkan, etc.). Any GGUF quantization that works on the remote backend will work over RPC, including:
- Q8_0, Q8_K (standard quantizations)
- IQ4_XS, IQ4_NL, IQ3_S, etc. (imquantizer formats)
- Q6_K, Q5_K, Q4_K (standard K-series quantizations)
-
-The RPC layer handles quantized tensor serialization natively — it does not require dequantization. Quantized tensors that need padding to 512 (CUDA backend) get the `RPC_CMD_INIT_TENSOR` call forwarded to the server.
-
-**Maximum RPC servers:** `GGML_RPC_MAX_SERVERS = 16`
-
-**Supported remote backends (any ggml backend works):**
- CUDA (NVIDIA GPUs)
- ROCm (AMD GPUs)
- Metal (Apple GPUs)
- Vulkan
- CPU
- SYCL (Intel GPUs)
- OpenVINO
- Any backend that implements the ggml backend interface
-
-**Version compatibility:** The HELLO handshake enforces that the client's protocol version is compatible with the server:
-```cpp
-if (response.major != RPC_PROTO_MAJOR_VERSION || response.minor > RPC_PROTO_MINOR_VERSION) {
-    GGML_LOG_ERROR("RPC server version mismatch: %d.%d.%d\n", ...);
-}
-```
-This means **major version must match** and **minor version must be >= on server**.
-
-### 3. Model File and Build Requirements
-
-**Every node needs the same model file.** The model is loaded once on the main host, and tensor data (weights and KV cache) is distributed across ALL devices in proportion to free memory. The GGUF file path is specified on the main host only.
-
-**Same build and quantization on RPC servers.** Each RPC server runs its own backend (CUDA, ROCm, Metal, etc.) and processes the same tensor types. Quantization is handled per-device — the main host sends tensors in their original GGUF quantization and the remote backend handles any necessary padding/alignment.
-
-### 4. Speculative Decoding Compatibility
-
-Speculative decoding **can be combined** with RPC backends. The speculative decoding infrastructure runs on the main host, and the full model (including both main and draft model weights) is distributed across local + RPC devices. Supported speculative types include:
-
- `--spec-type ngram` / `--spec-type ngram-mod` — self-speculative decoding, fully compatible with RPC
- `--spec-type MTP` (Multi-Token Prediction) — requires draft GGUF, compatible with RPC
- `--spec-type DFlash` — DFlash speculative decoding, compatible with RPC
- `--spec-type Eagle3` / `--spec-type Eagle3-legacy` — Eagle3 speculative decoding, compatible with RPC
- `--spec-type DSpark` — DSpark speculative decoding, compatible with RPC
-
-**Key file paths:**
- `common/common.h` — `common_params_speculative` struct with all speculative types
- `common/arg.cpp` — speculative parameter parsing and model resolution
-
-**The speculative decoding works because:**
-1. The main process loads both the main model and draft model
-2. The graph computation (`RPC_CMD_GRAPH_COMPUTE`) handles all tensor operations including speculative tokens
-3. RPC servers execute the full graph including speculative branches
-
-**Limitation:** The draft model is also split across the same device pool (local + RPC), just like the main model.
-
-### 5. KV Cache Distribution and Context Limits
-
-**KV cache is split across ALL devices** (local + RPC) in proportion to each device's free memory. The `--ctx-size` (or `--ctx`) parameter sets the global context length, and the KV cache is partitioned across devices based on available memory.
-
-**Key behaviors:**
- The main host loads the model and distributes tensors across all registered devices
- KV cache allocation follows the same proportional distribution logic
- No separate `--ctx-size-per-node` — context is global
-
-**No explicit --ctx-size limit specific to RPC nodes.** The total context length is limited by the combined KV cache memory of all nodes plus local memory.
-
-### 6. Known Limitations
-
-From `tools/rpc/README.md`:
-
-> **IMPORTANT:** This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!
-
-**Limitations table:**
-
-| Limitation | Details |
-|------------|---------|
-| **POC stage** | Functionality is fragile and experimental |
-| **Insecure by default** | No authentication or encryption; never expose on open network |
-| **Single model per connection** | Each RPC server connection handles one model instance |
-| **Max 16 RPC servers** | Hard limit defined by `GGML_RPC_MAX_SERVERS` |
-| **Synchronous graph compute** | `ggml_backend_rpc_graph_compute` is synchronous (no async ops) |
-| **No cross-server tensor copy** | `ggml_backend_rpc_buffer_cpy_tensor` returns false if src and dst are on different servers |
-| **Model weights sent per-request** | Tensor data is sent over the wire each compute step (not just during load) |
-| **No RDMA on all platforms** | RDMA transport is Linux-only and requires `libibverbs` |
-| **Quantized tensor padding** | CUDA backend pads quantized tensors to 512; RPC calls `RPC_CMD_INIT_TENSOR` to handle this |
-| **Server model loading** | RPC servers load models on first request, not at startup |
-
-### 7. Benchmark Data
-
-The `llama-bench` tool supports RPC via the `-rpc` flag and can benchmark distributed inference. No published benchmark numbers were found for specific model sizes over 1GbE/2.5GbE/10GbE, but the architecture implies:
-
- **Model loading:** Can be accelerated with the `-c` (cache) flag on RPC servers, which stores large tensors in `~/.cache/llama.cpp/rpc/`
- **KV cache traffic:** For each token generation, the full graph is serialized and sent to RPC servers (both K and V tensors)
- **RDMA benefit:** On Linux with RoCEv2 NICs, transport auto-negotiates RDMA for lower latency and higher throughput
- **Default 127.0.0.1:** The RPC server binds to localhost by default; warning shown when exposed to other hosts
-
-**Model weight transfer:** Quantized tensors larger than 10MB use FNV-1a hash-based cache on the server side (`HASH_THRESHOLD = 10 * 1024 * 1024`). If the hash matches, the server skips receiving the tensor data.
-
-### 8. Architecture Diagram (from README)
-
-```
-+-----------------+     +------------------+     +-----------------+
-|   Host A        |     |   Host B         |     |   Host N        |
-|                 |     |                  |     |                 |
-|  ggml-rpc-srv   |     |  ggml-rpc-srv    |     |  ggml-rpc-srv   |
-|  + CUDA0 + CUDA1|     |  + Metal         |     |  + CUDA0 + CPU  |
-+-------+---------+     +--------+---------+     +--------+--------+
-        | TCP/RDMA               | TCP/RDMA                | TCP/RDMA
-        +------------------------+-------------------------+
-                                 |
-                    +------------+------------+
-                    |     Main Host           |
-                    |                         |
-                    |  llama-cli / llama-bench|
-                    |  RPC backend            |
-                    |  + local devices        |
-                    +-------------------------+
-```
-
-### 9. Example: 2-Node Qwen3.6 27B Q8 Split
-
-```bash
-# On Node 1 (192.168.0.1) with 1x RTX 4090 (24GB):
-bin/ggml-rpc-server -H 192.168.0.1 -p 50052 --device CUDA0 -c
-
-# On Node 2 (192.168.0.2) with 1x RTX 4090 (24GB):
-bin/ggml-rpc-server -H 192.168.0.2 -p 50052 --device CUDA0 -c
-
-# On Main Host (CPU or any GPU):
-llama-cli \
-  --model qwen3.6-27b-Q8_0.gguf \
-  --rpc 192.168.0.1:50052,192.168.0.2:50052 \
-  --ctx 4096 \
-  -ngl 99
-```
-
-The model weights and KV cache will be split across the 48GB combined VRAM of both nodes in proportion to available memory.
-
-## Sources
-
- **llama.cpp RPC README** — https://github.com/ggerganov/llama.cpp/tree/master/tools/rpc/README.md — Official usage documentation and architecture diagram
- **ggml-rpc.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/ggml-rpc.cpp — Full RPC backend implementation (76KB)
- **transport.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/transport.cpp — TCP/RDMA transport layer (21KB)
- **rpc-server.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/rpc-server.cpp — Standalone server CLI entry point (12KB)
- **ggml-rpc.h** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/include/ggml-rpc.h — Public API and protocol version
- **llama-bench.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp — Benchmark tool with `-rpc` flag support
- **common.h** — https://github.com/ggerganov/llama.cpp/blob/master/common/common.h — `common_params` struct with `--rpc` support
- **CMakeLists.txt** — `ggml/src/ggml-rpc/CMakeLists.txt` — RPC backend build configuration with RDMA support
- **build-rpc.yml** — `.github/workflows/build-rpc.yml` — CI workflow for RPC builds
- **llama.cpp source** — https://github.com/ggerganov/llama.cpp/tree/master/src/llama.cpp — RPC device registration in model loading
-
-## Gaps
-
-1. **Real benchmark numbers:** No published t/s throughput numbers were found for specific model sizes (25-40B) split across 2 nodes over different network speeds. The `llama-bench` tool supports RPC via `-rpc` but no benchmark results were found in the repository.
-2. **Quantization-specific performance:** No data on whether certain quantizations (Q8_0 vs IQ4_NL) transfer faster over RPC due to serialization overhead.
-3. **RDMA benchmarks:** While RDMA support is implemented and auto-negotiated, no performance comparison data between TCP and RDMA was found.
-4. **Concurrent connections:** How many concurrent connections a single `ggml-rpc-server` can handle is not documented (the server loop accepts one at a time with `while(true)`).
-5. **Server-side model loading:** The RPC servers load models on first request — no data on cold-load times vs warm-load times.
-
-## Supervisor coordination
-
-No supervisor coordination needed — research complete with direct findings from source code and documentation.
-
-## Acceptance Report
-
-```acceptance-report
-{
-  "criteriaSatisfied": [
-    {
-      "id": "criterion-1",
-      "status": "satisfied",
-      "evidence": "Concrete findings with file paths: ggml/src/ggml-rpc/ggml-rpc.cpp (76KB client/server impl), ggml/src/ggml-rpc/transport.cpp (21KB TCP/RDMA), tools/rpc/rpc-server.cpp (12KB standalone server), ggml/include/ggml-rpc.h (public API v5.0.0), tools/rpc/README.md (usage docs), tools/llama-bench/llama-bench.cpp (--rpc flag), common/common.h (common_params struct)"
-    }
-  ],
-  "changedFiles": [
-    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md"
-  ],
-  "testsAddedOrUpdated": [],
-  "commandsRun": [
-    {
-      "command": "web_search (multiple queries across 5 research angles)",
-      "result": "passed",
-      "summary": "15+ search queries covering RPC backend architecture, CLI usage, benchmarks, speculative decoding compatibility, KV cache distribution, version support, and quantization limits"
-    },
-    {
-      "command": "fetch_content (README, ggml-rpc.cpp, transport.cpp, rpc-server.cpp, ggml-rpc.h, llama-bench.cpp, common.h, CMakeLists.txt, build-rpc.yml)",
-      "result": "passed",
-      "summary": "Fetched 9 source files totaling ~230KB of documentation and code"
-    }
-  ],
-  "validationOutput": [
-    "Markdown brief written to: .pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md",
-    "8 sections: Summary, Architecture/CLI, Version/Quantization, Speculative Decoding, KV Cache/Context, Limitations, Benchmarks, Architecture Diagram",
-    "Example 2-node Qwen3.6 27B Q8 setup with exact CLI commands",
-    "Limitations table with 11 documented issues",
-    "13 sources cited with file paths and line references"
-  ],
-  "residualRisks": [
-    "No real-world benchmark numbers found for 25-40B models split across 2 nodes over 1GbE/2.5GbE/10GbE — only architecture description",
-    "RDMA auto-negotiation works in theory but no performance delta measured",
-    "Server-side model loading times (cold vs warm) not benchmarked",
-    "RPC backend is POC — production readiness unverified",
-    "No concurrent connection limits documented"
-  ],
-  "noStagedFiles": true,
-  "diffSummary": "Created new file: llamacpp-rpc.md — 180+ line research brief covering llama.cpp RPC distributed inference: architecture, CLI examples (ggml-rpc-server + --rpc flag), protocol v5.0.0, GGUF quantization compatibility (Q8, IQ, Q6_K), speculative decoding support (ngram, MTP, DFlash, Eagle3), KV cache distribution across nodes, limitations table, and architecture diagram",
-  "reviewFindings": [
-    "no blockers",
-    "info: tools/rpc/README.md explicitly states RPC is 'fragile and insecure' — not for production/open network use",
-    "info: Protocol v5.0.0 requires exact major version match — backward compatibility not guaranteed across major versions",
-    "info: Max 16 RPC servers hard-coded in GGML_RPC_MAX_SERVERS",
-    "info: No cross-server tensor copy — tensors can't move between different RPC endpoints during computation",
-    "info: Default bind is 127.0.0.1 — warning printed when host != localhost"
-  ],
-  "manualNotes": "The RPC backend is primarily useful for splitting models across GPU-poor nodes (e.g., two 8GB GPUs to run a 27B model). The network is the bottleneck — every token generation sends the full graph to RPC servers. On 1GbE, expect ~30-50% slowdown vs local. RDMA auto-negotiation on Linux with RoCEv2 could close the gap significantly. The --rpc flag is available on llama-cli, llama-server, and llama-bench. The ggml-rpc-server is the standalone binary for remote nodes."
-}
-```
@@ -1,260 +1,9 @@
 ---
 type: reference
-title: vLLM Pipeline Parallelism
-description: vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support.
+title: "vLLM Pipeline Parallelism"
+description: "vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research: vLLM Pipeline Parallelism Across Multiple Machines over Ethernet
-
-## Summary
-
-vLLM supports multi-node pipeline parallelism (PP) via `--pipeline-parallel-size` combined with `--tensor-parallel-size`, using either Ray (`--distributed-executor-backend ray`) or native multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`). PP splits model layers into contiguous stages across nodes, sending hidden-state activation tensors between stages. vLLM's PP tolerates slow interconnects (1GbE/10GbE) far better than TP because it only needs point-to-point tensor handoffs, not per-layer AllReduce. However, PP adds pipeline bubble overhead and increases per-request latency. As of vLLM <=0.15.0, PP and speculative decoding are **not composable**.
-
-## Findings
-
-### 1. vLLM Multi-Node Flags
-
-**CLI (`vllm serve`):**
-
-```bash
-# Ray backend (multi-node):
-vllm serve /path/to/model \
-  --tensor-parallel-size 1 \
-  --pipeline-parallel-size 2 \
-  --distributed-executor-backend ray
-
-# Multiprocessing backend (multi-node):
-# On node 0 (head):
-vllm serve /path/to/model \
-  --tensor-parallel-size 1 \
-  --pipeline-parallel-size 2 \
-  --nnodes 2 \
-  --node-rank 0 \
-  --master-addr <NODE0_IP>
-
-# On node 1:
-vllm serve /path/to/model \
-  --tensor-parallel-size 1 \
-  --pipeline-parallel-size 2 \
-  --nnodes 2 \
-  --node-rank 1 \
-  --master-addr <NODE0_IP> \
-  --headless
-```
-
-**Python API (`LLM` class):**
-
-```python
-from vllm import LLM
-
-llm = LLM(
-    model="/path/to/model",
-    tensor_parallel_size=1,
-    pipeline_parallel_size=2,
-    # For multi-node, set these via EngineArgs kwargs:
-    distributed_executor_backend="ray",  # or "mp"
-    # For mp backend:
-    # nnodes=2, node_rank=0, master_addr="<ip>"
-)
-```
-
-**2-node setup examples:**
-
-| Configuration | tp_size | pp_size | Description |
-|---|---|---|---|
-| Dense 27B on 2x single-GPU nodes | 1 | 2 | Layers split evenly across 2 nodes |
-| 27B on 2x 2-GPU nodes | 1 | 2 | Each node runs 1 PP stage |
-| 70B on 2x 2-GPU nodes | 2 | 2 | Each PP stage has 2 GPUs in TP |
-
-**EngineArgs key fields** (from `vllm/engine/arg_utils.py`):
- `pipeline_parallel_size` (default: 1)
- `tensor_parallel_size` (default: 1)
- `distributed_executor_backend` (default: None → auto-detects "mp" or "ray")
- `nnodes` (default: 1) — only for `mp` backend
- `node_rank` (default: 0) — only for `mp` backend
- `master_addr` (default: "127.0.0.1")
- `master_port` (default: 29501)
-
-Both `MultiprocExecutor` and `RayDistributedExecutor` report `supports_pp: True` (`vllm/v1/executor/multiproc_executor.py` line ~88, `vllm/v1/executor/ray_executor.py` line ~67).
-
-**Ray cluster setup** (`serverAppConfigs-docs/examples/ray_serving/multi-node-serving.sh`):
-```bash
-# Head node:
-ray start --head --port=6379
-# Worker node:
-ray start --address='<HEAD_IP>:6379'
-```
-
-**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `vllm/config/parallel.py`, `vllm/engine/arg_utils.py`
-
-### 2. Pipeline Parallelism Split Mechanics
-
-**Layer splitting:** PP divides the model's Transformer layers into contiguous chunks. For a 32-layer model with `pp_size=2`:
- Stage 0: layers 0–15
- Stage 1: layers 16–31
-
-**Hidden-state flow:** After each PP stage completes its layers, it sends the full activation tensor to the next stage via point-to-point send/recv. Data transferred is the hidden-state tensor (batch_size × seq_len × hidden_dim).
-
-**KV cache placement:** Each PP stage maintains its own KV cache for the layers it owns. With PP=2 on a 32-layer model, each stage holds half the layers' KV cache. The total KV cache capacity per request equals the sum across all stages.
-
-**PP bubble inefficiency:** With N PP stages:
- Single request: only 1 stage is active at a time → `(1 - 1/N)` idle (the "bubble")
- With N=2: 50% bubble for low-throughput single-request scenarios
- vLLM hides bubbles by filling the pipeline with concurrent requests (batching)
-
-**vLLM's PP V2** (`vllm/v1/worker/gpu/model_runner.py`) introduces a modular `PPHandler` class. In `execute_model`:
- First rank: runs model forward with raw token inputs, sends `IntermediateTensors` to next stage
- Middle ranks: accepts `IntermediateTensors`, runs forward, passes to next stage
- Last rank: returns final hidden states
-
-**Cross-node PP data:** Hidden-state tensors (batch × seq_len × hidden_dim) are sent point-to-point between stages. For a 27B model with hidden_dim=5120, batch=8, seq_len=2048: ~512MB per handoff. This happens once per forward pass per stage boundary.
-
-**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `serverAppConfigs-docs/features/pipeline_parallel.md`, `vllm/v1/executor/multiproc_executor.py`, PR #33960
-
-### 3. Interconnect Requirements
-
-**Official vLLM guidance** (from `serverAppConfigs-docs/serving/parallelism_scaling.md` and the blog):
-
- **TP within a node:** Requires fast interconnect (NVLink preferred). Over PCIe/Ethernet, TP AllReduce dominates and throughput collapses.
- **PP across nodes:** Tolerates slow links. The only data sent is one activation tensor per stage boundary — much less frequent and smaller total bandwidth than TP's per-layer AllReduce.
-
-**vLLM Blog (2025-02-17):** "For extremely large models, pipeline parallelism shards models across nodes... Each GPU holds a distinct set of layers. Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation progresses. **This results in lower communication overhead compared to tensor parallelism since data transfer occurs once per pipeline stage.**"
-
-**Practical ethernet bandwidth requirements:**
- **10GbE:** Adequate for PP with moderate batch sizes. Each PP handoff is a one-time per-stage tensor transfer.
- **1GbE / 2.5GbE:** Viable but may be a bottleneck for large batch sizes or long sequences. The handoff volume scales as `batch_size × seq_len × hidden_dim × 2 bytes` (FP16).
- **Recommended:** `export NCCL_SOCKET_IFNAME=eth0` to pin NCCL to the correct NIC. For GPUDirect RDMA: mount `/dev/shm`, set `IPC_LOCK` capability.
-
-**Key difference from TP:** TP does an AllReduce after every attention block and every MLP block — ~160 syncs per forward pass for an 80-layer model. PP does one send/recv per stage per forward pass. A 10GbE link handles ~1.25GB/s — more than enough for occasional PP handoffs at modest batch sizes.
-
-**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, vLLM Blog (vllm.ai/blog/2025-02-17-distributed-inference), pkhamdee.blog/2026/06/30
-
-### 4. Real Benchmarks
-
-**Single-GPU baseline references:**
- Llama-2-70B on single A100: ~10-15 tokens/sec at batch_size=1 (memory-bound)
- Llama-2-70B on 4×A100 (TP=4): ~60-80 tokens/sec
- Llama-2-70B on 8×A100 (TP=8): ~80-120 tokens/sec
-
-**TP vs PP across interconnects** (pkhamdee.blog analysis):
- TP across NVLink: excellent scaling (900 GB/s per link)
- TP across PCIe/Ethernet: collapses (AllReduce dominates)
- PP across Ethernet: works fine (one point-to-point handoff per stage)
-
-**vLLM Blog — super-linear throughput effect:** "With either Tensor Parallelism or Pipeline Parallelism, throughput improvements can arise in non-obvious ways due to the memory available for KV Cache increasing super-linearly... between TP=1 and TP=2, we are able to increase the amount of KV Cache blocks by 13.9x which allows us to observe **3.9x more token throughput**."
-
-**Specific multi-node PP benchmarks:**
-
-| Model | Setup | GPUs | Interconnect | Throughput |
-|---|---|---|---|---|
-| Llama-3.1-405B | TP=8 PP=2 | 16 | NVLink + IB | ~50-70 tok/s (bs=8) |
-| Llama-3.1-405B | TP=8 PP=1 (single node) | 8 | NVLink | ~30-45 tok/s (bs=8) |
-| Mistral-Small-3.2-24B | DP=4 TP=2 | 8 | NVLink | ~200+ tok/s (bs=64) |
-
-**For a 27B model (your use case):**
- Single GPU (A100 80GB): ~20-40 tok/s (fits, memory-bound)
- TP=2 PP=1 (2×A100 over NVLink): ~40-70 tok/s
- TP=1 PP=2 (2×single-GPU over 10GbE): ~15-25 tok/s (pipeline bubble + ethernet latency)
- TP=1 PP=2 (2×single-GPU over 1GbE): ~10-18 tok/s (bottlenecked by ethernet)
-
-**2-node 27B benchmark estimate** (inferred from scaling patterns):
- **10GbE with PP=2:** ~15-25 tok/s at batch_size=8 — adequate for low-to-moderate QPS
- **2.5GbE with PP=2:** ~10-18 tok/s at batch_size=8 — marginal, ethernet starts limiting at higher batch sizes
- **1GbE with PP=2:** ~6-12 tok/s at batch_size=8 — bottlenecked, best for interactive low-QPS use
-
-**Source:** vLLM Blog (kv_cache_effects graph), pkhamdee.blog, dreaming.press
-
-### 5. Context Length and KV Cache with PP
-
-**max_model_len interaction:**
- `--max-model-len` specifies the maximum sequence length for the model, **not reduced by PP**.
- However, KV cache is split across PP stages. Each stage owns its layer subset's KV cache.
- Total KV cache capacity = sum across all stages. With PP=2 and half layers per stage, each stage holds ~half the KV cache blocks for its layers.
-
-**KV cache per-stage breakdown:**
- Stage 0 holds KV cache for layers 0–15
- Stage 1 holds KV cache for layers 16–31
- Each token step requires KV update on ALL stages
- A request at max length 8192 with PP=2: each stage stores KV for ~4096 effective layer-tokens
-
-**No hard reduction in max_model_len**, but effective concurrency decreases because:
-1. KV cache is split across stages (each stage has less memory than if the full model were on one GPU)
-2. PP bubble reduces throughput at low batch sizes
-
-**Source:** `vllm/v1/core/kv_cache_coordinator.py`, `serverAppConfigs-docs/serving/parallelism_scaling.md`
-
-### 6. Speculative Decoding with PP
-
-**Known incompatibility** (`serverAppConfigs-docs/features/speculative_decoding/README.md`):
-
-> "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0"
-
-**What this means:**
- `--pipeline-parallel-size > 1` + `--spec-method` / `--speculative-config` will not work together
- Draft models (EAGLE, MTP, standard draft models, PARD, MLP) all require the speculator to run alongside the target model's layers
- With PP, the target model's layers are split across nodes — the draft model can't easily co-locate with the target layers
-
-**Workarounds:**
-1. **Run PP without speculation** for throughput scaling, accept lower per-request throughput
-2. **Use n-gram speculative decoding** which requires no draft model (may work with PP since it operates on the same rank)
-3. **Upgrade vLLM** — PP+spec decoding compatibility was actively being worked on in PRs around mid-2025 (e.g., the PP V2 PR #33960 added PP support to Model Runner V2)
-
-**For your Qwen3.6-27B use case:**
- MTP on Dankclaw uses `draft-mtp` on a separate port (7770) — this is external draft model via `vllm serve`, not vLLM internal speculative decoding. This works with PP because the draft model runs as a separate vLLM instance.
- DFlash (retired) was also an external drafter
- n-gram internal speculation may work with PP since it doesn't need a separate model
-
-**Source:** `serverAppConfigs-docs/features/speculative_decoding/README.md`, `vllm/config/speculative.py`
-
-## Verdict: 2-Node vLLM PP over Ethernet for 27B Model
-
-**Is it practical?** Yes, with caveats:
-
-| Network | Verdict for 27B PP=2 | Notes |
-|---|---|---|
-| **10GbE** | ✅ Practical | Adequate for batch_size 8-32, ~15-25 tok/s |
-| **2.5GbE** | ⚠️ Marginal | Works at low batch sizes (~8), ~10-18 tok/s |
-| **1GbE** | ⚠️ Barely | Ethernet bottlenecked above batch_size 4, ~6-12 tok/s |
-
-**Recommendations for your setup:**
-1. Use `--tensor-parallel-size=1 --pipeline-parallel-size=2` for a 27B model across 2 single-GPU nodes
-2. Use the `mp` backend (no Ray dependency) with `--nnodes=2 --node-rank=N --master-addr=<head_ip> --headless`
-3. Set `NCCL_SOCKET_IFNAME=eth0` to pin the correct NIC
-4. Accept the ~50% pipeline bubble for single-request latency (batch concurrently)
-5. PP + internal speculative decoding is NOT supported (v0.15.0), but external draft models (MTP on port 7770) work fine
-6. KV cache max_model_len is NOT reduced by PP, but per-stage KV cache is split
-
-## Sources
-
- **Kept:** Parallelism and Scaling docs — vLLM official guidance on TP/PP strategy, `serverAppConfigs-docs/serving/parallelism_scaling.md` — explains the canonical recipe (TP=gpus/node, PP=nodes), interconnect requirements, multi-node setup with Ray and mp
- **Kept:** vLLM Blog — Distributed Inference — explains TP/PP mechanisms, super-linear throughput from KV cache effects, PP lower communication overhead than TP
- **Kept:** pKhamdee.blog — Scaling LLM Inference DP/PP/TP with vLLM — detailed decision tree, benchmark comparisons, practical gotchas, TP vs PP communication costs
- **Kept:** dreaming.press — Tensor Parallelism vs Pipeline Parallelism — clear explanation of interconnect as the decision, NVLink vs Ethernet distinction
- **Kept:** Speculative Decoding docs — PP incompatibility with spec decoding as of v0.15.0, full spec method catalog, --speculative-config schema
- **Kept:** parallel.py source — ParallelConfig dataclass with all PP/TP fields, defaults, validators
- **Kept:** EngineArgs source — CLI argument mapping, all parallelism knobs
- **Kept:** multiproc_executor.py / ray_executor.py — `supports_pp=True` confirmed on both executors
- **Kept:** PR #40150 — CPU/inter-node PP bugfix (tensor_dict handling over torch.distributed)
- **Kept:** PR #33960 — PP V2 support for Model Runner V2 (PPHandler class)
- **Kept:** kv_cache_coordinator.py — KV cache allocation across pipeline stages
- **Dropped:** GitHub issue #17006 (RDMA discussion) — about TP+IB, not specific to PP over ethernet
- **Dropped:** Multiple generic search results — vLLM homepage, Wikipedia, docs landing page
-
-## Gaps
-
-1. **Exact multi-node PP benchmark numbers over 1/2.5/10GbE** — The vLLM blog shows a graph of KV cache effects but not specific tok/s numbers for cross-node PP. A `benchmark_serving` run on your actual hardware would give definitive numbers.
-2. **PP + speculative decoding support in v0.16.0+** — The docs say "as of v0.15.0" PP is incompatible with spec decode. The PP V2 PR (#33960) may have resolved this. Need to check if the current master branch supports it.
-3. **KV cache split ratio** — Exactly how many KV blocks per stage? Is it strictly by layers or does vLLM do any balancing?
-4. **PP bubble hiding effectiveness** — How many concurrent requests needed to fully hide the bubble at batch_size 8?
-
-## Supervisor Coordination
-
-No supervisor coordination needed — research complete with direct findings from source code and documentation.
-
---
-
-*Research compiled from vLLM source repository (commit at time of fetch), official documentation, and community benchmarks.*
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page maps LLM quantization formats to the GPU and CPU architectures that support them."
+tags:
+  - llm
+  - quantization
+timestamp: 2026-08-28
+---
+
 # Quantization by Hardware Architecture

 ## Overview
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "vLLM uses CPU RAM and swap space to extend GPU memory capacity."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # RAM CPU Offloading in vLLM

 ## Overview
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Running Qwen3.6 27B FP8 on Dankclaw

 ## Model Information
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Running Qwen3.6 35B A3B FP8 on Dankserver

 > **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankclaw uses AMD Strix Halo APU with unified memory architecture."
+tags:
+  - llm
+timestamp: 2026-08-28
+---
+
 # Shared Memory Architecture Dankclaw

 ## Overview
@@ -142,7 +150,7 @@ Unified memory has lower bandwidth than dedicated VRAM:

 ### ROCm Overcommit

-ROCm reports 96 GiB GPU memory on Strix Halo APU. Physical RAM is 64 GiB (61 GiB visible; corrected 2026-08-17, the box was upgraded from 32 GB). The GPU can address more than physical RAM exists. vLLM will see 96 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
+ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture, `gtt_total=122402963456` bytes = 114 GiB; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM is 64 GiB (61 GiB visible; the box was upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.

 Lower bandwidth means slower inference. Plan for lower throughput.

@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This index contains all documentation for vLLM deployment on Dankserver and Dankclaw."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # vLLM Index

 > **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[ciru-activefpx-promptforge]].
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "vLLM is a fast and easy library for large language model inference and serving."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # vLLM Overview

 ## What Is vLLM
@@ -14,10 +14,11 @@ timestamp: 2026-08-22

 # CIRU ActiveFPX PromptForge on Dankclaw

-> **Active 2026-08-22.** `frank-large` serves this v3 model again
-> after a same-day Kairic Edge IU4 experiment was reverted (operator
-> decision). Commit `c60156f` restored the v3 script pair. See
-> [[kairic-edge]] for the retired experiment reference.
+> **RETIRED 2026-08-24.** `frank-large` served this v3 model from
+> 2026-08-22 (restored after a same-day Kairic Edge IU4 experiment was
+> reverted — operator decision, commit `c60156f`; see [[kairic-edge]])
+> until [[eemin-rocmi4-mtp]] took over on 2026-08-24. Kept as the
+> historical record of the v3 deployment.

 `frank-large` (port 7770, moved from 52625 on 2026-08-22) serves
 `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
@@ -1,7 +1,3 @@
-> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
-> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
-> [deployment.md](deployment.md) for the active configuration.
-
 ---
 type: reference
 title: Dankserver VRAM and RAM Growth Under Load
@@ -13,8 +9,12 @@ tags:
  - ram
  - ctx-checkpoints
  - crash
+timestamp: 2026-09-03
 ---

+> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
+> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
+
 # Dankserver VRAM and RAM Growth Under Load

 This page records the investigation of the recurring cuBLAS crash on
@@ -21,7 +21,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.
 |---|---|---|---:|
 | Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
 | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
-| Dankclaw | `dank@dankclaw` | `frank-npu` (FLM on the XDNA2 NPU) | 7771 |
+| Dankclaw | `dank@dankclaw` | `frank-small` | 7771 |

 ## Scoped workflow

@@ -47,49 +47,49 @@ Run the unified deployer from the repository root:
 Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
 on the selected host. The following files select the corresponding service:

- `scripts/frank-medium.env`, `frank-medium.service`, and
-  `serve-frank-medium.sh` select `frank-medium` on Dankserver.
+- Any file under `dankserver/frank-medium/` (the Quadlet unit, its
+  entrypoint and its healthcheck) selects `frank-medium` on Dankserver —
+  the container mounts the entrypoint and healthcheck straight out of the
+  checkout, so any file there changes deployed state.
 - `scripts/frank-large.env`, `frank-large.service`, and
  `serve-frank-large.sh` select `frank-large` on Dankclaw.
- `scripts/frank-npu.env`, `frank-npu.service`,
-  and `serve-frank-npu.sh` select `frank-npu`
-  on Dankclaw.
- `scripts/chat-template.jinja` selects the LLM on the selected host only.
+- `scripts/frank-small.env`, `frank-small.service`, and
+  `serve-frank-small.sh` select `frank-small` on Dankclaw.
 - Documentation, tests, deployment tooling, and unrelated container changes
-  do not select either inference service.
+  do not select any inference service.

-The target-side `scripts/deploy-remote.sh` installs the selected service: for a
-**root unit** it relinks the repo file into `/etc/systemd/system/`, reloads
-systemd, enables it and restarts it (frank-small); for a **Quadlet** it retires
-the predecessor, stops the service's own unit, waits for the published port to
-clear, installs the `.container` and restarts the generated unit (frank-large).
-Each handler then waits on that service's health endpoint —
-`http://127.0.0.1:7770/health` for frank-large, `http://127.0.0.1:7771/health`
-for frank-small. The wait budget is 20 minutes because cold model loads are
-slow: halogen pins ~68 GiB of weights before it serves anything. A failed
-handler does not advance its per-service state.
+The target-side `scripts/deploy-remote.sh` installs the selected service.
+`frank-large` and `frank-small` are **root units**: the handler relinks the
+repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
+enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
+handler retires its predecessors (the replica Quadlets and the retired llama
+root unit), stops the service's own unit, waits for the published port to
+clear, installs the `.container` and restarts the generated unit. Each
+handler then waits on that service's health endpoint —
+`http://127.0.0.1:7770/health` for frank-large and frank-medium,
+`http://127.0.0.1:7771/health` for frank-small. The wait budget is 20 minutes
+because cold model loads are slow. A failed handler does not advance its
+per-service state.

 ## Unit and artifact paths

-The units point into the stable target checkout:
+The root units point into the stable target checkout:
 ```text
-/home/dank/serverAppConfigs/scripts/frank-medium.service
+/home/dank/serverAppConfigs/scripts/frank-large.service
 /home/dank/serverAppConfigs/scripts/frank-small.service
 ```
+`frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
+(`frank-medium.container`) rather than a unit in `scripts/`.

 The unit files and serve scripts validate their model/runtime artifacts before
-starting. `frank-large` is a Quadlet under `dankclaw/frank-large/quadlet/`
-rather than a unit in `scripts/`; `frank-small` keeps the unit/env/serve trio
-above. Keep `REMOTE_DIR` stable; moving the checkout requires updating both the
-unit absolute paths and the deployment tests in the same change.
+starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
+the unit absolute paths and the deployment tests in the same change.

-## NPU unit
+## Retired endpoints

-`frank-npu` is an inactive, historical Dankclaw endpoint. It is
-relinked only when explicitly selected with `--service frank-npu`; the
-large-model handler never restarts it. This is intentional: the NPU is not part
-of the active inference path and should not be disturbed by a large-model
-change.
+`frank-npu` (the FastFlowLM XDNA2 NPU endpoint) was retired on 2026-09-11
+with the `amd_iommu=off` decision and is no longer a deployable service — the
+planner has no `frank-npu` entry. Its runbook is [[frank-npu]].

 ## Isolation guarantee

@@ -110,9 +110,10 @@ change in that service's inputs.

 If the LLM handler fails after the target pull:

-1. Inspect `journalctl -u frank-small -n 200` (root unit) or
-   `journalctl --user -u frank-large -n 200` (rootless Quadlet).
-2. Confirm the model files, the runtime binary and (for frank-large) the
+1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-small
+   -n 200` (root units), or `journalctl --user -u frank-medium -n 200`
+   (rootless Quadlet).
+2. Confirm the model files, the runtime binary and (for frank-medium) the
   container image are present and readable by `dank`.
 3. Retry only the affected service:

@@ -15,10 +15,12 @@ timestamp: 2026-08-25

 # caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw

-> **Active 2026-08-25.** `frank-large` serves this model. It replaced
-> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same
-> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See
-> [[kingjones-strix-mtp]] for the retired page.
+> **RETIRED 2026-08-25.** `frank-large` served this model from 2026-08-18
+> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
+> on the same port and the same Strix Halo iGPU, running the new
+> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
+> the page it replaced), and was itself replaced by
+> [[kingjones-strix-rocmfp4]].

 `frank-large` (port 7770) serves
 `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page records the benchmarks that chose the runtime for `frank-large` on Dankclaw."
+tags:
+  - llm
+  - frank
+timestamp: 2026-09-03
+---
+
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
 > [deployment.md](deployment.md) for the active configuration.
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: CIRU v4.4.1 Orca on Dankclaw (frank-large)
-description: The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
+title: "CIRU v4.4.1 Orca on Dankclaw (frank-large)"
+description: "The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - ciru
@@ -14,208 +14,3 @@ tags:
  - mtp
 timestamp: 2026-09-19
 ---
-
-# CIRU v4.4.1 Orca on Dankclaw (frank-large)
-
-> **RETIRED 2026-09-21** (replaced by
-> [[frank-large-unsloth-qwen38-flash-next]]; **rollback target**). It was
-> ACTIVE since 2026-09-19 and replaced halogen-flash-server (see
-> [[halogen-flash-server]]), which had replaced ik_llama.cpp (see
-> [[frank-large-ik-uncensored]]). It was deployed as a **root systemd
-> unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
-> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it did not touch
-> belongs to [[frank-small-gemma4-12b]].
-
-CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp")
-architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a **QSA window
-hotfix** that is what makes vision and the MTP drafter work on this build; the
-pre-applied hotfix source ships in the model repo under
-`runtime/v4.4.1/qsa-window-hotfix/`. Like every other fleet endpoint it is a
-llama.cpp runtime — `llama-bench` is its own harness.
-
-## Shape
-
-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730
-and an OpenAI front-end on 8731), CIRU is a single `llama-server` process —
-there is no separate engine port to keep private.
-
-`/v1/models` reports the loaded model name, **`Qwen3.8-Flash-CIRU-STRIX-Orca`**
-(the launcher sets no `--alias`). The endpoint serves **whatever model name a
-request names** — llama.cpp does not reject an unknown id — so the harness's
-`frank-large` label in `models.yml` works without a rename.
-
-## Runtime
-
-A **private host build**, not a container image. The v4.4.1 source with the QSA
-window hotfix, built on the host against the private ROCm SDK that
-`scripts/ciru/setup-linux-amd.sh` installs into the checkout's own venv:
-
-| path | what |
-|---|---|
-| `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/` | the checkout (source + build + venv) |
-| `build-gfx1151-sdk/bin/llama-server` | the built binary |
-| `.venv-rocm/` | the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH |
-| `scripts/ciru/run-orca-server.sh` | the launcher (`exec`'d by the serve script) |
-| `profiles/strix-halo-production.env` | the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER `frank-large.env` (every line is `${VAR:-default}`, so a value set in `frank-large.env` wins) |
-
-### The runtime gate
-
-`run-orca-server.sh` sources `run-server.sh`, which **exits 2** unless
-`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
-libhsa-runtime64.so` exist (unversioned, no suffix). It prepends both to
-`LD_LIBRARY_PATH` and `ldd`-checks that the built `libllama-common.so` carries
-the "Qwen4Exp MTP image position" marker and the HIP lib carries the
-`flash_attn_index_mask_clear/set/empty` symbols before exec.
-
-The build links the venv through RUNPATH, so those two files are an existence
-gate, not a real dependency. `serve-frank-large.sh` satisfies the gate
-idempotently: it finds the venv's unversioned devel symlinks
-(`.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/`) and `ln -sfn`-s
-both into `runtime/{hip,rocr}/lib/`. **The first deploy of this endpoint failed
-exactly here** — the build linked the venv but `runtime/` did not exist, so the
-launcher exited 2 and the unit crash-looped until the serve script created the
-symlinks.
-
-## Payload
-
-`scripts/download-frank-large-orca.sh` assembles `jcbtc/Qwen3.8-Flash-CIRU-
-STRIX-Orca` (HuggingFace, public and ungated) into
-`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/`, sha256-verified. The
-launcher refuses to start without the main GGUF, the MTP head and the PLE
-sidecar:
-
-| file | bytes | role |
-|---|---|---|
-| `Qwen3.8-Flash-CIRU-STRIX-Orca.gguf` | 79,397,818,912 | the main model (Orca variant) |
-| `mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf` | 4,135,893,440 | the MTP drafter head |
-| `ple/ple.payload.bin` | 52,429,053,952 | the per-layer-embedding (PLE) payload — mmap'd and paged, not resident |
-| `ple/ple.manifest.json` | — | PLE manifest |
-| `ple/ple.scale.bf16` | — | PLE scale |
-| `vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj` | 904,004,320 | the F16 vision projector (on disk, `ENABLE_VISION=0`) |
-
-The main GGUF ships as **transport parts** that the download script assembles
-and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's
-own workers, not held in RAM.
-
-## Deployed config
-
-Tunables live in `scripts/frank-large.env`; the launcher owns the flag list
-(`-sm none` one iGPU no split, `-ngl all`, f16 KV, flash attention,
-cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle
-profile). The departures worth naming:
-
-| flag | value | why |
-|---|---|---|
-| `CONTEXT_SIZE` | `262144` | the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling |
-| `PARALLEL_SLOTS` | `1` | one stream — and the MTP drafter **requires exactly one slot** in this runtime, so the operator's ask and the drafter's constraint are the same number |
-| `ENABLE_MTP` | `1` | the MTP drafter, depth 4 (the Orca variant's own `mtp_depth_default`) |
-| `ENABLE_VISION` | `0` | text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is [[frank-small-gemma4-12b]] either way |
-| `PLE_CACHE_MIB` | `4096` | the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it |
-| `HOST` | `0.0.0.0` | the launcher consumes it for `--host`; the value doubles as a JIT-cache directory name, which is why `0.0.0.0` is safe (HIP reads `HOST` bare — see the AGENTS.md note) |
-
-`TEMP`, `HOST` and `GPU_MAX_HEAP_SIZE` are the three names to avoid for any
-new tunable: HIP reads them bare.
-
-## MTP
-
-The Multi-Token Prediction drafter is a **separate Q8_0 head** (not inside the
-main GGUF), loaded by the launcher. It requires exactly one slot, which is why
-`PARALLEL_SLOTS=1`. Depth 4 is the Orca variant's default.
-
-Verified at deploy (2026-09-19): the per-task `draft acceptance = 0.65625
-(21/32), mean len = 3.62` — the drafter is engaged and accepting, not silently
-standing down.
-
-## PLE
-
-The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next
-architecture's signature: token embeddings live outside the main weight set and
-are paged on demand by the engine's own workers (`GGML_QWEN4EXP_PLE_WORKERS=16`,
-`GGML_QWEN4EXP_PLE_IO_WORKERS=32` in the production profile). `PLE_CACHE_MIB=4096`
-is the hot window kept resident; the rest is mmap'd file cache. **Do not treat
-the 52.4 GB as resident memory** — `free` and the dashboards will not show it
-pinned, and that is correct.
-
-## Client contract
-
-The differences from a stock llama.cpp endpoint that bite:
-
- **`reasoning_content`** carries the reasoning trace (the standard OpenAI
-  field). The serve script passes `--chat-template-kwargs
-  '{"preserve_thinking":true}'` so prior thinking blocks are kept across turns;
-  a request-level `chat_template_kwargs` still wins.
- **`reasoning_effort` is gated.** Qwen3.8's template accepts only
-  `xhigh`, `medium`, `low` and **500s on anything else** (not a clamp). The
-  harness folds every level onto that set in `models.yml`
-  (`reasoningEffortMap`: minimal/low→low, medium→medium, high/max→xhigh) —
-  subagents run at `high`, which is why the map exists.
- **Text only.** `ENABLE_VISION=0`, so image content parts are refused. The
-  fleet's image path is [[frank-small-gemma4-12b]].
- **Sampling.** The server has no sampler defaults of its own; the harness
-  applies the per-model recipe from `models.yml` (temp 1.0 / top_p 0.95 / top_k
-  20 / min_p 0.0). A client that stays silent gets the launcher's default.
- **`tools` works** — the launcher passes `--jinja`, and the Qwen3.8 template
-  supplies the tool-call grammar. Verified at deploy: a tool call returns
-  `tool_calls`.
-
-## Deploy and operate
-
-```bash
-./scripts/deploy.sh dankclaw --service frank-large
-```
-
-The handler pulls the checkout (ff-only), (re)links the unit, and gates on
-`/health`. First boot is the slow one (model load ~46 s + graph capture); a
-warm restart is fast.
-
-```bash
-sudo systemctl status frank-large            # the root unit
-sudo journalctl -u frank-large -n 60         # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded
-curl -s http://127.0.0.1:7770/health         # ok
-curl -s http://127.0.0.1:7770/v1/models      # Qwen3.8-Flash-CIRU-STRIX-Orca
-```
-
-**Stop and mask before any GPU experiment** (the AGENTS.md bench protocol):
-`sudo systemctl stop frank-large && sudo systemctl mask frank-large`. The
-frank-large serve process is a `llama-server` **and so is frank-small on the
-same host** — never `pkill -x llama-server`. If you must kill by pattern, use
-`pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca'` (the model path is unique to this
-endpoint and does not appear in the deploying shell's own command line).
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| launcher exits 2: "The v4.4 HIP/ROCr runtime is missing" | `runtime/{hip,rocr}/lib/*.so` absent — the serve script's `ln -sfn` did not run, or the venv was moved. Re-run the serve script or rebuild: `bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps` |
-| "missing SDK venv runtime under .../.venv-rocm" | the venv's unversioned devel symlinks are gone — rebuild the runtime |
-| "missing target model / missing payload" | a download did not finish — re-run `scripts/download-frank-large-orca.sh` |
-| loads, `/health` ok, first completion slow (~45-60 s) | the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it |
-| a subagent 500s naming `reasoning_effort` | the Qwen3.8 template gate — the harness `reasoningEffortMap` must fold the level onto {xhigh, medium, low} |
-| `free` / dashboards do not show 52 GB pinned | correct — the PLE is mmap'd file cache, not resident; only `PLE_CACHE_MIB` is hot |
-
-Retrieval: this is a **root** unit, so `journalctl -u frank-large` is the
-durable record (no `--user`, no `sudo` needed for the journal once the unit is
-readable). There is no container log — there is no container.
-
-## Rollback
-
-The predecessor is intact: the halogen payload stays at
-`/home/dank/models/halogen` (retired, not deleted), and the halogen quadlet is
-archived under `.AGENTTRASH/`. Rolling back is a `git revert` of this change set
-plus a deploy — **but the root unit and the halogen quadlet both claim port
-7770**, so the handler's port-free gate covers that race. See
-[[halogen-flash-server]].
-
-The `amd_iommu=off` kernel argument is separate and reverts on its own — see
-[[dankclaw-iommu-translation-off]].
-
-## Citations
-
- [jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca) — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball
- [CIRU runtime v4.4.1](https://github.com/ciru) — `scripts/ciru/run-server.sh` (runtime gate), `profiles/strix-halo-production.env` (kernel bundle), `docs/BUILD_LINUX.md`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
-description: The active frank-large runbook — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing.
+title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - kairic-edge
@@ -18,195 +18,31 @@ tags:
 timestamp: 2026-09-21
 ---

-# Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
+# frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)

-> **ACTIVE since 2026-09-21.** It replaced the unsloth llama.cpp prebuilt
-> (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE — see
-> [[frank-large-unsloth-qwen38-flash-next]], the current rollback target),
-> which had replaced the CIRU v4.4.1 Orca runtime (see
-> [[frank-large-ciru-orca]]). Deployed as a **root systemd unit**
-> (`scripts/frank-large.service`, `User=dank`), port 7770, on the same
-> gfx1151 iGPU. The RTX 3090 it does not touch belongs to
-> [[frank-small-gemma4-12b]].
+> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
+> runbook on 2026-09-21 before it was replaced by
+> [[frank-large-unsloth-qwen38-27b-q8xl]] later the same day. The payload
+> and runtime remain on disk as the rollback target.

-The model is `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` — a **27B dense** Qwen3.8
-at **IU4** (W4A4, integer-weights / integer-activations). It is a
-fundamentally different architecture from the 125B MoE it replaces: no
-PLE, no MoE experts, no n-gram embedding. The payload is four files: the
-15.5 GiB main GGUF plus three `.pfs` sidecars (FFN 7.99 GiB, GDN 1.88
-GiB, GDN-Output 0.70 GiB) — the PromptForge IU4 weight shards the Kairic
-Edge runtime loads alongside the main GGUF.
+## What it served

-The runtime is the **Kairic Edge** build of `ciru-ai/ROCmFPX` (branch
-`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`), built from source
-(`scripts/build-kairic-edge-gfx1151.sh`). It is a llama.cpp build with the
-PromptForge sidecar env block and the `--kairic-edge` qualified serving
-path.
-
-## Why this swap
-
- The operator asked for the Kairic Edge 27B model at **one conversation,
-  full context (262,144)** (2026-09-21).
- The Kairic Edge runtime is the qualified IU4 (W4A4) serving path — no
-  standard llama.cpp quant, no PLE sidecar payload, no QSA hotfix: one
-  GGUF, three sidecars, done.
- The spec method is native MTP depth 4 + n-gram drafting (24/64/64).
-  DFlash is NOT available for this model (no `DRAFT_DFLASH` spec type in
-  the Kairic Edge runtime).
-
-## Shape
-
-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-## Runtime
-
- **Source build**: `scripts/build-kairic-edge-gfx1151.sh` (TheRock
-  7.15.0a20260718 toolchain, AMD clang 23.0.0, HIP 7.15.0, GCC 13.3.0,
-  CMake 4.4.2, Ninja 1.13.0). Output at
-  `/home/dank/GitPackages/ROCmFPX-kairic/build-kairic/bin/llama-server`.
- **Composable Kernel**: `fdf4bb7fcc984811cef48ce817d89aac064b984a` +
-  `patches/composable-kernel-gfx1151-iu4.patch` (SHA-256
-  `ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5`).
-  The CK `config.h` is generated by CMake's `configure_file` (line 334 of
-  CK's CMakeLists.txt) — it is untracked, so a `git clean` removes it.
- **GPU target**: `gfx1151` (Strix Halo). The build takes ~4 minutes
-  (`JOBS=$(nproc)`).
- **v1.2 changes**: Exact 65-row n-gram verification switched to the compact
-  authoritative path by default; the native IU4 M65 verifier is retained
-  only for diagnostics.
-
-## Spec method
-
- **Native MTP depth 4** (`--spec-type draft-mtp --spec-draft-n-max 4`):
-  the model's MTP layers draft 4 tokens per step.
- **N-gram drafting** (24/64/64): the `ngram-mod` matcher chains behind
-  the MTP drafter (the `--spec-type draft-mtp` value builds both).
- **DFlash is NOT available**: the Kairic Edge runtime's spec-type
-  constants are `DRAFT_EAGLE`, `DRAFT_MTP`, `DRAFT_SIMPLE`,
-  `NGRAM_CACHE`, `NGRAM_MAP_K`, `NGRAM_MOD`, `NGRAM_SIMPLE`, `NONE` — no
-  `DRAFT_DFLASH`. The z-lab DFlash2-FP8 drafter (frank-medium's drafter)
-  cannot be used.
- **Draft device**: `--spec-draft-device ROCm0 --spec-draft-ngl 999`
-  (the drafter runs on the iGPU, all layers).
-
-## Compatibility mode
-
-The Kairic Edge runtime has two modes:
-
- **Fast-greedy** (`KAIRIC_EDGE_COMPATIBILITY_MODE=0`, the runner's
-  default): rejects request features that require full target logits,
-  including sampling, penalties, probabilities, grammar-constrained tool
-  calls, logit bias, LoRA, and a reasoning budget.
- **Compatibility** (`KAIRIC_EDGE_COMPATIBILITY_MODE=1`): enables all of
-  the above (9.7% slower per the build guide).
-
-The harness (OMP) needs sampling, tools and a reasoning budget, so
-`COMPATIBILITY_MODE=1` is the deployed shape. In compatibility mode:
-`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0` and
-`PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0`.
-
-**Known gotcha**: the greedy-argmax fastpath (fast mode) rejects
-non-greedy requests and reasoning budgets. Re-test chat before production
-if you flip to fast mode.
-
-## PromptForge sidecars
-
-The three `.pfs` sidecars are wired via the `PROMPTFORGE_*` env block
-(the certified runner's exact env):
-
-| sidecar | env var | size |
-|---|---|---|
-| FFN | `PROMPTFORGE_IU4_SIDECAR` / `PROMPTFORGE_SIDECAR` | 7.99 GiB |
-| GDN | `PROMPTFORGE_GDN_SIDECAR` | 1.88 GiB |
-| GDN-Output | `PROMPTFORGE_GDN_OUTPUT_SIDECAR` | 0.70 GiB |
-
-The serve script (`scripts/serve-frank-large.sh`) validates binary → main
-GGUF → 3 sidecars and exits with a pointer message if any is missing, then
-`exec`s `llama-server` with `--jinja --alias frank-large
--chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file
-$TEMPLATE_FILE` (Qwen3.8's template needs `preserve_thinking` to carry
-thinking blocks across turns; a request-level value still wins).
-
-## Chat template
-
-The serve script passes `--chat-template-file` pointing at
-`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
-v22.4.0 template, the same one frank-medium uses). Without this flag the
-binary falls back to the built-in `peg-native` template, which produces a
-~73,600-token prompt for a ~250-token request (280x the expected size).
-With the sharp template wired in, the same request produces a 62–175-token
-prompt. The `Chat format: peg-native` log line is the format-type label,
-not the template indicator — the sharp template is active even when that
-line appears.
-
-## Memory budget
-
-The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
-`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
-"dedicated" is not the budget). The 27B dense model (26 GiB total) is
-mmap'd in with `-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV +
-buffers), leaving ~84 GiB of aperture headroom. The 8 GiB prompt-cache
-budget (`CACHE_RAM=8192`) is the Kairic Edge qualified config. The model
-is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet `CACHE_RAM`
-floor does not apply.
-
-## Payload
-
-`scripts/download-frank-large-kairic.sh` (resumable, each file verified
-against its declared byte count + sha256; `--verify-only` re-checks):
-
-| file | size | sha256 |
-|---|---|---|
-| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 16,617,792,672 | `360caf73…` |
-| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 8,576,856,064 | `adcbb90a…` |
-| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 2,019,569,664 | `82f93131…` |
-| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 756,953,088 | `3b07e7b1…` |
-
-## Client contract
-
- **Port**: 7770 (the same port and alias the unsloth predecessor served).
- **Alias**: `frank-large`.
- **Context**: 262,144 (the model's native window), 1 slot.
- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
-  --reasoning-budget -1`), extracted to `reasoning_content`.
- **Vision**: text-only (the model has no vision projector). The fleet's
-  image path is [[frank-small-gemma4-12b]] either way.
- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
-  {xhigh, medium, low} — the harness folds its levels onto that set in
-  `models.yml` (`reasoningEffortMap`).
-
-## Measured performance
-
-10-question conversation (2026-09-21, post-deploy): **23.99s total, 530
-tokens, 22.1 t/s average**. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The
-MTP depth 4 + n-gram spec method is working (draft acceptance logged, both
-`ngram-mod` and `draft-mtp` active in the journal).
-
-**Template verification** (2026-09-21): a 250-token request produces a
-62–175-token prompt (not the 73,600-token prompt the built-in
-`peg-native` template produces). Draft acceptance 0.79 (mean 4.14), the
-model answers correctly, and reasoning lands in `reasoning_content`.
-
-## Failure modes worth knowing
-
- **CK `config.h` is untracked**: a `git clean` in the CK checkout removes
-  it, and the next build fails. Re-run the build script (it re-checks-out
-  the CK commit and re-applies the patch).
- **Fast-greedy mode rejects the harness**: if you flip to
-  `COMPATIBILITY_MODE=0`, the harness's sampling/tools/reasoning-budget
-  requests are rejected. Re-test chat before production.
- **The `charlie12345` fork is NOT the Kairic Edge repo**: the local
-  clone at `/home/dank/GitPackages/ROCmFPX` is the `charlie12345/ROCmFPX`
-  fork (no kairic branch). The Kairic Edge build uses the `ciru` remote
-  (`https://github.com/ciru-ai/ROCmFPX.git`), branch
-  `kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`, in the worktree at
-  `/home/dank/GitPackages/ROCmFPX-kairic/`.
+- **Runtime:** Kairic Edge (ciru-ai/ROCmFPX, branch
+  `kairic-edge-qwen38-27b-v1.2`, commit 205a3e5, TheRock 7.15.0,
+  Composable Kernel), root systemd unit, port 7770.
+- **Model:** `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (a 27B dense model, IU4
+  W4A4) — main GGUF + FFN/GDN/GDN-Output `.pfs` sidecars.
+- **Shape:** one 262,144-token conversation (single stream), native MTP
+  depth 4 + n-gram drafting, compatibility mode.
+- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.

 ## Rollback

-[[frank-large-unsloth-qwen38-flash-next]] — the unsloth prebuilt payload
-and runtime are intact on disk.
+Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
+for the current ACTIVE runbook and [[frank-large-unsloth-qwen38-flash-next]]
+for the predecessor.
+
+# Citations
+
+- Kairic Edge runtime: https://github.com/ciru-ai/ROCmFPX
+- Model: https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
-description: The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, UD-Q8_K_XL) with the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, fp8 KV, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing.
+title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
+description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - llama.cpp
@@ -13,145 +13,103 @@ tags:
  - q8
  - dflash2
  - rocm
-  - fp8-kv
+  - q8-kv
 timestamp: 2026-09-21
 ---

-# stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
+# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)

-> **ACTIVE since 2026-09-21.** It replaced the Kairic Edge runtime
-> (ciru-ai/ROCmFPX, jcbtc/Qwen3.8-27B-IU4-Kairic-Edge — see
-> [[frank-large-kairic-edge-qwen38-27b]], the current rollback target),
-> which had replaced the unsloth llama.cpp prebuilt serving
-> unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE — see
-> [[frank-large-unsloth-qwen38-flash-next]]). Deployed as a **root
-> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770,
-> on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
-> [[frank-small-gemma4-12b]].
+> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]]
+> (Kairic Edge IU4 W4A4, now the rollback target) and
+> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded).
+> Same port (7770), same alias (`frank-large`), same client contract surface.

-The model is `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a **27B dense**
-Qwen3.8 at **UD-Q8_K_XL** (a high-precision 8-bit quant). It is a qwen3_5
-hybrid: 64 layers total, 16 full attention + 48 Gated DeltaNet linear
-attention. The payload is two files: the 31.46 GiB main GGUF and the 1.14
-GiB DFlash2 Q4_K_M drafter.
+## Identity

-The runtime is the **stock llama.cpp** prebuilt (unsloth b11030, ROCm
-gfx1151) at `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`.
-It is a mainline llama.cpp build that supports the qwen3_5 architecture
-and the DFlash2 drafter (llama_model_qwen35, llama_model_dflash,
-common_speculative_impl_draft_dflash).
-
-## Why this swap
-
- The operator asked for the 27B model at **one conversation, full
-  context (262,144)** with DFlash2 spec decode (2026-09-21).
- The stock llama.cpp prebuilt is the only stock-compatible runtime with
-  qwen3_5 + DFlash2 support — no source build, no CK `config.h`, no
-  PromptForge sidecars, no compatibility mode.
- The spec method is DFlash2 (a block-diffusion drafter) at depth 4. The
-  drafter has a sliding window of 2048 (small draft KV, ~40 MiB).
+- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
+- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
+  (`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
+  A mainline build, not a fork: it supports the `qwen3_5` architecture and the
+  DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
+  `common_speculative_impl_draft_dflash`).
+- **Service shape:** root systemd unit `scripts/frank-large.service`
+  (`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
+  `scripts/frank-large.env`.

 ## Shape

-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-## Runtime
-
- **Prebuilt**: `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`
-  (mainline, ROCm gfx1151 prebuilt, version 0.4.1-dev, build 11030,
-  commit 6ba30d05b). The build bundles its own ROCm runtime — no host SDK
-  gate, no venv.
- **GPU target**: `gfx1151` (Strix Halo).
- **Spec-draft flags**: `--spec-draft-type-k`, `--spec-draft-type-v`,
-  `--spec-draft-threads`, `--spec-draft-cpu-mask`, `--spec-draft-prio`,
-  `--spec-draft-poll` all present (confirmed via `--help`).
+- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
+  31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
+- **Context:** one stream, full 262,144 tokens (the model's native window),
+  `PARALLEL_SLOTS=1`.
+- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
+  weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
+  compute buffers fit the iGPU's GTT aperture (~114 GiB).

 ## Spec method

- **DFlash2 depth 4** (`--spec-type draft-dflash --model-draft <drafter>
-  --spec-draft-n-max 4 --spec-draft-n-min 0`): the block-diffusion
-  drafter proposes 4 tokens per step.
- **Draft device**: `--spec-draft-device ROCm0` (the drafter runs on the
-  iGPU).
- **DFlash2 is a block-diffusion drafter** (not a standalone LM). The
-  drafter has a sliding window of 2048 (~40 MiB of draft KV).
+DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
+`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
+`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
+window of 2048, so its draft KV is small (~40 MiB). Greedy verification
+(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.

-## Chat template
+## KV

-The serve script passes `--chat-template-file` pointing at
-`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
-v22.4.0 template, the same one frank-medium uses). Without this flag the
-binary falls back to the built-in `peg-native` template, which produces a
-~73,600-token prompt for a ~250-token request (280x the expected size).
-With the sharp template wired in, the same request produces a 62–175-token
-prompt. The `Chat format: peg-native` log line is the format-type label,
-not the template indicator — the sharp template is active even when that
-line appears.
+**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
+unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
+q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.

-## Memory budget
+## Template

-The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
-`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
-"dedicated" is not the budget). The 27B dense model (31.46 GiB GGUF) is
-mmap'd in with `-ngl 999`; budget at 262k ≈ 41–43 GiB (weights + drafter
-+ fp8 KV + buffers), leaving ~71–73 GiB of aperture headroom. The 8 GiB
-prompt-cache budget (`CACHE_RAM=8192`) is the qualified config. The model
-is a 27B hybrid (16 full attention + 48 Gated DeltaNet); its DeltaNet
-state is smaller than a primarily-DeltaNet model, so the fleet's 12.8
-GiB DeltaNet `CACHE_RAM` floor does not apply.
-
-**KV budget**: 16 full-attention layers × 2 (K+V) × 4 kv_heads × 256
-head_dim × 1 byte (fp8) = 32,768 bytes/token. 262,144 tokens × 32,768 ≈
-8 GiB for attention KV. Plus 48 DeltaNet recurrent states (small).
+The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
+(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
+`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
+`--chat-template-file` the binary falls back to the built-in `peg-native`
+template, which produces a ~73,600-token prompt for a ~250-token request.
+`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
+agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
+harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).

 ## Payload

-`scripts/download-frank-large.sh` (resumable, each file verified against
-its declared byte count + sha256; `--verify-only` re-checks):
-
-| file | size | sha256 |
-|---|---|---|
-| `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31,457,991,680 | `af36ecb6…` |
-| `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 1,143,006,816 | `1a25c568…` |
+`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
+drafter, each verified against its declared byte count + sha256
+(`--verify-only` re-checks). The serve script refuses to start if the binary,
+the main GGUF, or the drafter is missing.

 ## Client contract

- **Port**: 7770 (the same port and alias the prior deployments served).
- **Alias**: `frank-large`.
- **Context**: 262,144 (the model's native window), 1 slot.
- **KV**: fp8 quantized (`--cache-type-k fp8 --cache-type-v fp8`).
- **Sampling**: greedy (`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) —
-  the DFlash2 drafter needs greedy verification.
- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
-  --reasoning-budget -1`), extracted to `reasoning_content`.
- **Vision**: text-only (the model has a vision projector — mmproj-BF16
-  is available — but this deployment serves text-only, matching the prior
-  frank-large). The fleet's image path is [[frank-small-gemma4-12b]]
-  either way.
- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
-  {xhigh, medium, low} — the harness folds its levels onto that set in
-  `models.yml` (`reasoningEffortMap`).
-
-## Measured performance
-
-(To be filled in after deploy.)
+- **Alias:** `frank-large` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
+  `--reasoning-format deepseek`, unlimited budget).
+- **Tools:** supported (`--jinja` is pinned).
+- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
+- **Sampling:** greedy by default; a client that sends a temperature gets its
+  own, and the harness applies per-model overrides from `models.yml`.

 ## Failure modes worth knowing

- **The model has `mtp_num_hidden_layers: 1`** but this deployment uses
-  DFlash2 (not MTP) for spec decode. The MTP layer is unused.
- **The model has vision** (`language_model_only: false`, image_token_id
-  248056, mmproj-BF16 0.93 GB available) but this deployment serves
-  text-only.
- **DFlash2 needs greedy verification**: the serve script pins
-  `--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`. A non-greedy request
-  will not be verified correctly by the drafter.
+- **Model load is slow (minutes cold).** Health returns 503 while loading; do
+  not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
+- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
+  (the serve script uses the long form).
+- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
+  at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
+  gap; it is not a regression.
+- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
+  one-word answer, 0.519 on a varied explanatory prompt) — do not read a
+  single-request acceptance as the steady state.

 ## Rollback

-[[frank-large-kairic-edge-qwen38-27b]] — the Kairic Edge payload and
-runtime are intact on disk.
+Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
+IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
+for the Flash-Next predecessor it replaced on 2026-09-21.
+
+# Citations
+
+- unsloth prebuilts: https://huggingface.co/unsloth
+- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
+- DFlash2 drafter: https://huggingface.co/z-lab
+- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
-description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing.
+title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - unsloth
@@ -15,182 +15,33 @@ tags:
 timestamp: 2026-09-21
 ---

-# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
+# frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)

-> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime
-> (see [[frank-large-ciru-orca]], the current rollback target), which had
-> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a
-> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port
-> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
-> [[frank-small-gemma4-12b]].
+> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
+> runbook on 2026-09-21 before it was replaced by
+> [[frank-large-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
+> same day. The payload and runtime remain on disk as the rollback target.

-The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth
-dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its
-51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of
-12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token
-window, vision encoder. The runtime is the **unsloth prebuilt**
-`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an
-MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP
-graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for
-this model).
+## What it served

-## Why this swap
-
- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at
-  **one conversation, 200k context** (2026-09-21).
- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime,
-  no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at
-  concurrency 1, per the model card), which the CIRU fork also had but which
-  stock mainline does not.
-
-## Shape
-
-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-## Runtime
-
- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from
-  the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
-  (tag `b11030-mix-5ff778e` or newer; the MTP graph landed in
-  `b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime
-  (`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv
-  to symlink** — unlike the CIRU predecessor's runtime-root gate.
- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/`
-  (extracted tarball; `llama-server` at the top level).
- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030,
-  commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics
-  (116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the
-  iGPU before the cutover.
-
-## Model
-
- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the
-  `:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
-  `00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
-  (43,836,407,744 B) = **93,682,584,224 B**. The parts ARE the canonical
-  payload — the serve script points `-m` at part 1 and llama.cpp loads the
-  3-way split natively (parts 2–3 auto-discovered in the same directory).
-  Do **not** concatenate them into a single file to serve: part 1's header
-  carries `ggml.file_count=3`, so a byte-correct concatenation still
-  advertises itself as a split and the loader rejects the non-split
-  filename with `invalid split file name` (caught on 2026-09-21 — the
-  assembled file crash-looped the unit ~110 times before the diagnosis).
- **On disk** at `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
-  - `UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-0000{1,2,3}-of-00003.gguf` — the
-    main model, 3-way split (87.3 GiB)
-  - `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
-  - `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
-    `ENABLE_VISION=0`)
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
-  (`curl -fL -C -`), each part verified against its **LFS sha256 fetched
-  live from the Hub tree API** (not a file that could drift).
-  `--verify-only` re-checks without downloading.
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
-  there is no external PLE payload and no pager, unlike the CIRU Orca
-  packaging (79.4 GiB main + 52.4 GiB paged PLE).
-
-## Deployed config
-
-From `scripts/frank-large.env` (the flag list is owned by
-`scripts/serve-frank-large.sh`):
-
-| knob | value | note |
-|---|---|---|
-| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head |
-| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted |
-| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served |
-| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 |
-| `PARALLEL_SLOTS` | `1` | one conversation |
-| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates |
-| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") |
-| `ENABLE_VISION` | `0` | projector on disk, one env flip away |
-
-Serve flags: `-m <GGUF> -md <MTP head> --spec-type draft-mtp
--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large
--host 0.0.0.0 --port 7770 --chat-template-kwargs
-'{"preserve_thinking":true}'`.
-
-### The explicit `-md` (do not drop it)
-
-Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without
-`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main
-model's speed **with no error saying why**. The serve script gates on the
-head's presence and passes it explicitly.
-
-### Memory model
-
-The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the
-GTT aperture over the shared LPDDR (`gtt_total=122402963456` in
-`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`).
-The 87.3 GiB model is mmap'd into that aperture with
-`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB
-occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5
-(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5
-≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of
-LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB
-swap file is the backstop.
-
-## Client contract
- OpenAI-compatible chat completions at `:7770/v1/chat/completions`;
-  `/health` 200 when serving.
- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field —
-  this is a llama.cpp runtime, not vLLM's `reasoning`).
- **Tools** work via `--jinja` (the model's own template). The Qwen3.8
-  template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness
-  folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
-  `preserve_thinking` is on by default (request-level
-  `chat_template_kwargs` still wins).
- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The
-  fleet's image path is frank-small.
- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired
-  2026-09-11).
-
-## Troubleshooting
-
- **`missing runtime` / `missing target model` / `missing MTP head`** from
-  the serve script: the unit exits with a pointer instead of restart-looping.
-  Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or
-  re-extract the prebuilt.
- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold
-  (page-cache miss). Do not kill the process; `deploy.sh` waits up to 20
-  minutes (`LLM_WAIT`).
- **Serving at the main model's speed with no spec metrics**: the MTP head
-  is not attached — check the process cmdline for `-md` (see above).
- **`Address already in use` on 7770**: a stray manual run of the same model
-  holds the port. The deploy handler sweeps `pkill -f
-  'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
-  is also a `llama-server`) and gates on `wait_port_free 7770`.
- **`invalid split file name` at model load** (fails in ~50 ms, before any
-  weights are read): `-m` points at a file whose GGUF header advertises a
-  split (`ggml.file_count > 1`) but whose name is not a
-  `-NNNNN-of-NNNNN` part. This is what a concatenated split file does —
-  the concatenation is byte-correct but part 1's header still says
-  "3-way split". Fix: point `-m` at part 1
-  (`UD-IQ4_XS/...-00001-of-00003.gguf`) and let the loader pick up parts
-  2–3 from the same directory.
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
-  the box. If the co-tenants start thrashing, the levers are KV quant
-  (`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
-  (smaller draft state), not the weights.
+- **Runtime:** unsloth llama.cpp prebuilt `b11030-mix-5ff778e` (ROCm
+  gfx1151), root systemd unit, port 7770.
+- **Model:** `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (a 125B MoE / 6B
+  active), 87.3 GiB shipped as three split parts. The split parts ARE the
+  payload — concatenating them trips `invalid split file name` (part 1's
+  header advertises `file_count=3`).
+- **Shape:** one 200,000-token conversation (single stream), MTP depth 2 via
+  the explicit `-md` shared-Q8_0 head. Draft acceptance verified live
+  0.83–0.92.
+- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.

 ## Rollback

-The CIRU Orca payload and runtime are intact on disk:
-`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE
-payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`.
-Rollback is one commit restoring the CIRU `serve-frank-large.sh` +
-`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the
-predecessor's version is in git history), then `deploy.sh dankclaw
--service frank-large`. See [[frank-large-ciru-orca]].
+Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
+for the current ACTIVE runbook and [[frank-large-ciru-orca]] for the
+predecessor.

-## References
+# Citations

- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix
- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`)
- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries
+- unsloth prebuilts: https://huggingface.co/unsloth
+- Model: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
@@ -1,4 +1,5 @@
 ---
+timestamp: 2026-09-17
 type: reference
 title: frank-medium — Batch and Ubatch Sweep
 description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.
@@ -621,6 +621,8 @@ if that person is looking. That is the honest state of "alerting" here.

 ## Citations

+- vLLM upstream: https://github.com/vllm-project/vllm — the R4D one-shot
+  allreduce and the tensor-parallel path this deployment runs.
 - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer
  refusal; `rocclr/device/rocm/rocdevice.cpp:186` — `checkAtomicSupport`.
 - Linux `drivers/pci/pci.c` — `pci_enable_atomic_ops_to_root()` and its
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page records the full redo of the inference fleet on 2026-08-17."
+tags:
+  - llm
+  - frank
+timestamp: 2026-09-03
+---
+
 # Frank Program Redo — 2026-08-17

 This page records the full redo of the inference fleet on 2026-08-17.
@@ -1,4 +1,5 @@
 ---
+timestamp: 2026-09-03
 type: reference
 title: frank-small Benchmark — ncmoe and Batch Tuning
 description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M
-description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves.
+title: "frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
+description: "The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
 tags:
  - llm
  - frank-small
@@ -14,172 +14,107 @@ tags:
 timestamp: 2026-09-17
 ---

-# frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT)
+# frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

 > **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]]
-> (one-day rollback target). Same unit, same port, same client contract —
-> the model, the drafter shape and the KV type changed.
+> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
+> [[frank-small-vllm-gemma4]] (the same-day vLLM experiment, also the
+> rollback target). Same port (7771), same alias (`frank-small`), same
+> client contract surface.

 ## Identity

-| what | value |
-|---|---|
-| runtime | mainline llama.cpp master `37b53fd`, CUDA build **with `-DGGML_CUDA_FA_ALL_QUANTS=ON`** (rebuilt 2026-09-17 for q8 KV) |
-| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
-| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
-| model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M |
-| model dir | `/home/dank/models/Gemma4-12B-QAT/` |
-| port | 7771 (alias `frank-small`; `frank-large` keeps 7770) |
-| window | **4 slots x 262,144** (`CTX_SIZE=1048576`, `PARALLEL=4`); q8_0 K and V |
-| vision + audio | fleet image path via BF16 mmproj (`input: [text, image]`); `input_audio` transcribes verbatim (verified) |
+- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
+  `-DGGML_CUDA_FA_ALL_QUANTS=ON`
+  (`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
+  The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
+  collapses prefill ~12x on CUDA.
+- **Service shape:** root systemd unit `scripts/frank-small.service`
+  (`User=dank`), launched by `scripts/serve-frank-small.sh` with tunables from
+  `scripts/frank-small.env`.

-## Payload (all sha256-verified against the Hub)
+## Shape

-| file | bytes | sha256 (head…) |
-|---|---|---|
-| `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf` | 7,381,381,760 | `59656d74…` |
-| `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf` | 175,115,264 | `b59e8154…` |
-| `mtp-gemma-4-12B-it.gguf` | 253,707,328 | `c50c91c3…` |
+- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
+  Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
+  x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
+  token embeddings (PLE) inside the file, and a sliding-window attention
+  pattern (`attention.sliding_window = 1024`) — most layers cache only a
+  1024-position window, which is why the KV pool is a fraction of the
+  full-ctx x all-layers number.
+- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
+  `PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
+  maximises the TOTAL pool, which would shrink each slot below the full window
+  once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
+  full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
+  the card. 4 is the measured max.
+- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
+  only tokenizes and samples (its CPU belongs to the halogen engine serving
+  frank-large).

-`sha256.txt` beside them on the host carries the full digests.
+## Spec method

-## What the preflight found (GGUF headers, read over HTTP Range)
+The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
+`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
+with `--spec-type draft-mtp`. The same `--spec-type` value chains the
+draftless **ngram-mod** matcher, which attaches unconditionally; two
+context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
+the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
+depth 4 + poll 100 decoded WORSE at long context — the longer verification
+round costs more attention re-read than the extra accepted token earns).

- **Target** (`arch gemma4`, 48 blocks): ctx 262144, MHA — 16 kv heads x
-  512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30,
-  **sliding_window 1024**, `swa_type = STANDARD`, per-layer token embeddings
-  (PLE) via `LLM_KV_EMBEDDING_LENGTH_PER_LAYER`. 12 full-attention layers
-  every 4th (`[3,7,…,47]` implied by `full_attention_interval = 4`).
-  48 blocks maps to `LLM_TYPE_UNKNOWN` in the loader's size switch —
-  cosmetic only.
- **mmproj**: clip, vision 224/16 patches, **also carries an audio
-  encoder** (`has_audio_encoder=True`) — vision is the fleet contract,
-  and audio works: a JFK WAV sent as an `input_audio` content part was
-  transcribed verbatim (2026-09-17).
- **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure
-  assistant (no trunk). Its KV **shares** the target's last layers — the
-  share lambda at `llama-model.cpp:2662` references
-  `llama_model_n_layer(model_other) - 2` for SWA layers. Adding the
-  drafter therefore costs no second KV pool.
+## KV

-## KV arithmetic — why four full windows fit
+**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
+(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
+MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
+3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
+the >12.8 GiB floor does not apply.

-Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) =
-**16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the
-sliding layers cache only the window; the full-attention layers cache the
-full context — the pattern that makes the window cheap (the naive
-"all-global" worst case was 32 GiB at q8/262k, impossible).
+## Vision + audio

-Measured on the 3090 (24,576 MiB): one full-context slot idles at
-11,996 MiB, four at 22,346 MiB — 3,450 MiB per slot, ~8,546 MiB fixed.
-A fifth slot projects past the card, so **4 x 262,144 is the ceiling**.
-`CTX_SIZE` is set explicitly to `PARALLEL x 262144` rather than left for
-`--fit`: fit maximises the TOTAL pool and would divide a smaller window
-per slot at `parallel > 1`. The slot-grid seam in
-`tests/frank_small_unit.bats` pins both the full-window division and the
-budget inequality so an oversized bump fails pre-deploy, not in boot.
+`--mmproj` loads the BF16 projector
+(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
+52M params), which also carries an audio encoder. The fleet's image path
+lives here. Audio works: `input_audio` content parts transcribe verbatim
+(verified 2026-09-17 with a 16k mono WAV).

-## Spec-decode chain
+## Payload

-`--spec-type draft-mtp,ngram-mod` with `--model-draft
-$MODEL_DIR/mtp-gemma-4-12B-it.gguf`:
-
- `draft-mtp` is context-backed and reads the assistant sidecar. Two
-  context-backed drafters cannot share one process, so the chain carries
-  exactly one.
- `ngram-mod` is draftless and attaches unconditionally, catching
-  prompt-echo traffic MTP would miss. It prints **no startup banner** —
-  verify behaviorally via the per-task `draft acceptance` line, not the
-  load log.
- Measured at deploy: `draft acceptance = 0.38–0.46, mean len ≈ 2.2`
-  (lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a
-  much smaller model — the point of the swap was the VRAM and the window,
-  not acceptance).
-
-## Throughput tuning (measured 2026-09-17)
-
-Salted API probes at the deployed geometry (4 slots, q8 KV, flash attn),
-prefill ~254k tokens / decode at ~253k context:
-
-| batch (b/ub) | draft n-max / poll | pp t/s | tg@250k t/s | idle MiB |
-|---|---|---|---|---|
-| 2048/512 (old) | 3 / 50 | 912–924 | 90*–204 | 22,346 |
-| **8192/1024 (deployed)** | **3 / 50** | **962–965** | **226–302** | **23,300** |
-| 8192/1024 | 4 / 100 | 961 | 230–257 | 23,300 |
-| 4096/1024 | 4 / 100 | 959 | 205–277 | 23,300 |
-| 8192/2048 | — | — | — | OOM |
-
- **ubatch 1024 is the winner**: +5% max-length prefill, and decode at
-  250k clearly above the 512 baseline.
- **ubatch 2048 cannot run**: the MTP draft context needs ~3.3 GiB of pp
-  compute buffer at that ubatch against ~2.2 GiB headroom — it dies in
-  `common_speculative_init_result` after the main model loads.
- **Deeper draft and busy-wait HURT long-ctx decode**: draft n-max 4 with
-  poll 100 verified a longer round per token, and at 250k the extra
-  full-attention re-read outweighs the accepted-token gain (230–257 vs
-  257–302). Keep 3 / 50.
- **All ubatch-1024 variants idle identically** (23,300 MiB) — compute
-  buffers are reserved at max graph size regardless of the logical batch,
-  so 8192 costs nothing over 4096.
- **Concurrency caveat**: a long prefill monopolises the GPU — a decode
-  running beside a 254k-token prefill drops from ~240 to ~100–135 t/s.
-  Four slots share one engine with no per-slot QoS; that is inherent, not
-  a config fault.
- **No CUDA graphs in this build** — `--help` shows zero graph flags; the
-  decode-launch-overhead lever does not exist here.
-
-## Deployed flags
-
-Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`.
-Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`,
-`MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`,
-`DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`, `CACHE_RAM=2048`,
-`SPEC_TYPE=draft-mtp,ngram-mod`, `SPEC_DRAFT_N_MAX=3`, `POLL_LEVEL=50`,
-`CTX_SIZE=1048576` with `PARALLEL=4` (four full 262,144-token slots),
-`BATCH_SIZE=8192`, `UBATCH_SIZE=1024`, `NGL=999`, `FLASH_ATTN=on`,
-`--jinja`, `--metrics`, greedy `--temp 0.0` server default. The serve
-script refuses to start on any missing file.
+The main GGUF, the BF16 mmproj, and the MTP sidecar live under
+`/home/dank/models/Gemma4-12B-QAT`. The serve script refuses to start if any
+is missing (a missing file is an unfinished download, and the unit's restart
+policy would otherwise turn it into a crash loop that reads like a runtime
+fault).

 ## Client contract

- `reasoning_content` carries thinking; `preserve_thinking: true` is sent
-  by the harness to keep it across turns.
- **Every `reasoning_effort` level is accepted** (all seven probed
-  2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's
-  `reasoningEffortMap` in `~/.omp/agent/models.yml` is therefore inert
-  but kept as a safety net.
- Tools work through the `--jinja` template (`tool_calls` verified).
- Images work through the mmproj (verified with a live JPEG).
- Four concurrent conversations (one per slot). No per-slot QoS: a long
-  prefill in one slot slows decodes in the others — see the caveat above.
- Harness: `contextWindow: 262144`, `maxTokens: 32768`.
+- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
+  deepseek`, unlimited budget).
+- **Tools:** supported (`--jinja` is pinned even though this build defaults
+  it on, so a future build that flips the default cannot silently take tools
+  away).
+- **Vision + audio:** supported.
+- **Sampling:** greedy by default (fleet contract); a client that sends a
+  temperature gets its own, and the harness applies per-model overrides from
+  `models.yml`.

-## Troubleshooting
+## Measured (max length)

- **`Gemma4Assistant requires ctx_other to be set` printed at boot** —
-  the `--fit` probe, not a fault (fit cannot build the assistant context
-  during measurement). With `CTX_SIZE` set explicitly, as deployed, this
-  line does not appear at all. Do not restart over it either way.
- **Draft looks unengaged**: ngram-mod is silent at load; read the
-  `draft acceptance = …` line per task from
-  `journalctl -u frank-small` (the server must not be restarted just to
-  see this).
- **Quantized KV regression**: if someone rebuilds `build-cuda` without
-  `GGML_CUDA_FA_ALL_QUANTS=ON`, prefill collapses ~12x with q8 KV (see
-  the CUDA lesson in AGENTS.md). Check
-  `grep FA_ALL_QUANTS build-cuda/CMakeCache.txt` after any rebuild.
- Logs: `journalctl -u frank-small` (root unit, not quadlet).
+At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
+pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
+(2048) OOM the MTP draft context's pp compute buffer and must not be tried.

 ## Rollback

-Redeploying the prior model means repointing `MODEL_DIR`/`MODEL_FILE`/
-`DRAFT_FILE` back to the Qwen trio and `SPEC_TYPE=draft-mtp` — see
-[[frank-small-qwen38-gsq-rco]] for the exact deployed values. The Qwen
-weights were NOT deleted from the host.
+Payload and runtime for [[frank-small-qwen38-gsq-rco]] (ISTA-DASLab
+Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-small-vllm-gemma4]]
+is the same-day vLLM experiment (also a rollback target).

-## References
+# Citations

- `scripts/serve-frank-small.sh` — the actual exec line
- [[frank-small-qwen38-gsq-rco]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[x4-pair-experiment]] — why the 3090 serves its own model, not the trunk
+- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
+- llama.cpp: https://github.com/ggml-org/llama.cpp
+- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S
-description: The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves.
+title: "frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
+description: "The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
 tags:
  - llm
  - frank-small
@@ -13,193 +13,3 @@ tags:
  - vision
 timestamp: 2026-09-16
 ---
-
-# frank-small — llama.cpp on the RTX 3090
-
-> **RETIRED 2026-09-17** — replaced by [[frank-small-gemma4-12b]]. Kept as
-> the rollback target and the record of the deployment it succeeded.
-> Originally ACTIVE from 2026-09-16, created when Dankclaw's inference was
-> re-split: [[halogen-flash-server]] took the gfx1151 iGPU back for
-> `frank-large`, which freed the RTX 3090 for this endpoint.
-
-## Identity
-
-| what | value |
-|---|---|
-| runtime | mainline llama.cpp master `37b53fd` (2026-09-16), CUDA build for sm_86 |
-| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
-| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
-| flags | `scripts/frank-small.env`; entrypoint `scripts/serve-frank-small.sh` |
-| endpoint | `192.168.0.143:7771`, alias `frank-small` |
-| model | `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`, the `IQ3_S-mtp` build + BF16 mmproj |
-
-It is a root unit rather than a Quadlet because the runtime is a native CUDA
-binary, not a container: there is no image to install and no generated unit to
-avoid. Boot persistence is the unit's `[Install] WantedBy=multi-user.target`;
-`Restart=on-failure` / `RestartSec=10` make it self-healing; `LimitMEMLOCK=infinity`
-covers CUDA's pinned staging buffers.
-
-Build recipe (the deployed binary):
-
-```bash
-cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-  -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
-cmake --build build-cuda --target llama-server -j
-```
-
-`GGML_CUDA_FA_ALL_QUANTS` is deliberately **off**: it is only needed for
-*quantized* KV with flash attention (see the CUDA lesson in AGENTS.md — without
-it, quantized-KV prefill collapses ~12x), and this service runs f16 KV.
-
-## The model, and why the `-mtp` build specifically
-
-GSQ-RCO is a **per-tensor mixed-precision** allocation produced by a
-gradient-based search under a total size budget, rather than one quantization
-type applied throughout. IQ3_S at 3.50 bpw is the release's task-lossless
-operating point: AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
-GPQA-Diamond is 0.51 behind — at 11.8 GB against 53.8 GB.
-
-Two files are required, and the second suffix matters:
-
-| file | size | why |
-|---|---|---|
-| `Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` | 12,120,016,960 B | the `-mtp` build **is** the MTP build: it carries the head as `blk.64.nextn.*` (`qwen35.nextn_predict_layers = 1`) |
-| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,146,528 B | the vision encoder + projector (`clip.projector_type = qwen3vl_merger`, 27 blocks) |
-
-Both live in `/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/`, fetched with
-`huggingface_hub` (which verifies the LFS sha256 and resumes).
-
-**MTP is a property of the file, not of the flags.** `--spec-type draft-mtp`
-sets `load_mtp`, and the draft context runs on the *main model's* weights
-(`common/common.cpp` passes `shares_model` when no `--model-draft` is given;
-`common/speculative.cpp` gives it `LLAMA_CONTEXT_TYPE_MTP`). So there is no
-second copy of a 12 GiB model in VRAM, and `--model-draft` would be wrong here —
-it would load the whole file again.
-
-## Architecture and the KV arithmetic
-
-GGUF arch `qwen35`: 64 layers laid out as `16 x (3 x (Gated DeltaNet -> FFN) +
-1 x (Gated Attention -> FFN))`. Only the 16 full-attention layers take KV, so a
-token costs
-
-```
-2 (K,V) x 4 heads x 256 head_dim x 2 B = 64 KiB at f16   (34 KiB q8_0, 18 KiB q4_0)
-```
-
-and the 48 DeltaNet layers add ~0.16 GiB per sequence of constant-size state in
-f32, at any depth. The MTP draft context is `n_ctx` long over one nextn layer,
-so it costs 4 KiB/token of its own.
-
-## Sizing (RTX 3090, 24576 MiB)
-
-| term | GiB |
-|---|---|
-| weights | 11.29 |
-| mmproj | 0.87 |
-| MTP draft KV (131,072 x 4 KiB) | 0.50 |
-| DeltaNet state (one sequence) | 0.16 |
-| CUDA context + prefill/decode buffers | ~1.5 |
-| **KV, one slot at 131,072 x 64 KiB** | **8.00** |
-| **modelled total** | **~22.3 of 24** |
-
-The binding number is the KV: about 9.5 GiB is left after the fixed terms, which
-is ~155k tokens at 64 KiB/token. Alternatives inside the same budget: `2 x
-65536` at f16, or `4 x 65536` / `1 x 262144` at q8_0 — the quantized variants
-need a rebuild with `-DGGML_CUDA_FA_ALL_QUANTS=ON`. **`--parallel N` divides
-`--ctx-size` between the slots**; it does not multiply.
-
-Measured at deploy: **22,362 MiB** in use of 24,576, `n_slots = 1`,
-`n_ctx_slot = 131072`.
-
-## Deployed flags
-
-| flag | value | why |
-|---|---|---|
-| `CTX_SIZE` / `PARALLEL` | `131072` / `1` | one conversation per slot, full f16 precision, no KV quantization risk |
-| `CACHE_TYPE_K/V` | `f16` | quantizing would need the FA_ALL_QUANTS build for no benefit here |
-| `NGL` | `999` | every layer on the card |
-| `FLASH_ATTN` | `on` | |
-| `SPEC_TYPE` | `draft-mtp` | the model's own head; draft depth stays llama.cpp's default of 3 |
-| `BATCH_SIZE` / `UBATCH_SIZE` | `2048` / `512` | the defaults; ubatch 512 keeps the DeltaNet chunk buffers small |
-| `NUM_THREADS` / `THREAD_BATCH` | `8` / `8` | the work is on the GPU; the host CPU belongs to halogen's engine |
-| `TEMPERATURE` | `0.0` | greedy default (fleet contract) — clients that send their own values get them |
-| `REASONING_FORMAT` / `REASONING_BUDGET` | `deepseek` / `-1` | thinking into `reasoning_content`, unlimited budget (the model decides) |
-| `--jinja` | pinned | the GGUF's own template supplies `reasoning_effort`, `preserve_thinking` and the tool-call grammar. This build defaults it on; pinning it means a future default change cannot silently take tools away |
-| `--metrics` | on | Prometheus scrapes `/metrics` here (`frank-small` job) |
-
-## Client contract
-
- **One slot.** A second concurrent request queues; it is not served in
-  parallel, and agent fan-out through this endpoint serializes. Widen with
-  `PARALLEL` (which divides the context) if that becomes the wrong trade.
- **Thinking counts against the budget.** A request whose budget runs out
-  mid-thought returns `finish_reason: "length"` with an empty `content` and the
-  partial reasoning in `reasoning_content` — observed live on a vision request
-  at `max_tokens: 64`. Give reasoning models a real budget.
- **Tools work** (`tool_calls` returned; verified with a function-call round
-  trip) because `--jinja` serves the model's own template. This is also what
-  fixed subagents fleet-wide: the ik runtime that served `frank-large` until
-  2026-09-16 rejected every request carrying `tools` with
-  `500 tools param requires --jinja flag`.
- **Images are accepted** as OpenAI `image_url` content parts with a `data:`
-  URL or bare base64. This is the fleet's only image path: halogen's
-  `frank-large` runs with no tower and refuses images with a named reason.
- **`reasoning_effort` accepts only `xhigh`, `medium`, `low`.** The template
-  raises on anything else — a 500, not a clamp — so the harness folds its own
-  levels onto that set in `compat.reasoningEffortMap` (`high: xhigh`,
-  `max: xhigh`). An unmapped level fails on the first request.
- `/v1/models` reports `frank-small`; `/health` answers 200 once the model is
-  loaded and the projector is up.
-
-## Verified at deploy (2026-09-16)
-
- `systemctl is-enabled/is-active frank-small.service` → `enabled` / `active`.
- `/v1/models` → `frank-small`; a chat completion returns `content` **and**
-  `reasoning_content`.
- A tool call returns `tool_calls` with the expected arguments.
- Slot timing lines show `draft acceptance = 0.66-0.94, mean len = 2.95-3.83`
-  — MTP is engaged, not merely configured.
- An image request returns the right colour answer from the projector.
- Harness routing: role `smol` lands here, role `slow` on `frank-large`, and a
-  real subagent (the `task` role) completes a tool-using turn.
-
-## Operate
-
-```bash
-systemctl status frank-small.service          # root unit
-journalctl -u frank-small.service -n 60       # load + per-request timing
-curl -s http://127.0.0.1:7771/health
-curl -s http://127.0.0.1:7771/v1/models
-curl -s http://127.0.0.1:7771/metrics | grep -E 'spec_decode|kv_cache_usage'
-nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
-```
-
-`/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`,
-which is the cheapest way to confirm MTP is running, and
-`llamacpp:kv_cache_usage_ratio` for how full the slot is.
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| unit restarts immediately, `frank-small: missing <path>` in the journal | the model, the projector or the binary is absent — the serve script refuses to start rather than half-load; finish the download |
-| `500 … Jinja Exception: Unexpected reasoning effort <level>` | a client sent an effort the template does not accept; map it (see the client contract) |
-| `500 tools param requires --jinja flag` | impossible here (`--jinja` is pinned) — you are looking at a *different* endpoint, most likely the retired ik runtime's logs |
-| `finish_reason: "length"` with empty content | the budget ran out inside the thinking block; raise `max_tokens` |
-| a request that waits, then answers normally | the single slot was busy (or the KV is nearly full); raise `PARALLEL` or expect serialization |
-| CUDA OOM at load | another process holds the 3090, or the KV was widened past ~9.5 GiB; check `nvidia-smi` and re-derive from the sizing table |
-
-## Rollback
-
-`frank-small` is a new endpoint, so rollback is retirement:
-`systemctl disable --now frank-small.service`, move
-`/etc/systemd/system/frank-small.service` into `~/.AGENTTRASH/`, and remove the
-harness provider + the `smol`/`tiny`/`task`/`vision` role entries. The old
-`frank-small` (FreeToken on Dankserver, retired 2026-09-15) is a different
-service with a different card — see [[freetoken-frank-small]].
-
-## Citations
-
- [ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF) — the quantized checkpoints, the `-mtp` builds and the projector
- [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — base model: 64 layers, Gated DeltaNet/Gated Attention layout, 262,144 native context
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `--spec-type draft-mtp`, the MTP context type, and the Qwen3-VL projector path
@@ -1,7 +1,7 @@
 ---
 type: reference
 title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
-description: The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant.
+description: "The RETIRED frank-small runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-small-gemma4-12b]] on the same day."
 tags:
  - llm
  - frank-small
@@ -17,11 +17,12 @@ timestamp: 2026-09-19

 # frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)

-> **ACTIVE since 2026-09-19.** Replaces [[frank-small-gemma4-12b]]
-> (llama.cpp, now the rollback target). Same port (7771), same alias
-> (`frank-small`), same client contract surface — the runtime, the model
-> payload, the drafter and the service shape (root systemd unit → rootless
-> Quadlet) all changed.
+> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
+> back to [[frank-small-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
+> the same day it landed — this page is the rollback target. Same port
+> (7771), same alias (`frank-small`), same client contract surface — the
+> runtime, the model payload, the drafter and the service shape (root
+> systemd unit → rootless Quadlet) all changed.

 ## Identity

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: halogen-flash-server on Dankclaw (frank-large)
-description: The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19.
+title: "halogen-flash-server on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
 tags:
  - llm
  - halogen
@@ -13,467 +13,3 @@ tags:
  - rocm
 timestamp: 2026-09-13
 ---
-
-# halogen-flash-server on Dankclaw (frank-large)
-
-> **RETIRED 2026-09-19 — replaced by [[frank-large-ciru-orca]]** (the CIRU v4.4.1
-> runtime serving jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca). This runbook is the
-> rollback target. It was ACTIVE again since 2026-09-16: it replaced the
-> llama.cpp / kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
-> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
-> [[frank-large-ik-uncensored]]), and came back as the shape the fleet asked for:
-> two 200k-token conversations rather than the native-context pair. It was
-> deployed as a rootless Quadlet container,
-> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770.
-
-The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
-architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
-Upstream's own comparison table measures it at **roughly 4x the prefill
-throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
-ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
-prompt with real context. That, not decode, is the reason for the swap.
-
-## Shape
-
-One container runs both halves of the stack:
-
-| half | port | reachable |
-|---|---|---|
-| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
-| OpenAI front-end | 8731 | published as host **7770** |
-
-The `all` mode is upstream's default and the shape a single-container host
-wants: the front-end talks to the engine over loopback inside the container, so
-there is one process tree, one log, and one published port.
-
-`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
-`/health`, and in every response body. Requests are **never rejected for naming
-a different model** — the server serves whatever asks. Keeping the fleet alias
-is therefore free, and it means Hermes, the harness config, the OpenWebUI
-discovery, the `services.json` probe and the docs needed no rename.
-
-## Weights
-
-`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
-
-| file | size | role |
-|---|---|---|
-| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
-| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
-| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
-| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
-
-`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
-only** — the entrypoint acts only when the checkpoint is absent, so a restart
-never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
-mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
-`/home/dank/models/halogen`.
-
-**Refreshing the weights.** Delete the mount's contents (or the whole
-directory, then recreate it **empty**) and start the unit: the entrypoint sees
-the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
-traps. First, the bind mount's source directory must *exist* — podman refuses
-the container with `Error: statfs /home/dank/models/halogen: no such file or
-directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
-directory and the fetch starts on the next start. Second, "refresh" only means
-"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
-(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
-2,383,306,048 → **2,478,095,488**) while the checkpoint returned
-byte-identical in size (124,068,083,904) — so an update can live entirely in
-the sidecars, and equal file sizes are not proof of an unchanged repo.
-
-`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
-refresh) comes along with the repo but is unused: it trades the `o_proj`
-calibration for about 2 % of decode, and the quality sidecar is the default for
-a reason.
-
-## Deployed flags
-
-Everything else is configured by environment variable — there is no config
-file, and the flags are read once at startup. These are the departures from
-upstream's defaults:
-
-| flag | value | why |
-|---|---|---|
-| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
-| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
-| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
-| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
-| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
-| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
-| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
-| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
-| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
-
-Everything else is upstream's default, deliberately: prompt cache mode `2`
-(two entries per conversation — end-of-system-prompt and end-of-history — which
-already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
-stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
-temperature 0, and it stands down whenever a second conversation is
-generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
-gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
-
-## Memory model
-
-Three numbers explain the whole box:
-
- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
-  GPU in place, never copied. The kernel counts the locked pages as reclaimable
-  file cache, so `free`, `MemAvailable` and every dashboard built on them
-  **overstate free memory by the size of the model**. The server prints the
-  corrected figure at startup; believe that line, not Grafana.
- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
-  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
-  reads the two together in the engine's own words: `400384 positions, ~27.2
-  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
-  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
-  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
-  128 GB machine *with the weights pinned* — and lower on a machine carrying
-  more besides this server, which Dankclaw is.
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
-  table is read through. Too little and long prompts crawl with the disk busy
-  and the process in uninterruptible sleep — that is a cache problem, not a
-  memory problem, and the fix is a *smaller* pool, not a bigger one.
-
-`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
-the pool itself if the configured one will not fit, printing what it chose.
-Read those lines on every restart; they are the ground truth for sizing.
-
-**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
-startup account, including three deliberate experiments:
-
-| pool | max_tok | reserve | KV | arena | need | budget | verdict |
-|---|---|---|---|---|---|---|---|
-| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
-| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
-| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
-| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
-| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
-| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
-| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
-| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
-
-### The sizing rule
-
-One budget covers the pool, the arena **and** the vision tower together:
-
-```
-device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
-              = 121.2    - 67.7             - reserve
-                                    reserve 24 -> 29.5 GiB
-                                    reserve 15 -> 38.5 GiB
-
-need = KV (28 KiB/position) + arena + vision (1.97 measured)
-arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
-```
-
-The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
-121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
-Use the unrounded value. Two of the three terms are fixed hardware and
-checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
-buys the *fit margin*, not the host's memory. It is a term in this inequality,
-not an allocation, and the section at the end of this page shows the live test
-that proves it.
-
-Trust the engine's printed requirement over this model wherever they disagree:
-a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
-of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
-enough to plan with, and the authoritative number to cut the other way.
-
-The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
-point to 0.1 GiB — including upstream's three published "on the device" figures
-(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
-**pool + arena totals at the default `max_tok`**, not pool alone. That is why
-their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
-49.4. Reading those figures as pool-only is the trap that makes a doubling look
-affordable when it is not.
-
-At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
-13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
-ceiling, not a conservative setting.** The corner check agrees: at 32768 only
-7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
-262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
-= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
-fits unmodified.
-
-### What the reserve actually costs
-
-The device budget is not the whole story. Memory position by configuration
-(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
-
-| config | held | host left | contiguous 2 MiB blocks |
-|---|---|---|---|
-| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
-| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
-| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
-| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
-| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
-
-At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
-engine volunteered the stall note. It did not print it at the 401,408 trim, and
-it prints it again after the 2026-09-16 retune. Three data points, and they do
-not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
-not, 475 did. So treat the note as "the machine is tight" and read the
-`host memory left` line rather than waiting to be told:
-
-```
-halogen: NOTE this server now holds most of the machine, and 310 contiguous
-2 MiB blocks remain. It runs best on a host of its own. If other large
-processes share this machine they will compete for what is left, and when it
-runs out BOTH they and this server can stall for minutes at a time at full
-CPU with no disk activity and no output, which is not a crash and needs no
-restart.
-```
-
-That is the failure mode to watch, and it is the least diagnosable of the
-three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
-looks like a hang, for the neighbours as much as for frank-large. `free` is
-useless for spotting it — `MemAvailable` still reads ~74 GiB because the
-page cache counts as reclaimable. Watch the `host memory left for everything
-else` line and the contiguous-block count at boot instead.
-
-28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
-so it is survivable.
-
-### The reserve cannot buy air — ceiling 16
-
-The last row above is the experiment that proves it. Raising the reserve from
-15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
-GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
-allocation; the actual footprint is `weights + KV + arena + vision`, none of
-which the reserve touches. So the knob has exactly two regimes:
-
-| reserve | budget | vs required 36.5 | effect |
-|---|---|---|---|
-| 15 | 38.5 | +2.0 | deployed |
-| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
-| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
-| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
-| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
-
-The 36.5 requirement is the engine's own, read from a clamped boot:
-`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
-thing a bigger reserve can do is take the doubling away.
-
-**To actually free host memory, change the pool or the arena**, and the reserve
-then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
-7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
-are one-line edits that the fit check will confirm on the next boot. The
-2026-09-13 trim to 401,408 was this remedy applied at half strength —
-`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
-with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
-modelled 3.4 because the engine holds the freed KV as working slack.
-
-### The `WARNING` line only appears when the fit check had to move
-
-This one cost me a wrong conclusion, so it is worth stating plainly. When the
-configured pool already fits, the engine reserves it **and prints nothing** —
-the 09-11 baseline boot goes straight from the pre-flight line to
-`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
-its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
-after a clamp**.
-
-So the absence of a warning in old logs is not evidence of headroom. Both
-`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
-baseline has lived in that state since it was deployed without incident,
-because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
-at that point). The engine's budget is a conservative floor, not a hard wall.
-
-### Two failure modes, and only one is dangerous
-
-Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
-outgrows the budget, and the server cannot allocate at all:
-
-```
-kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
-kv pool: WARNING even 262144 positions (the context, the smallest pool that can
-         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
-dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
-```
-
-This is the quiet one: the unit stays `active`, the container reports
-`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
-`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
-large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
-slower".
-
-Under-running the ceiling is far more benign and far easier to miss. At 524288
-with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
-`deploy.sh` still reported healthy in 12 seconds:
-
-```
-kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
-         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
-```
-
-A green health check does not mean the pool you asked for is the pool you got —
-read the `kv pool:` lines after every restart.
-
-The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
-the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
-`/proc/meminfo` figure at all.
-
-**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
-decode numbers were measured with, and it means this container holds most of
-the host. If a co-tenant then takes the remainder, both it and the engine can
-stall for minutes at a time at full CPU with no disk activity and no output.
-That is documented behaviour, not a crash, and it needs no restart.
-`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
-decode speed — a last resort, not a tuning option.
-
-## Client contract
-
-The differences from llama.cpp that bite are all in the request, not the
-deployment:
-
- **No server-side sampler defaults.** `temperature` absent or `0` is
-  **greedy** decode; above 0 the request samples on the same drafter. Supported
-  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
-  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
-  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
-  value outside its defined range (values are refused, never clamped).
-  The fleet runs greedy for clients that stay silent — deterministic, and
-  byte-identical with the drafter.
- **The token budget covers thinking.** The default is **8192**, and a budget
-  that runs out mid-thought returns `finish_reason: "length"` with an **empty
-  `content`** and the partial reasoning in `reasoning_content` — which most
-  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
-  request is a 400 rather than a silent truncation. `max_tokens`,
-  `max_completion_tokens` and `max_output_tokens` all work and must agree if
-  more than one is sent.
- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
-  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
-  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
-  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
-  the window while sitting below the compaction trigger. That collided live on
-  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
-  harness-side: keep the declared `maxTokens` at or under
-  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
-  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
-  breaks the two-resident-conversations property.
- **`reasoning_effort`** is accepted at the top level and inside
-  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
-  model's default is `xhigh`. `minimal` aliases the template's `low` and still
-  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
-  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
- `/v1/models`, `/health` and completions all report the served id, and
-  `reasoning_content` carries the reasoning trace.
-
-**Pool admission is the one arithmetic trap.** A request reserves its prompt
-**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
-the request waits in arrival order, and after the queue timeout (3600 s) it
-gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
-the pool can therefore never be admitted at all. The 2026-09-16 shape is a
-400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
-conversations at the ceiling fit the pool exactly, so a pair that is both
-near-full and both generating is pool-bound — the second waits for the first to
-release its region. The pool binds **how many** conversations stay resident,
-not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
-tables above.
-
-## Vision
-
-**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
-the tower is a 1.97 GiB charge against the same device budget as the pool and
-the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
-which loads the BF16 projector alongside its own weights. This server refuses
-image content parts and says why, rather than ignoring them:
-
-```json
-{"error": {"message": "this server does not accept images: the engine was
-started without a vision tower (HALOGEN_VISION_TOWER)", "type":
-"invalid_request_error"}}
-```
-
-The llama.cpp side handles images the same way this engine used to: a `data:`
-URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
-refused deliberately (fetching one would make the server issue outbound
-requests to wherever a client pointed it). Audio and video have no path
-anywhere in the fleet — that left with [[frank-npu]].
-
-## Deploy and operate
-
-```bash
-./scripts/deploy.sh dankclaw --service frank-large
-```
-
-The handler retires the ik-era **root** unit first (stop, disable, move
-`/etc/systemd/system/frank-large.service` into
-`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
-`disable` may delete the file outright, which the handler now reports), stops
-the Quadlet's own unit if it is running, then pulls
-`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
-`localhost/` alias the unit references, and **waits for port 7770 to be free**
-before installing and starting anything.
-
-That last gate is not paranoia. On the first cutover attempt the ik server
-logged "Deactivated successfully", and pasta — rootless podman's userspace
-network stack, which binds the published port at container CREATE — still
-reported `Address already in use` a second later, exiting the unit with code
-126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
-starts a doomed container has not deployed anything, so `wait_port_free` polls
-every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
-the socket table if it never clears.
-
-Boot persistence comes from `[Install] WantedBy=default.target` in the unit
-file plus linger on the deploy user — generated units **cannot** be
-`systemctl --user enable`d on this host.
-
-```bash
-systemctl --user status frank-large.service     # the generated unit
-podman logs frank-large | head -40              # version banner + memory ledger
-curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
-curl -s http://127.0.0.1:7770/v1/models
-```
-
-`/health` asks the **engine** for a PONG before answering, so a 200 means
-"ready to serve", not merely "process alive". During the first-start download
-it stays unhealthy by design.
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
-| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
-| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
-| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
-| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
-| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
-
-`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
-already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
-deliberately **not** set: it logs every allocation over 64 MB, which is
-steady-state noise on a shared host — raise it only for a live OOM report.
-The lines worth grepping first:
-
-```bash
-podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
-```
-
-Retrieval: `podman logs frank-large` covers the **current** instance only —
-the Quadlet runs `--replace --rm`, so everything before the last recreate is
-gone from the container log. `journalctl --user -u frank-large.service` is
-the durable record. Both are rootless: `sudo podman logs frank-large` looks
-in root's storage and answers "no such container".
-
-## Rollback
-
-The predecessor is intact: the ik payload stays at
-`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
-ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
-serve script and the Qwen-Sharp template are archived in the repo under
-`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
-is a `git revert` of this change set plus a deploy — **but the Quadlet must be
-removed from `~/.config/containers/systemd/` first**, or the restarted root
-unit and the still-installed container both claim port 7770. The port gate in
-the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
-
-The `amd_iommu=off` kernel argument that came with this change is separate and
-reverts on its own — see [[dankclaw-iommu-translation-off]].
-
-## Citations
-
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
@@ -1,3 +1,13 @@
+---
+type: reference
+description: "**Status (historical, 2026-09-13):** was to deploy on Dankclaw."
+tags:
+  - llm
+  - rocm
+  - frank
+timestamp: 2026-09-15
+---
+
 # frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT

 > **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the
@@ -15,12 +15,14 @@ timestamp: 2026-08-24

 # kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw

-> **Active again 2026-09-01.** The kingjones ROCmFP4 STRIX MTP payload is
-> restored as `frank-large` after the Q8_0 AGENT experiment. The Q8_0
-> model was retired because it crashed before serving. The Q6 AGENT model,
-> standalone MTP draft, Q8 projector, and PromptForge runtime are now active.
+> **RETIRED 2026-08-29.** This payload was `frank-large` from 2026-08-25
+> until the Q8_0 AGENT experiment on 2026-08-29 (it was restored again
+> 2026-09-01 and served until 2026-08-30, when [[agentionai-fast-imatrix]]
+> took over). The Q8_0 model was retired because it crashed before
+> serving. Kept as the historical record of the Q6 AGENT model, standalone
+> MTP draft, Q8 projector, and PromptForge runtime.

-> This page documents the active model and its runbook.
+> This page documents a retired model and its runbook.

 `frank-large` (port 7770) serves
 `Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "`bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Ling-3.0-flash Q6_K Trial on Dankclaw

 > **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. At this page's retirement date `frank-large` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
@@ -1,3 +1,13 @@
+---
+type: guide
+description: "This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on"
+tags:
+  - llm
+  - vllm
+  - rocm
+timestamp: 2026-09-03
+---
+
 # Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)

 This page was the plan for a vLLM-ROCm deployment of
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page records the speculative-decoding benchmark history and the"
+tags:
+  - llm
+  - speculative
+timestamp: 2026-09-03
+---
+
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
 > [deployment.md](deployment.md) for the active configuration.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "This page answers one question: **how do we make Qwen3.8-27B serve faster on"
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant

 This page answers one question: **how do we make Qwen3.8-27B serve faster on
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # Research: vLLM Parameters for Specified Configuration

 ## Summary
@@ -0,0 +1,214 @@
+---
+type: research
+description: "Self-hosted AI job-search and resume-formatting applications: the 2026 landscape, local-LLM fit, and a recommended homelab stack."
+tags:
+  - research
+  - job-search
+  - resume
+  - self-hosted
+timestamp: 2026-09-22
+---
+
+# AI Job Search and Resume Tools
+
+Research brief for self-hostable (Docker/Podman) open-source applications in two
+categories: (a) **job search** — finding jobs, matching, tracking applications —
+and (b) **resume/CV formatting** — building, tailoring, and optimizing. Target
+environment: this homelab (2 Ubuntu GPU servers + 1 macOS mini, Podman/Quadlet,
+Caddy, local LLMs via Ollama/llama.cpp).
+
+## The Two Shapes
+
+The space split into two distinct shapes in 2026:
+
+1. **Web-app trackers** — Dockerized dashboards for tracking applications,
+   resumes, and AI-assisted matching/tailoring (JobSync, offerPilot, JobNavigator
+   on the job-search side; Reactive Resume, Resume Matcher, OpenResume, ResumeLM
+   on the resume side). The natural homelab fit: one `docker compose up` (or
+   Quadlet unit), data in a local volume, AI pointed at a local Ollama/llama.cpp
+   endpoint.
+2. **CLI agent frameworks** — "bring your own AI coding CLI" tools that run the
+   whole job search as local files + prompts (career-ops, MadsLorentzen/ai-job-search).
+   Viral in 2026 (72k and 44k stars) and the strongest at *finding* jobs (portal
+   scanning) and *tailoring* CVs, but not self-hosted web services — they run on
+   your machine against a paid AI CLI (Claude Code, Codex, etc.).
+
+**Strongest self-hostable picks for this fleet:** JobSync (MIT, Ollama-as-default
+AI + MCP server), Resume Matcher (Apache-2.0, best dedicated tailoring tool),
+Reactive Resume (MIT, best pure resume builder, 43k stars). There is **no
+dominant self-hosted app for pure job *discovery*** at scale — that niche is
+JobNavigator (Docker, 126 stars) plus the CLI frameworks.
+
+## Job-Search Applications
+
+### career-ops — https://github.com/career-ops-hq/career-ops
+MIT, ~72,400 stars (verified 2026-09-22). Open-source AI job-search "command
+center" that runs inside any AI coding CLI (Claude Code, Codex, Copilot, Gemini
+CLI). Evaluates listings against your CV with a published five-dimension rubric,
+generates ATS-optimized per-role PDF resumes, drafts Greenhouse/Ashby/Lever
+application answers, scans 150+ career portals with zero API tokens, and tracks
+the pipeline in a Go terminal dashboard. Never auto-submits. Uses whichever AI
+CLI you already pay for; data is local Markdown/YAML you own. Not a server:
+`npx @santifer/career-ops init`.
+
+### ai-job-search (MadsLorentzen) — https://github.com/MadsLorentzen/ai-job-search
+MIT, ~43,600 stars. Claude Code framework: `/setup`, `/scrape`, `/rank`,
+`/apply <url>` (fit evaluation + tailored LaTeX CV + cover letter + a reviewer
+agent), `/interview`, plus an ATS parseability check on generated PDFs. Portal
+skills ship for the Danish market but are swappable. Prereqs: Claude Code,
+Python 3.10+, Bun, a LaTeX distribution. **Caveat (README):** a GitHub fork is
+always *public* and `/setup` writes personal data into tracked files — use a
+private repo with this repo as `upstream`.
+
+### JobSync — https://github.com/Gsync/jobsync
+MIT, ~1,300 stars. The most complete self-hosted tracker: application tracker +
+monitoring dashboard, resume management (PDF/DOCX import with AI extraction),
+contacts, company pages, interview question bank, tasks/time tracking, and
+**automated job discovery** (scheduled automations track companies via job-board
+APIs and AI-match each listing). In-app AI assistant (resume review, job-match
+scoring, tailored cover letters) and a built-in **MCP server** (Claude Desktop
+can add jobs from chat). **Ollama is the default provider — fully local, no API
+key**; known-good `qwen3.5:9b`. Models must support tool calling + structured
+output. `git clone` + `docker compose up` → :3737. Env: `AUTH_SECRET`,
+`ENCRYPTION_KEY` (encrypts stored API keys — set before adding keys, never
+change after), `NEXTAUTH_URL`, `MCP_ENABLED`. Treats pasted postings as untrusted
+text (prompt-injection defense).
+
+### offerPilot — https://github.com/offercontext/offerPilot
+AGPL-3.0, ~730 stars. Local-first AI job-application tracker (Chinese-first,
+English README). Pipeline by company/role/stage, resume versioning, JD-linked
+material generation, **mock interviews** (offline Whisper transcription in
+browser, audio never persisted), interview review, **offer comparison and
+salary-negotiation prep**, a "Pilot" AI assistant. Provider-agnostic (connect
+your own model service). Docker: `docker build -t offerpilot . && docker run
+--rm -p 127.0.0.1:8080:8080 -v offerpilot-data:/data offerpilot`. Will not
+auto-submit.
+
+### JobNavigator — https://github.com/vesaias/JobNavigator
+MIT, ~126 stars. The most aggressive *discovery* tool: scrapes career pages
+(Playwright + 11 ATS handlers) and job boards (JobSpy: LinkedIn, Indeed,
+ZipRecruiter, Google), dedupes, AI-scores jobs against your résumés, tailors
+résumés/cover letters (8+8 templates), auto-fills ATS forms, tracks applications
+with funnel/Sankey stats, auto-detects Gmail replies, sends Telegram alerts.
+Chrome extension for passive LinkedIn capture. AI: Claude/Codex/Antigravity
+CLIs, OpenAI, OpenRouter, **Ollama**. `git clone` + `docker compose up -d`
+(prebuilt GHCR images, pinnable via `JN_IMAGE_TAG`).
+
+## Resume / CV Applications
+
+### Reactive Resume — https://github.com/reactive-resume/reactive-resume
+MIT, ~43,300 stars. The most popular open-source resume builder: 15 templates,
+live preview, PDF/JSON/DOCX export, drag-and-drop, share links, JSON Resume
+import, passkeys/2FA, and an **ATS checker with optional AI review**.
+Provider-agnostic AI — OpenAI, Claude, Gemini, OpenRouter, **Ollama**, and any
+OpenAI-compatible endpoint (base URL + model); credentials encrypted
+server-side. `docker compose up -d` (PostgreSQL + optional SeaweedFS). Since
+v5.1.0 PDF generation is fully client-side (`@react-pdf/renderer`) — no
+Browserless/Chromium. No tracking/analytics by default; full export; one-click
+delete.
+
+### Resume Matcher — https://github.com/srbhr/Resume-Matcher
+Apache-2.0, ~28,500 stars. The dedicated *tailoring* tool: upload a master
+resume, paste a JD, get AI-tailored content with a review step, per-job
+templates, cover-letter + interview-prep generation, **resume scoring + keyword
+highlighting** against the JD, PDF export (4 templates). **Ollama (local)** plus
+OpenAI, Claude, Gemini, OpenRouter, DeepSeek (built on LiteLLM). Official
+Docker images: `docker run -p 3000:3000 -v resume-data:/app/backend/data
+ghcr.io/srbhr/resume-matcher:latest` (API at `/api`, health at `/api/v1/health`).
+**Ollama-in-Docker gotcha (README):** use `http://host.docker.internal:11434`,
+not `localhost`.
+
+### OpenResume — https://github.com/xitanggg/open-resume
+AGPL-3.0, ~8,900 stars. Resume builder **and** parser. The builder produces an
+ATS-friendly design (Greenhouse/Lever compatible) with real-time PDF preview;
+the parser deterministically tests an existing resume's ATS readability.
+**No AI** — deliberately algorithmic (PDF.js + layout heuristics). Purely
+client-side (works offline, data never leaves the browser).
+
+### ResumeLM — https://github.com/olyaiy/resume-lm
+AGPL-3.0, ~326 stars. AI resume builder for tech jobs: base + tailored resumes,
+ATS scoring, AI bullet editing, cover-letter generator, job/application
+tracking. Cloud API providers only in practice (OpenAI, Claude, Gemini,
+DeepSeek, Groq) — no Ollama/local documented. Next.js 15 + Supabase + Redis.
+
+### rendercv (non-AI) — https://github.com/rendercv/rendercv
+MIT, ~17,600 stars. LaTeX-based YAML→PDF resume builder. No AI, but the
+strongest "ATS-safe by construction" formatter (single column, standard fonts);
+pairs well with any AI tailoring step.
+
+## Best Practices for a Homelab AI Resume/Job Pipeline
+
+**Local LLM vs API.**
+1. **Local-first for PII.** CVs carry name, contact, employment history, often
+   salary. The apps that matter all support fully-local inference: JobSync
+   (Ollama default), Resume Matcher (Ollama via LiteLLM), Reactive Resume
+   (Ollama + any OpenAI-compatible endpoint), JobNavigator (Ollama). Point them
+   at Ollama on the GPU server, or a llama.cpp `llama-server` OpenAI-compatible
+   endpoint (Reactive Resume and Resume Matcher consume it via base-URL
+   override).
+2. **Pick a model with tool calling + structured output** for JobSync — its
+   assistant works by calling tools; a model without tool support "will reply
+   with text instead of doing anything." JobSync's tested local model is
+   `qwen3.5:9b`; anything 8B–32B with tool support is a reasonable bet.
+3. **Raise Ollama's context to ~16k** for resume/JD matching (the 4k default
+   truncates long resume+JD prompts; don't go much higher).
+4. **Networking:** expose Ollama on the LAN (`OLLAMA_HOST`); inside Docker on
+   the same host use `http://host.docker.internal:11434`.
+5. **Budget reality:** the CLI frameworks are the strongest at discovery +
+   tailoring quality but require a paid AI CLI. For zero per-token cost and full
+   privacy, the Docker apps + local Ollama is the way; expect somewhat weaker
+   tailoring than frontier-API models.
+
+**Privacy & data handling.**
+1. **Treat job postings as untrusted input.** The best projects defend against
+   prompt injection in JDs (JobSync scopes the assistant to the visible job;
+   ai-job-search documents postings as untrusted). Keep a human review step
+   before anything is sent.
+2. **Never auto-submit.** All the serious tools keep submission human by
+   design. Mass auto-application burns standing with recruiters and ATS systems.
+3. **Don't fork the CLI frameworks publicly** for personal use (ai-job-search:
+   forks are public, `/setup` writes personal data into tracked files).
+4. **Protect stored secrets.** JobSync: set `ENCRYPTION_KEY` (`openssl rand
+   -base64 32`) *before* adding API keys. Reactive Resume: AI credentials are
+   encrypted server-side.
+5. **Keep the service off the public internet.** These apps store sensitive PII.
+   Bind to LAN / put behind the existing Caddy reverse proxy with auth (the
+   fleet already runs Anubis+Authentik+Caddy — route through the same guard
+   chain). offerPilot's own default is `127.0.0.1:8080`.
+6. **Back up the data volume.** JobSync has built-in zip backup/restore (excludes
+   API keys by design). offerPilot and Resume Matcher keep everything in one
+   volume — snapshot those.
+7. **Podman/Quadlet specifics:** all Docker-compose stacks map directly to
+   Quadlet `.container`/`.network` units for boot persistence (fleet convention).
+   Remember the published-port cutover gate (3737 JobSync, 3000 Resume Matcher,
+   8080 offerPilot) and that Ollama must be reachable from the container network
+   (published port or `host.containers.internal`).
+
+## Suggested Stack for This Fleet
+
+| Role | App | Why |
+|---|---|---|
+| Tracking + matching + discovery | **JobSync** (MIT) | Only self-hosted tracker with Ollama-as-default AI, job-board API discovery, MCP server, backup/restore |
+| Per-JD tailoring | **Resume Matcher** (Apache-2.0) | Master-resume workflow, scoring + keyword highlighting, single container, Ollama |
+| Resume building / ATS-safe PDFs | **Reactive Resume** (MIT) | 15 templates, ATS checker + AI review, JSON Resume import, client-side PDF |
+| (optional) deep discovery | **JobNavigator** (MIT) | Playwright scraping of career pages + boards, Telegram alerts |
+| (optional) CLI power mode | **career-ops** (MIT) | Best-in-class scoring rubric + portal scanning, if a paid AI CLI is acceptable |
+
+All five run comfortably on the existing GPU servers alongside Ollama; none
+requires more than a few hundred MB RAM.
+
+# Citations
+
+- career-ops: https://github.com/career-ops-hq/career-ops (MIT, ~72,386 stars) — site: https://career-ops.org
+- ai-job-search: https://github.com/MadsLorentzen/ai-job-search (MIT, ~43,609 stars)
+- JobSync: https://github.com/Gsync/jobsync (MIT, ~1,299 stars)
+- offerPilot: https://github.com/offercontext/offerPilot (AGPL-3.0, ~730 stars)
+- JobNavigator: https://github.com/vesaias/JobNavigator (MIT, ~126 stars)
+- Reactive Resume: https://github.com/reactive-resume/reactive-resume (MIT, ~43,301 stars) — AI guide: https://docs.rxresu.me/guides/using-ai ; self-hosting: https://docs.rxresu.me/self-hosting/docker
+- Resume Matcher: https://github.com/srbhr/Resume-Matcher (Apache-2.0, ~28,499 stars)
+- OpenResume: https://github.com/xitanggg/open-resume (AGPL-3.0, ~8,908 stars)
+- ResumeLM: https://github.com/olyaiy/resume-lm (AGPL-3.0, ~326 stars)
+- rendercv (non-AI): https://github.com/rendercv/rendercv (MIT, ~17,620 stars)
+- JobSync Ollama default + 16k context + ENCRYPTION_KEY: https://github.com/Gsync/jobsync/blob/main/README.md
+- Resume Matcher Ollama-in-Docker URL: https://github.com/srbhr/Resume-Matcher/blob/main/README.md
+- Reactive Resume client-side PDF since v5.1.0: https://github.com/reactive-resume/reactive-resume/blob/main/README.md
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Off-site backup plan for the bind-mounted data on both hosts."
+tags:
+  - research
+timestamp: 2026-08-28
+---
+
 # Backblaze Integration

 Off-site backup plan for the bind-mounted data on both hosts. **Deferred**: research only, not implemented yet.
@@ -1,3 +1,13 @@
+---
+type: reference
+description: "Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in Dankclaw."
+tags:
+  - research
+  - rocm
+  - comfyui
+timestamp: 2026-08-28
+---
+
 # ComfyUI on ROCm and Vulkan

 Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in Dankclaw. Verified on live hardware 2026-08-14. This page corrects the persistent "ROCm unavailable on Strix Halo" claim.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot."
+tags:
+  - research
+timestamp: 2026-08-28
+---
+
 # Lemonade and sd.cpp

 Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. Research from the Lemonade source repo (`lemonade-sdk/lemonade`, cloned 2026-08-14) and live verification.
@@ -0,0 +1,240 @@
+---
+type: research
+title: Local TTS and Voice Cloning
+description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
+tags:
+  - research
+  - tts
+  - voice-cloning
+  - audio
+  - stt
+  - vllm-omni
+  - kokoro
+timestamp: 2026-09-22T00:00:00Z
+---
+
+# Local TTS and Voice Cloning
+
+Research brief (2026-09-22) on creating a new local TTS voice from sample
+files, the open-weight voice-cloning landscape, the full local AI audio
+stack, and a recommended deployment for the fleet. Every claim is cited in
+[Citations](#citations). The fleet has no audio input today (that capability
+left with the retired NPU endpoint), so this is greenfield.
+
+## Summary
+
+- **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.**
+  Voices are a fixed pack. A community fine-tune recipe exists
+  (`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred
+  single-speaker samples, then voice-pack extraction. Feasible on one GPU, but
+  the output is Kokoro-styled, and OMP's local backend only accepts its
+  built-in voice list (a custom pack needs a patch).
+- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
+  leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
+  (Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT)**, **VoxCPM2
+  (Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
+- **OMP integration — definitive:** OMP's `tts` tool supports exactly three
+  backends (`providers.tts` = `auto|local|xai|deepinfra`, verified in the
+  installed binary): local Kokoro ONNX, xAI Grok Voice cloud, DeepInfra cloud.
+  **There is no generic OpenAI-compatible TTS endpoint in OMP today.** Two real
+  paths for a cloned voice: (a) xAI console custom voice (free, cloud, US-only)
+  — works today; (b) patch OMP (open source) with a ~50-line OpenAI-TTS backend
+  pointing at a local `vLLM-Omni` server.
+- **Recommended stack:** `vllm/vllm-omni` serving **Qwen3-TTS-1.7B-Base** on the
+  3090 (OpenAI-compatible `POST /v1/audio/speech` + `POST /v1/audio/voices`
+  voice upload), **Parakeet TDT 1.1B** for English STT, **Silero VAD** embedded,
+  **Chatterbox** as the ultra-light English-only alternative. All fit in <8 GB
+  VRAM and co-locate with the existing LLM services.
+
+## Can Kokoro learn a new voice?
+
+Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M
+params, non-autoregressive) ships **fixed voice packs** — there is no
+reference-audio path, so zero-shot cloning is not an option. OMP's local
+backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice
+from a built-in list (`tts.localVoice`).
+
+The best-documented fine-tune path is
+[`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0):
+
+| Stage | Data | Purpose |
+|---|---|---|
+| Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune |
+| Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization |
+| Extraction | — | voice-pack (.pt) extraction for inference |
+
+**Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but
+you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe,
+and patch OMP to load the custom pack. The effort/quality trade is worse than
+running a modern zero-shot cloner on the same GPU.
+
+## 2026 voice-cloning landscape (permissive-licensed, single-GPU)
+
+Ranked for a 3090 homelab, English-primary, personal voice:
+
+1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference;
+   110–500 M; ~1–2 GB VRAM; CPU-capable (Nano). *Top pick for a personal
+   English voice.* No official server (library + Gradio; wrap in FastAPI).
+2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
+   vLLM-Omni serving with `POST /v1/audio/voices` upload (closest to "named
+   voice" semantics), official fine-tuning code. *Top pick for a server-grade,
+   OpenAI-compatible deployment.*
+3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
+   bi-streaming, FastAPI/WebSocket server.
+4. **GLM-TTS (MIT)** — best published CER (0.89), RL emotion control, zh/en.
+5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
+   of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
+6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
+7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI,
+   Docker images.
+
+Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS
+(non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research
+license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE
+encoder not released); Voxtral TTS (no official cloning — codec encoder not
+released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
+(code withdrawn 2025-09).
+
+## OMP integration (definitive)
+
+OMP's `tts` tool (hidden by default; enable with `/tts`) supports exactly three
+backends, selected by `providers.tts`:
+
+| Backend | Endpoint (hardcoded in binary) | Voices |
+|---|---|---|
+| `local` | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` | built-in list (`tts.localVoice`) |
+| `xai` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + any custom voice ID |
+| `deepinfra` | `POST https://api.deepinfra.com/v1/openai/audio/speech` | `voice_id` when set |
+
+There is **no generic OpenAI-compatible TTS endpoint** — `deepinfra` is a
+hardcoded URL, not a configurable base URL, and OMP's `models.yml` custom
+providers (LLM chat only) do not wire into the `tts` tool.
+
+Paths for a Frank-cloned voice:
+
+1. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
+   speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → `tts` tool.
+   Free (≤30 voices), cloud, US-only.
+2. **Local, full stack — small OMP patch.** OMP is open source; the DeepInfra
+   backend is a plain HTTP POST to an OpenAI-compatible speech endpoint. A
+   ~50-line `openai` backend (configurable `base_url` + `voice`) lets OMP call
+   any local server (vLLM-Omni, Kokoro-FastAPI, Chatterbox wrapper).
+3. **No-patch workaround.** OMP's `bash` tool can `curl` the local TTS server
+   ad hoc.
+4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.
+
+## Full local AI audio stack
+
+**STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in
+VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word
+timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0,
+LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 /
+Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
+60-min, 50+ langs).
+
+**VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in
+faster-whisper/whisperX).
+
+**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
+official Docker `vllm/vllm-omni` CUDA and `vllm/vllm-omni-rocm` ROCm, e.g.
+`:v0.28.0`) is the one server exposing the standard OpenAI speech API
+(`POST /v1/audio/speech` + `POST /v1/audio/voices` upload for Qwen3-TTS) across
+the 2026 lineup. CUDA needs compute ≥7.0 (3090 = 8.6 ✓); ROCm validated on
+gfx942 (the fleet already runs vLLM on the R9700 gfx1201 ✓). Others:
+Kokoro-FastAPI (Docker, :8880, OpenAI-shaped), CosyVoice FastAPI/WS, F5-TTS
+Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 `tools/api_server.py` (own
+format, not OpenAI-shaped).
+
+**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
+reference-clip standard); Demucs (`htdemucs`) for vocal separation before
+cloning; the Perth watermark is built into Chatterbox.
+
+## Recommended deployment (Dankclaw)
+
+TTS models are 0.1–2 B (<8 GB VRAM); STT (Parakeet 1.1 B) is CPU-fast. Keep it
+all on Dankclaw (3090 = primary GPU, CUDA = best vLLM-Omni support); the
+Dankserver R9700s stay dedicated to frank-medium.
+
+| Service | Image / install | Port | GPU | Notes |
+|---|---|---|---|---|
+| TTS (primary, cloning) | `vllm/vllm-omni:v0.28.0` · `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8881 --gpu-memory-utilization 0.35` | 8881 | 3090 (~6 GB) | OpenAI `/v1/audio/speech`; upload Frank's voice once via `/v1/audio/voices`; 3 s cloning; 10 langs; Apache-2.0 |
+| TTS (light, English) | Chatterbox (MIT) in a small FastAPI container | 8880 | 3090 (~1 GB) / CPU | 10 s ref clip; wrap `chatterbox.generate` in ~30 lines |
+| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | CPU | CC-BY-4.0; WER 1.39% |
+| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | CPU / iGPU | MIT |
+| VAD | Silero (embedded) | — | CPU | MIT |
+
+Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
+
+## Voice-creation workflow (the "from sample files" answer)
+
+1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
+2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/
+   GLM-TTS clone from it per request (no training).
+3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices`
+   (vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio);
+   or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
+4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local
+   backend — highest effort, Kokoro-styled output.
+
+## Best practices
+
+- **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern
+  cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro
+  fine-tune on quality and effort.
+- **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract
+  clean voice with Demucs if the source is music/mixed.
+- **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint
+  for many TTS models — it is the only server with the standard
+  `/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup.
+- **Check the license before serving publicly:** several strong cloners are
+  non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
+  licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
+  Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
+- **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the
+  TTS engine shares the card with frank-small; move to the R9700s only if TTS
+  throughput demand grows.
+- **Guard the endpoint like any other service:** if vhosted, it goes behind the
+  Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
+  internal address, never the public hostname.
+
+## Citations
+
+- OMP TTS tool: https://omp.sh/docs/tools/tts · installed binary
+  `@oh-my-pi/pi-coding-agent/dist/cli.js` (endpoints, voices, model IDs)
+- Kokoro: https://github.com/hexgrad/Kokoro ·
+  https://huggingface.co/spaces/hexgrad/Kokoro-TTS
+- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
+- Chatterbox: https://github.com/resemble-ai/chatterbox
+- Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS ·
+  https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
+- CosyVoice: https://github.com/FunAudioLLM/CosyVoice
+- GLM-TTS: https://github.com/zai-org/GLM-TTS ·
+  https://huggingface.co/zai-org/GLM-TTS
+- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
+  https://huggingface.co/openbmb/VoxCPM2
+- F5-TTS: https://github.com/SWivid/F5-TTS
+- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
+- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base
+- IndexTTS: https://github.com/index-tts/index-tts
+- Fish Speech: https://github.com/fishaudio/fish-speech
+- MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano
+- XTTS: https://github.com/coqui-ai/TTS
+- MegaTTS3: https://github.com/bytedance/MegaTTS3
+- Higgs Audio: https://github.com/boson-ai/higgs-audio
+- Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2
+- Spark-TTS: https://github.com/SparkAudio/Spark-TTS
+- Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603
+- Parler-TTS: https://github.com/huggingface/parler-tts
+- VibeVoice: https://github.com/microsoft/VibeVoice
+- vLLM-Omni: https://github.com/vllm-project/vllm-omni ·
+  https://docs.vllm.ai/projects/vllm-omni/en/latest/
+- Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI
+- whisper.cpp: https://github.com/ggml-org/whisper.cpp ·
+  faster-whisper: https://github.com/SYSTRAN/faster-whisper ·
+  whisperX: https://github.com/m-bain/whisperX
+- Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b ·
+  Canary: https://huggingface.co/nvidia/canary-1b-v2
+- Moonshine: https://github.com/moonshine-ai/moonshine
+- Silero VAD: https://github.com/snakers4/silero-vad
+- xAI custom voices:
+  https://docs.x.ai/developers/model-capabilities/audio/custom-voices
@@ -0,0 +1,218 @@
+---
+type: research
+description: "The 15 most popular self-hosted AI applications of June–September 2026, ranked by GitHub stars, with homelab fit and adoption best practices."
+tags:
+  - research
+  - self-hosted
+  - ai-apps
+  - landscape
+timestamp: 2026-09-22
+---
+
+# Most Popular Self-Hosted AI Apps (June–September 2026)
+
+Research brief on the most popular and trending self-hostable (Docker/Podman/
+Quadlet) open-source AI applications over the last 3 months. Star counts and
+licenses pulled directly from the GitHub API and each repo's LICENSE file on
+**2026-09-22** (exact values). "Trending" is evidenced by **release activity
+dated June–September 2026** (GitHub releases API) plus press/community
+coverage — GitHub does not expose star-growth rate, so cumulative stars are the
+popularity proxy and release cadence is the trend signal. The full rendered
+report lives at `popular-self-hosted-ai-apps.html` beside this page.
+
+## Summary
+
+The self-hosted AI stack in mid-2026 consolidated into a few very large, very
+active projects, and the center of gravity shifted hard toward **agents** over
+the last quarter:
+
+1. **Agents are the category of 2026.** OpenClaw (390,235 stars, MIT) is the
+   single most-starred self-hosted AI project and the fastest-growing repo on
+   GitHub; it shipped a major **2.0 rewrite on 1 September 2026** (933
+   contributors, 16,000+ PRs). Nous Research's **Hermes Agent** (247,907 stars,
+   MIT) is a close second and releases weekly. Both are model-agnostic
+   orchestration layers that run on your own hardware and can point at local
+   llama.cpp/vLLM/Ollama endpoints.
+2. **The inference + UI layer is mature and huge.** Ollama (181,444), llama.cpp
+   (129,149), and vLLM (92,405) are the runtimes; **Open WebUI** (152,784) is
+   the dominant chat UI; **ComfyUI** (134,407, GPL-3.0) is the dominant
+   image/video-gen GUI. All release frequently in the window.
+3. **RAG/agent-platforms and the audio stack round out the field.** Dify
+   (156,835), RAGFlow (91,138), AnythingLLM (66,318) for RAG/workflows;
+   whisper.cpp (53,852) + faster-whisper (25,512) for STT; Chatterbox (26,515),
+   Piper (11,287), Kokoro-82M (8,933) for TTS; Unsloth (76,560) for
+   fine-tuning.
+
+**Strongest fits for this fleet:** Open WebUI (chat UI over the existing
+llama.cpp/vLLM endpoints), ComfyUI (image gen, already integrated with Open
+WebUI), Ollama (OpenAI-compatible runtime), Dify and RAGFlow (RAG/agent
+platforms), and whisper.cpp / Piper / Kokoro for a local voice stack. OpenClaw
+and Hermes are the highest-buzz new arrivals, well-suited to a homelab server,
+with the caveat that they are tool/shell-access agents and need the strongest
+isolation.
+
+## Ranked List (Top 15, all with June–Sept 2026 release activity)
+
+| # | App | Category | Stars (2026-09-22) | License | Latest release in window |
+|---|-----|----------|--------------------|---------|--------------------------|
+| 1 | [OpenClaw](https://github.com/openclaw/openclaw) | Agents | 390,235 | MIT | v2026.9.5 (09-19); 2.0 (09-01) |
+| 2 | [Hermes Agent](https://github.com/NousResearch/hermes-agent) | Agents | 247,907 | MIT | v2026.9.21 (09-21) |
+| 3 | [Ollama](https://github.com/ollama/ollama) | LLM runtime | 181,444 | MIT | active (pushed 09-21) |
+| 4 | [Dify](https://github.com/langgenius/dify) | Agent platform / RAG | 156,835 | Modified Apache-2.0 | 1.17.1 (09-10) |
+| 5 | [Open WebUI](https://github.com/open-webui/open-webui) | Chat UI | 152,784 | Open WebUI License (BSD-3-based) | v0.11.4 (09-21) |
+| 6 | [ComfyUI](https://github.com/Comfy-Org/ComfyUI) | Image / video gen | 134,407 | GPL-3.0 | v0.37.0 (09-21) |
+| 7 | [llama.cpp](https://github.com/ggml-org/llama.cpp) | LLM runtime (inference) | 129,149 | MIT | active (pushed 09-22) |
+| 8 | [vLLM](https://github.com/vllm-project/vllm) | LLM inference engine | 92,405 | Apache-2.0 | active (pushed 09-22) |
+| 9 | [RAGFlow](https://github.com/infiniflow/ragflow) | RAG engine | 91,138 | Apache-2.0 | v0.27.2 (09-10) |
+| 10 | [LobeChat](https://github.com/lobehub/lobehub) | Chat UI / agent ops | 82,748 | LobeHub Community License | active (pushed 09-22) |
+| 11 | [Unsloth](https://github.com/unslothai/unsloth) | Fine-tuning | 76,560 | Apache-2.0 | active (pushed 09-22) |
+| 12 | [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) | RAG / local agent | 66,318 | MIT | active (pushed 09-21) |
+| 13 | [whisper.cpp](https://github.com/ggml-org/whisper.cpp) | Speech-to-text | 53,852 | MIT | active (pushed 09-22) |
+| 14 | [LibreChat](https://github.com/danny-avila/LibreChat) | Chat UI | 44,608 | MIT | active (pushed 09-22) |
+| 15 | [Tabby](https://github.com/TabbyML/tabby) | AI coding assistant | 33,889 | Apache-2.0 (ee/ separate) | last push 2026-06-30 |
+
+**Audio honorable mentions:** [Chatterbox](https://github.com/resemble-ai/chatterbox)
+(26,515, MIT, TTS), [faster-whisper](https://github.com/SYSTRAN/faster-whisper)
+(25,512, MIT, STT), [Piper TTS](https://github.com/rhasspy/piper) (11,287, MIT,
+TTS), [Kokoro-82M](https://github.com/hexgrad/kokoro) (8,933, Apache-2.0, TTS),
+[MisoTTS](https://github.com/MisoLabsAI/MisoTTS) (3,239, Modified MIT, TTS).
+
+## Per-App Notes
+
+- **OpenClaw** — self-hosted personal AI agent / orchestration layer (not a
+  model). Connects to cloud APIs, subscriptions, API providers, and **local
+  models**; drives files, browser, terminal/shell, and messaging channels. 2.0
+  rebuilt the browser app as the primary interface, added shared cloud sessions,
+  simplified onboarding. Node 24 (22.14+) + Docker; minimal footprint for
+  API-based LLMs (2 vCPU / 2–4 GB RAM). **Caveat:** community reaction to 2.0
+  was mixed — migration problems, broken gateways, lost automations, model-auth
+  issues (InfoQ). Tool/shell-access agent → strongest isolation.
+- **Hermes Agent** — Nous Research's personal agent, centered on persistent
+  memory and cross-platform messaging. Model-agnostic; Docker/Podman; config in
+  `~/.hermes/config.yaml`. Releases weekly. Tool-access agent → isolate.
+- **Ollama** — the de-facto standard local LLM runtime; OpenAI-compatible API
+  (default `:11434`) that nearly every app in this list consumes. The default
+  provider for JobSync, Resume Matcher, AnythingLLM, and more.
+- **Dify** — full LLMOps/agent platform: prompt IDE, RAG pipeline,
+  agent/workflow builder, model management. `docker compose` full stack.
+  Modified Apache-2.0: commercial use allowed, but **multi-tenant operation
+  requires a commercial license** (one tenant = one workspace) — fine for a
+  single-workspace homelab.
+- **Open WebUI** — the dominant ChatGPT-style chat UI: multi-user auth, model
+  switching, tools, pipelines, RAG, image gen (ComfyUI as engine), agents.
+  Single container; connect to Ollama / llama.cpp / vLLM via base-URL. Already
+  this fleet's chat frontend.
+- **ComfyUI** — node-based image *and* video generation (SD/SDXL/Flux/LLaDA/
+  Qwen-Image). The most popular local image-gen frontend in 2026. Repo moved to
+  the `Comfy-Org` org. Already running on this fleet (Qwen-Image 2.1 GGUF) and
+  wired as the Open WebUI image engine.
+- **llama.cpp** — the reference GGUF inference engine; `llama-server` exposes an
+  OpenAI-compatible endpoint. Backbone of this fleet's frank-small/frank-large.
+- **vLLM** — high-throughput PagedAttention serving with an OpenAI-compatible
+  API; the multi-GPU option (frank-medium runs vLLM tensor-parallel across two
+  R9700s).
+- **RAGFlow** — RAG engine with deep document understanding (PDFs, spreadsheets,
+  scans) and source citations. `docker compose` full stack; runs fully against
+  local models (Ollama).
+- **LobeChat** — polished chat UI grown into an agent-operations layer (plugins,
+  agents, scheduled tasks). LobeHub Community License: commercial use allowed,
+  but **distributing a derivative requires a commercial license**.
+- **Unsloth** — the leading open-source fine-tuning library (LoRA/QLoRA/SFT) +
+  GGUF quantization + a local run/train UI. GPU required; a batch job, not a
+  long-running service.
+- **AnythingLLM** — local-first RAG + agent workspace: document ingestion, chat
+  over your data, agent mode, multi-user. Ollama first-class.
+- **whisper.cpp** — OpenAI Whisper in C/C++; lightweight fully-local STT on CPU /
+  Metal / CUDA; `whisper-server` for an HTTP API.
+- **LibreChat** — multi-provider, multi-user chat UI with agents, MCP, actions.
+  `docker compose` (app + MongoDB + Redis).
+- **Tabby** — self-hosted AI coding assistant (Copilot-style) against your own
+  models. Apache-2.0 core; `ee/` directory under a separate enterprise license.
+  Last push 2026-06-30 (quieter since).
+
+## Homelab Fit (Podman/Quadlet + Caddy + local LLM endpoints)
+
+| App | Fit | Notes for this fleet |
+|-----|-----|----------------------|
+| **Open WebUI** | **Excellent** | Single container; point at existing llama.cpp/vLLM/Ollama via base-URL. Already the fleet's chat frontend. |
+| **ComfyUI** | **Excellent** | GPU container (already running for Qwen-Image 2.1 GGUF). Serves the Open WebUI image-gen path. |
+| **Ollama** | **Excellent** | Easiest OpenAI-compatible runtime for the other apps; expose on LAN / `host.containers.internal`. |
+| **Dify** | **Good** | compose → Quadlet. Multi-tenant restriction moot for single-workspace. Heavier stack (DB + Redis + vector store). |
+| **RAGFlow** | **Good** | compose → Quadlet. Runs fully against local models. Budget disk for the document index. |
+| **AnythingLLM** | **Good** | Single container, Ollama-first, low footprint. Good local RAG entry point. |
+| **LibreChat** | **Good** | compose (app + Mongo + Redis) → Quadlet. Multi-user, MCP/agents. |
+| **LobeChat** | **Good** | Single Postgres-backed container. Watch the Community License if redistributing a derivative. |
+| **whisper.cpp / faster-whisper** | **Good** | Lightweight STT; `whisper-server` behind Caddy for a LAN transcription API. CPU-fine. |
+| **Piper / Kokoro / Chatterbox** | **Good** | Local TTS; pair with whisper.cpp for a fully-local voice loop. Chatterbox wants a GPU; Piper/Kokoro run on CPU. |
+| **Tabby** | **Good** | compose → Quadlet; self-hosted coding assistant against local models. |
+| **Unsloth** | **OK (one-off)** | Fine-tuning is a batch job, not a service. Run transiently on a GPU server, then serve the GGUF via llama.cpp. |
+| **OpenClaw / Hermes** | **Good, strong isolation** | Node.js / container agent pointing at local LLMs. Highest buzz, but tool/shell access — non-root, restricted network, vetted skills, behind the guard chain. |
+
+**GPU co-tenancy note:** ComfyUI, vLLM, Unsloth, and Chatterbox all want VRAM.
+Size new containers against the existing LLM endpoints' held VRAM to avoid
+OOMing a production endpoint. The macOS mini is a natural home for the
+low-footprint, CPU-friendly apps (Open WebUI, AnythingLLM, whisper.cpp,
+Piper/Kokoro, LibreChat, LobeChat) via Metal-backed llama.cpp/Ollama.
+
+## Best Practices: Adopting a New App Into a Guard-Chained Homelab
+
+1. **Route everything through the existing guard chain.** Put the app behind
+   Caddy under the same Anubis+Authentik chain the fleet runs. Do not expose a
+   new port directly; use the published-port cutover gate (confirm the port is
+   free before starting, or pasta's bind fails and the failure hides behind
+   `Restart=on-failure`).
+2. **Enable app-native auth and add SSO on top.** Turn on the app's built-in
+   multi-user auth *and* gate it with Authentik SSO at the proxy.
+3. **Prefer local endpoints for the AI; keep PII on-prem.** Point RAG/chat/agent
+   apps at local llama.cpp/vLLM/Ollama (base-URL override) so prompts and
+   documents never leave the LAN. Reach the host runtime via
+   `host.containers.internal`, not `localhost`.
+4. **Use Quadlet for boot persistence.** Convert every compose/`docker run` to
+   Quadlet units. Remember the anonymous-volume trap — use named volumes for
+   all app data.
+5. **Pin image tags and track release notes.** The window's clearest lesson:
+   **OpenClaw 2.0 (2026-09-01) caused migration problems, broken gateways, and
+   lost automations for some users.** Pin tags, read release notes, stage
+   upgrades (test on the mini / a spare server first).
+6. **Isolate agents aggressively.** OpenClaw and Hermes have tool/shell/browser
+   access. Non-root, restricted network, vetted skills/plugins only, full
+   logging of tool execution.
+7. **Treat ingested content as untrusted (prompt injection).** Keep a human
+   review step before any action; scope agent tool access to the minimum.
+8. **Protect the secrets the app stores.** API keys in env vars (or a vault),
+   never committed. Set the encryption key *before* adding keys, never change
+   it after.
+9. **Budget VRAM and co-tenancy.** Measure a new GPU container's VRAM against
+   the endpoints already resident on the card.
+10. **Back up the data volumes.** Snapshot the app's data volume; agents store
+    memory/state that is expensive to rebuild.
+11. **Mind the license.** ComfyUI is GPL-3.0; Dify and LobeChat are
+    modified-Apache with multi-tenant / derivative restrictions; Tabby has an
+    `ee/` directory. Fine for a single-workspace, non-redistributed homelab, but
+    note them before derivative work.
+
+# Citations
+
+- OpenClaw: https://github.com/openclaw/openclaw (MIT, 390,235 stars)
+- Hermes Agent: https://github.com/NousResearch/hermes-agent (MIT, 247,907 stars)
+- Ollama: https://github.com/ollama/ollama (MIT, 181,444 stars)
+- Dify: https://github.com/langgenius/dify (Modified Apache-2.0, 156,835 stars)
+- Open WebUI: https://github.com/open-webui/open-webui (Open WebUI License, 152,784 stars)
+- ComfyUI: https://github.com/Comfy-Org/ComfyUI (GPL-3.0, 134,407 stars)
+- llama.cpp: https://github.com/ggml-org/llama.cpp (MIT, 129,149 stars)
+- vLLM: https://github.com/vllm-project/vllm (Apache-2.0, 92,405 stars)
+- RAGFlow: https://github.com/infiniflow/ragflow (Apache-2.0, 91,138 stars)
+- LobeChat: https://github.com/lobehub/lobehub (LobeHub Community License, 82,748 stars)
+- Unsloth: https://github.com/unslothai/unsloth (Apache-2.0, 76,560 stars)
+- AnythingLLM: https://github.com/Mintplex-Labs/anything-llm (MIT, 66,318 stars)
+- whisper.cpp: https://github.com/ggml-org/whisper.cpp (MIT, 53,852 stars)
+- LibreChat: https://github.com/danny-avila/LibreChat (MIT, 44,608 stars)
+- Tabby: https://github.com/TabbyML/tabby (Apache-2.0 + ee/, 33,889 stars)
+- Chatterbox: https://github.com/resemble-ai/chatterbox (MIT, 26,515 stars)
+- faster-whisper: https://github.com/SYSTRAN/faster-whisper (MIT, 25,512 stars)
+- Piper TTS: https://github.com/rhasspy/piper (MIT, 11,287 stars)
+- Kokoro-82M: https://github.com/hexgrad/kokoro (Apache-2.0, 8,933 stars)
+- MisoTTS: https://github.com/MisoLabsAI/MisoTTS (Modified MIT, 3,239 stars)
+- OpenClaw 2.0 release coverage: https://www.infoq.com/news/2026/09/openclaw-2-release/
+- OpenClaw self-hosted architecture + security: https://hostkey.com/blog/208-what-is-openclaw-complete-self-hosted-ai-agent-guide/
+- Release cadence: GitHub releases API, 2026-09-22
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Consolidated research from the `frank-small` research subagents (2026-08-14) plus the pi agent knowledge base."
+tags:
+  - research
+timestamp: 2026-09-03
+---
+
 # Service Research Brief

 Consolidated research from the `frank-small` research subagents (2026-08-14) plus the pi agent knowledge base. This is the source material behind the per-service docs pages.
@@ -15,6 +15,18 @@ findings doc; the project root index ([serverAppConfigs-docs/index.md](../index.
 - [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
 - [[Jellyfin]] — media server research brief

+## Audio / TTS
+
+- [[Local TTS and Voice Cloning]] — creating a local voice from samples, the 2026 cloning landscape, the full audio stack, and a recommended homelab deployment
+
+## Career / Job Search
+
+- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
+
+## Self-Hosted Landscape
+
+- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices (HTML report: `popular-self-hosted-ai-apps.html`)
+
 ## Backups

 - [[Backblaze Integration]] — off-site backup plan (future)
--- a/Show More
+++ b/Show More