dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): fleet-wide fact re-verification pass (67 pages)

Browse Source 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank
2026-10-18 13:15:43 -07:00
parent 466fd477d8
commit e7f0d93c86
67 changed files with 2619 additions and 1152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Agent/Agent Docs and Edits.md
+7 -4
View File
@@ -3,7 +3,7 @@ type: reference
title: Agent Docs and Edits
description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
tags: [agents, docs, markdown, hashline, edit-recovery]
timestamp: 2026-10-05T00:00:00-07:00
timestamp: 2026-10-18T00:00:00Z
---
# Agent Docs and Edits
@@ -54,9 +54,12 @@ Gotchas: agents can write AFTER they appear finished (job settles, file
changes) — re-verify structure after the last background job reports, not
before. Editing an agent's already-damaged file with fresh line numbers is
safe, but its snapshot tag changes on every edit — re-read before each
subsequent edit. `.AGENTTRASH/` is gitignored: trash is local-only, so
nothing moved there reaches the repo — that is intended (temporary hold, not
archive).
subsequent edit. `.AGENTTRASH/` is gitignored (`.gitignore:25`), so NEW trash
stays local — but do not read that as "the repo has none of it": on 2026-10-18
`git ls-files | grep -c AGENTTRASH` returns **37** tracked paths
(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`,
`tests/.AGENTTRASH/`), because gitignore never untracks what was committed
before the rule. Trash is a temporary hold, not an archive.
## Hashline edit recovery
serverAppConfigs-docs/Agent/OMP Harness.md
+66 -30
View File
@@ -3,7 +3,7 @@ type: reference
title: OMP Harness
description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
tags: [omp, harness, compaction, models-yml, context, samplers]
timestamp: 2026-10-05T00:00:00-07:00
timestamp: 2026-10-18T00:00:00Z
---
# OMP Harness
@@ -25,6 +25,10 @@ picker/grants, the homepage docs and the raw API all use one id per model
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
note on radiance echoing the requested name).
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
192.168.0.143:7771 each answer with the unit name (`frank-archon`,
`frank-dragoon`, `frank-zealot`).
## Compaction chain arithmetic (verify before ANY window change)
When a fleet endpoint's context grows past its harness-declared window, OMP
@@ -35,27 +39,40 @@ breaks compaction.
Where the contract lives:
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL),
`methodOrder`, `thresholdPercent` (70), idle settings
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL:
`http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`),
`methodOrder` (`shake → snapcompact → soft → handoff → remote`,
`config.yml:77-82`), `thresholdPercent` (**95**, `config.yml:84`), idle
settings (`idleEnabled`, `idleThresholdTokens: 100000`)
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
`compactionModel` references
- Implementation: omp plugin `dist/core/compaction/compaction.js`
- Implementation: omp plugin
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes):
encodes). Recomputed from the live files 2026-10-18 — contextWindows
200,000 (archon) / 155,648 (dragoon) / 200,704 (zealot) at
`models.yml:48,131,206`; maxTokens 16,384 / 45,056 / 16,384 at
`models.yml:56,143,212`:
1. **Source trigger** = `thresholdPercent × source contextWindow`
(0.70 × 200,704 = 140,492 for frank-archon on the current shape)
2. **Output budget** = capped at the target model's `maxTokens`
(32,768 for frank-zealot; 36,864 frank-archon; 57,344 frank-dragoon — the
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
bundle clamps the percentage to 1–99 and converts it to a token threshold.
2. **Output budget** = capped at the target model's `maxTokens` — every role
here compacts to frank-zealot, so the budget is its 16,384.
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
200,704 (frank-archon); 140,490 + 57,344 = 197,834 ≤ 200,700
(frank-dragoon); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-zealot).
window. **At 95 % this no longer holds**: 190,000 + 16,384 = 206,384 >
200,704. The registry's own comments design around an 85 % trigger (archon:
170,000 est × 1.03 + 16,384 ≈ 191,500 ≤ 200,704, `models.yml:49-56`), which
does hold. While `config.yml` and those comments disagree, treat 85 % as the
design point and re-run this arithmetic after ANY window change — a 400 from
the compaction target mid-conversation is the symptom.
(Historic 2026-10-05 pairs, at `thresholdPercent: 70`: 140,492 + 36,864 =
177,356 ≤ 200,704 archon; 140,490 + 57,344 ≤ 200,700 dragoon; 140,492 +
32,768 ≤ 200,704 zealot.)
Procedure (recompute from the LIVE files every time — windows rotate with
every swap; the numbers above are the 2026-10-05 state):
every swap; the block above is the 2026-10-18 state):
1. Read the `config.yml` compaction block + `models.yml` entries for source
and target. Every endpoint's `contextWindow` comment carries its own
@@ -76,15 +93,22 @@ Known reconciliation trap: comments in models.yml have described compaction
as running "on the dankserver endpoint" while `remoteEndpoint` pointed at
dankclaw — the comment/config disagreed. The comments now name the real
target (`compaction.remoteEndpoint` = dankclaw :7771). **The config is
truth; fix comments in the same change.** `maxTokens` never changes for a
reslice — it caps output; `contextWindow` caps input. Boot-shape OOM
laddering is a different problem ([[Llama Cpp Serving Sizing]]).
truth; fix comments in the same change.** A fresh instance of the same trap is
open 2026-10-18: every `maxTokens` comment still says "compaction fires at
85 % of the window" while `config.yml:84` sets `thresholdPercent: 95`.
`maxTokens` never changes for a reslice — it caps output; `contextWindow` caps
input. Boot-shape OOM laddering is a different problem
([[Llama Cpp Serving Sizing]]).
## Per-model sampler config (`compat.extraBody`)
`models.yml` has **no per-model sampler keys**. Sampling is global
(`config.yml`: `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
`repetitionPenalty`; default `-1` = **not sent**). The per-model hook is
`models.yml` has **no per-model sampler keys**. The global knobs are
`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
`repetitionPenalty`, and a knob reaches the wire only when it is set — the
bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`),
and this workstation's `config.yml` sets **none** of them. The 2026-10-05 note
that the unset default is `-1`/"not sent" is retired: there is no `-1`
sentinel, an unset key is simply absent. The per-model hook is
`compat.extraBody`:
**Why it works:** `applyOpenAIExtraBody` runs **last** in the
@@ -117,15 +141,23 @@ Use **wire names**, not OMP camelCase. Two traps:
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
also sets it — `Object.assign` is shallow and replaces the whole object,
silently dropping the `reasoning_effort` kwarg the compat policy set.
Top-level `preserve_thinking` is honoured by halogen. (frank-archon's
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
a comment asking to verify-and-prune — the exact trap to check at that
endpoint's next swap.)
Top-level `preserve_thinking` is honoured by halogen. Live state 2026-10-18:
frank-archon's radiance entry carries `chat_template_kwargs:
{enable_thinking, preserve_thinking}` in `extraBody` (`models.yml:89-91`)
and nothing else sets that object at that endpoint, so the shallow-merge
hazard is latent rather than firing; the FreeToken-era "verify-and-prune at
the next swap" note is retired with that swap (2026-10-05,
`scripts/serve-frank-archon.sh:9-10`).
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
endpoint decodes greedy while looking configured. This is why frank-dragoon's
entry carries the recipe on the wire and a comment saying "do not drop it".
**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier flash-server
builds carried **no server-side sampler defaults**, so an absent
`temperature` decoded greedy while the endpoint looked configured. That era is
over: `HALOGEN_TEMPERATURE/TOP_P/TOP_K` now set the card's thinking recipe
(1.0 / 0.95 / 20) as engine defaults (`scripts/frank-dragoon.env:98-103`,
exported at `scripts/serve-frank-dragoon.sh:186-188`), so an absent temperature
is no longer greedy. frank-dragoon's `extraBody` still carries the same trio —
consistent, not load-bearing. Treat any older "do not drop it or you get
greedy" wording as history.
**Reasoning-effort folding.** Qwen3.8's template accepts only
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
@@ -169,7 +201,7 @@ never `rm`.
What loads statically every session (measure each, don't guess):
1. Repo rules file — `wc -c AGENTS.md` (baseline 2026-10-05: 72,179 B, ~18k
tokens — historically the largest block).
tokens — historically the largest block; re-measured 2026-10-18: 89,723 B).
2. Skill index — only names+descriptions load; bodies are on-demand. Sum
description bytes across all SKILL.md frontmatter:
```bash
@@ -186,7 +218,8 @@ What loads statically every session (measure each, don't guess):
EOF
```
Baseline 2026-10-05: 171 managed skills / ~38 KB descriptions + 9 user
skills.
skills. Re-measured 2026-10-18 with the same glob: **87** managed
`SKILL.md` + 9 user skills — the dedupe happened.
3. Memory lessons — `find ~/.omp/agent/memories -name 'MEMORY.md' -o -name
'memory_summary.md' | xargs wc -c`. The injected block ≈ the largest
MEMORY.md.
@@ -231,6 +264,9 @@ The static-context budget (measured 2026-10-05):
| Learned-lessons block | ~14 KB | can contain corrupted/duplicate captures |
| Harness system prompt + tool schemas | ~23 KB | fixed |
Re-measured 2026-10-18: the skill library is down to 87 managed skills, but
`AGENTS.md` has grown to 89,723 B — the dedupe landed, the slimming did not.
Audit procedure:
1. Build an inventory: per skill, name, mtime, body word count, description,
serverAppConfigs-docs/Architecture/E2E Suite.md
+11 -4
View File
@@ -5,7 +5,7 @@ tags:
- architecture
- e2e
- homepage
timestamp: 2026-09-01
timestamp: 2026-10-18
---
# E2E Suite — homepage + guard chain (Playwright)
@@ -28,16 +28,21 @@ user sees it; the `bats` seams assert config shape, not behavior.
| `specs/19-frankbook.spec.ts` | `frankbook` project: walks the six admin links that terminate on FrankBook-Pro (192.168.0.112) — conditional, see below |
| `specs/13-sso.spec.ts` | direct guarded-vhost visits resolve silently with ONLY the homepage session (single-login enforcement) |
| `specs/14-guard.spec.ts` | `guest` project, no storage state: unauthenticated traffic lands on the IdP login |
| `specs/15-unauth-fetch.spec.ts` | `guest` project: a wget-style client must never receive application bytes from a guarded service — it asserts on **leakage** (app-code markers), not on status code |
| `specs/16-marreta.spec.ts` | regression guard for the 2026-09-12 incident: the marreta tile resolving to the authentik login flow because the `marreta-proxy` ProxyProvider had no linked Application |
| `specs/17-hermes-vhost.spec.ts` | `hermes.franksdank.cloud` fronts a host-networked container, so the vhost upstream is the host LAN address; guards the 2026-09-16 loopback-bind 502 that the rest of the suite stayed green through |
| `specs/18-openwebui-sso.spec.ts` | `ai.franksdank.cloud` completes its OIDC round-trip silently with only the homepage session |
| `lib/login.ts` | `akLogin` helper (two-stage flow, shadow-DOM locators) |
| `lib/env.ts` | loads `.env.local`, exports `REAL_UA` |
| `tests/e2e_scaffold.bats` | bats seam: structure + no-credentials-tracked (never drives the fleet) |
| `tests/e2e_scaffold.bats` | bats seam: structure, placeholder-only example env, **every spec matched by a playwright project** (`tests/e2e_scaffold.bats:51`), and no credentials/auth state tracked. Never drives the fleet |
## Projects and auth
- **setup** — logs in as `qa-e2e` and `qa-e2e-admin` (fleet-managed IdP test
users; the admin is a `homepage-admins` member), writes
`.auth/qa-e2e{,-admin}.json` storage states.
- **standard** — 10/12/13 with the user state.
- **standard** — 10/12/13/16/17/18 with the user state
(`tests/e2e/playwright.config.ts:33`).
- **admin** — 11 with the admin state.
- **guest** — 14/15 with an empty state.
- **frankbook** — 19 with the admin state. **Excluded from every default run**: the
@@ -64,7 +69,9 @@ cd tests/e2e && npx playwright test # ALL projects, frankbook included
Requires LAN reachability of `www.franksdank.cloud` (Pi-hole DNS or public).
The suite is **not** part of `./tests/run.sh` — it needs the live fleet and
real IdP credentials.
real IdP credentials. It **is** a deploy gate: `scripts/deploy.sh` runs it and
aborts with `DEPLOY INCOMPLETE: e2e gate FAILED` when it does not pass
(`scripts/deploy.sh:278-285`).
## Gotchas (all paid for live)
serverAppConfigs-docs/Architecture/Network Architecture.md
+37 -16
View File
@@ -4,7 +4,7 @@ description: "One Caddy fronts both hosts."
tags:
- architecture
- network
timestamp: 2026-09-02
timestamp: 2026-10-18
---
# Network Architecture
@@ -40,7 +40,7 @@ graph TB
POST --> PROM[Prometheus :9090]
DIRECT -->|3000 + ssh 2222| GITEA[Gitea]
DIRECT -->|3005| OWEBUI[OpenWebUI]
DIRECT -->|3002| GRAFANA[Grafana :3000]
POST -->|authenticated| GRAFANA[Grafana :3000]
DIRECT -->|8083 web UI| PIHOLE[Pi-hole]
AUTH -->|login UI + sessions| AUTHDB[(Authentik Postgres :5432)]
SEARCH -.->|tunnel exit| GLU[Gluetun VPN]
@@ -53,21 +53,31 @@ graph TB
CAL[Calibre-Web :8083]
PLEX[Plex :32400]
JEL[Jellyfin :8096]
ARR[*arr stack + qBittorrent]
end
POST -->|romm| ROMM
subgraph FRANKBOOK["FrankBook-Pro 192.168.0.112"]
ARR[*arr stack + qBittorrent]
end
POST -->|files| COPY
DIRECT -->|plex plain-HTTP passthrough| PLEX
DIRECT --> CAL
DIRECT --> JEL
DIRECT --> ARR
DIRECT -->|own auth, no Anubis| ARR
```
Guarded (through Anubis): homepage, search, prometheus, romm, files.
Exempt by construction: `auth.` (the IdP login itself), plex (plain HTTP),
gitea (git smart-HTTP clients cannot solve JS challenges — `/info/refs` and
`git-*-pack` pass unchallenged), grafana (own auth).
Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`):
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`),
prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta
(`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.`
(the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP
clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP),
books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts
that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own
auth"). Grafana is **not** exempt even though it signs in through authentik as
an OAuth2 relying party (`:123`) — the chain wraps it too. `sonarr.` has no
vhost at all (LAN `192.168.0.112:8989` only, `:265-267`).
Dankserver runs no proxy of its own. Its services expose published LAN ports, and the Dankclaw Caddy routes the names to those ports.
@@ -78,20 +88,31 @@ gitea (git smart-HTTP clients cannot solve JS challenges — `/info/refs` and
| LAN | `romm.franksdank.cloud` → 192.168.0.143 | Pi-hole (`addn-hosts`) |
| WAN | `romm.franksdank.cloud` → 71.227.227.172 | Porkbun A record |
Users type a name and nothing else. Caddy maps the name to the upstream. The split between the two resolvers removes the need for router NAT loopback. See [[LAN-DNS]] for the flow and the router hand-off.
Users type a name and nothing else. Caddy maps the name to the upstream. The
split between the two resolvers removes the need for router NAT loopback.
Verified 2026-10-18: `dig +short @192.168.0.143 romm.franksdank.cloud` →
192.168.0.143, and a public resolver returns 71.227.227.172 for the same name.
See [[LAN-DNS]] for the flow and the router hand-off.
## TLS
Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through the DNS-01 challenge via the Porkbun module. No inbound port 80/443 is required. Details in [[Caddy]].
## Gluetun Path (SearXNG only)
## Gluetun Path (SearXNG + Marreta)
`gluetun.container`, `searxng.container` and `searxng-valkey.container` all set
`Network=searxng.network` in their Quadlet sources (runtime network
`systemd-searxng`); only those three share the tunnel. The old compose idiom
(`network_mode: service:gluetun`) is gone — a shared named network is what
carries SearXNG's outbound traffic through the VPN exit. Everything else on
Dankclaw uses the normal LAN route. This matches KB Pattern 2 in [[Gluetun]].
`gluetun.container` and `searxng-valkey.container` set
`Network=searxng.network` (runtime network `systemd-searxng`), while
`searxng.container`, `marreta.container` and `marreta-browser.container` join
the tunnel the modern way — `Network=gluetun.container`
(`dankclaw/searxng/quadlet/searxng.container:18`,
`dankclaw/marreta/quadlet/marreta.container:34`,
`dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet
equivalent of the retired compose `network_mode: service:gluetun`. **Five**
containers share the VPN exit, not three, and Marreta's outbound fetches leave
through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`);
everything else on Dankclaw uses the normal LAN route. This matches KB Pattern
2 in [[Gluetun]].
See [[Container Network Namespaces]] for which addresses are reachable from that
namespace (none of the unpublished host-netns listeners).
serverAppConfigs-docs/Architecture/Testing Policy.md
+5 -4
View File
@@ -4,7 +4,7 @@ description: "This repository tests its configuration at defined seams."
tags:
- architecture
- testing
timestamp: 2026-09-01
timestamp: 2026-10-18
---
# Testing Policy
@@ -16,13 +16,14 @@ This repository tests its configuration at defined seams. The tests run with `ba
| Seam | File | What it asserts |
|------|------|-----------------|
| Compose validity | `tests/compose_validity.bats` | YAML parses, services have images, no named volumes |
| Env consistency | `tests/env_consistency.bats` | every `${VAR}` is defined in `.env.example`; no real secrets; the live gitea password never appears |
| Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present |
| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page |
| Env consistency | `tests/env_consistency.bats` | every compose file has a **sibling** `.env.example` (`:9`) in which every `${VAR}` is defined (`:17`); placeholders only (`:31`); the live gitea database password never appears (`:37`) |
| Caddyfile validity | `tests/caddyfile.bats` | the Caddyfile lives on **dankclaw only** (`:15`, not per-host), flat `franksdank.cloud` names with no `.local` leftovers (`:20,:24`), balanced braces, `caddy validate` when the binary exists, the renderable homepage basicauth placeholder, the quadlet/Porkbun/loki/prometheus guardrails |
| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; `index.md` links every page; **no wikilink in the vault points at a page that does not exist** (`:37`) |
| Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available |
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
| Quadlet units | `tests/*quadlet*.bats` | every unit is boot-persistent, self-healing, repo-provenanced, secret-free |
| E2E (browser) | `tests/e2e/` + `tests/e2e_scaffold.bats` | Playwright drives the live guard chain; the bats seam asserts structure and credential hygiene. See [[E2E Suite]] |
| Per-service / unit seams | `tests/llm_units.bats`, `tests/frank_{archon,dragoon,zealot}_unit.bats`, `tests/llm_secrets.bats`, `tests/homepage_{caddy,checker,config,deploy,pages}.bats`, `tests/{anubis,authentik,hermes_webui,pihole,mount_entropy,searxng_limiter,autoresearch_measure,changed_tests}.bats`, `tests/{caddy,dankserver,frankbook,marreta,observability,open_webui,searxng}_quadlet.bats`, `tests/comfyui_rocm.bats` | the ~30 named seams the summary rows above do not list: LLM systemd-unit shape and env hygiene, homepage config/pages/deploy behaviour, and one detail-seam per deployed service |
## Focused pre-commit testing
serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md
+42 -20
View File
@@ -12,24 +12,31 @@ timestamp: 2026-10-05
Two ready-to-load text-to-image workflows for the Qwen-Image 2.1 payload
already on Dankclaw, both proven end-to-end on 2026-10-05 by POSTing to the
live API on port 8188 and verifying the rendered PNG. The repo copies are the
live API on port 8188 and verifying the rendered PNG (the renders are still on
disk: `qwen21-art_00001_.png` 22:41 and `qwen21-text_00001_.png` 22:44,
`ls -l /home/dank/comfyui-data/output/` 2026-10-18). The repo copies are the
source of truth; the host copies under `/home/dank/comfyui-data/workflows/`
are for convenience.
are for convenience and are byte-identical today (`md5sum` matches both files,
2026-10-18).
## What is on disk
Sizes are GiB, measured `ls -l` on the host 2026-10-18 (all files dated
2026-09-21).
| Path (under `/home/dank/comfyui-data/models/`) | Size | Role |
|---|---|---|
| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 14 GB | Full-precision fallback |
| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB | GGUF alternative (needs ComfyUI-GGUF custom node) |
| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.8 GB | Mid quant |
| `text_encoders/qwen3vl_8b_bf16.safetensors` | 17 GB | Full precision |
| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB | 2.1 VAE (used) |
| `vae/qwen_image_vae.safetensors` | 243 MB | Smaller VAE |
| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB (7,256,783,064 B) | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 13.3 GB (14,230,280,616 B) | Full-precision fallback |
| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB each (7,591,551,648 / 7,591,551,680 B) | GGUF alternative (needs ComfyUI-GGUF custom node, present in `custom_nodes/`) |
| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB (6,312,105,364 B) | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.7 GB (9,350,798,360 B) | Mid quant |
| `text_encoders/qwen3vl_8b_bf16.safetensors` | 16.3 GB (17,534,334,616 B) | Full precision |
| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB (675,509,688 B) | 2.1 VAE (used) |
| `vae/qwen_image_vae.safetensors` | 243 MB (253,806,246 B) | Smaller VAE |
No LoRAs are on disk yet — see [Recommended additions](#recommended-additions).
No LoRAs are on disk yet, and `models/lora/` / `models/controlnet/` do not
exist — see [Recommended additions](#recommended-additions).
## The workflows
@@ -45,7 +52,13 @@ EmptySD3LatentImage ─────────┴─ KSampler ─ VAEDecode ─
```
**Common sampler settings:** `cfg 1.0` (Qwen-Image 2.1 expects CFG-free
sampling), `euler` + `simple`, seed fixed in the file.
sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed
fixed in the file (`1930` in the art workflow, `2026` in the text one) —
read from `dankclaw/comfyui/workflows/*.json` node `9`.
Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`,
`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`,
`SaveImage` (ids 3–11).
| Workflow | File | Shape | Steps | Made for |
|---|---|---|---|---|
@@ -68,8 +81,9 @@ prompt is written for it:
Verified render: `SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS` + `DANKCLAW
DANKSERVER EST 2026` came back crisp and correctly spelled at 832×1216
(`qwen21-text_00001_.png`, 953 KB). One caveat from the same render: a
shorter secondary line is where misspellings creep in — proofread the first
(`qwen21-text_00001_.png`, 953,254 B — still on disk, `ls -l
/home/dank/comfyui-data/output/` 2026-10-18). One caveat from the same render:
a shorter secondary line is where misspellings creep in — proofread the first
render before shipping.
## Driving it from the API
@@ -89,20 +103,28 @@ Outputs land in `/home/dank/comfyui-data/output/`. The queue is FIFO — the
two workflows above ran back-to-back unattended.
**Note:** the container's UI-saved workflows live in the ephemeral
`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — hence the
host-side `workflows/` dir and the repo copy.
`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — the unit mounts
only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
(`podman inspect comfyui` 2026-10-18; `dankclaw/comfyui/quadlet/comfyui.container:48-52`)
— hence the host-side `workflows/` dir and the repo copy.
## Recommended additions
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
frank-zealot's). All repos verified 2026-10-05. None are on disk yet.
Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`,
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`,
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`,
`huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk
yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18).
| # | What | Repo | Grab | Size | Why |
|---|---|---|---|---|---|
| 1 | **Lightning 8-step LoRA** (T2I speed) | `lightx2v/Qwen-Image-Lightning` | `qwen_image_lightning_8steps_*.safetensors` (LoRA) | ~1–2 GB | 20 steps → 8 with a `LoraLoader` node; 2–3× faster renders on the iGPU, negligible quality loss |
| 2 | **Qwen-Image-Edit-2511 GGUF** (image editing) | `unsloth/Qwen-Image-Edit-2511-GGUF` | Q4_K_M (~13.2 GB) or Q4_0 (~11.9 GB) | ~12–13 GB | Unlocks img2img "edit this photo" workflows — a capability class the fleet has zero of; Q4 keeps it iGPU-friendly |
| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI |
| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` CLIP type is already in this build's loader enum) |
| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) |
| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) |
Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
serverAppConfigs-docs/Dankclaw/ComfyUI.md
+52 -28
View File
@@ -11,27 +11,31 @@ timestamp: 2026-09-16
ComfyUI is a node-based AI image generation engine. Workflows are built from nodes that load models, sample latents, and post-process images; everything is scriptable through a web UI and an API.
**Host:** Dankclaw — **Port:** `8188` published on the host; **no vhost and no
guard chain** — ComfyUI is LAN-only by construction (the Caddyfile routes no
name to it) — **Backend:** ROCm GPU (gfx1151) — **Docs:**
https://github.com/Comfy-Org/ComfyUI
**Host:** Dankclaw — **Port:** `0.0.0.0:8188` published, **no auth** (`ss -ltnp`
2026-10-18) — **no vhost and no guard chain**: the Caddyfile routes no name to
it (no `comfyui` or `8188` line anywhere in `dankclaw/Caddyfile`), so LAN access
rides the Pi-hole A record `comfyui.franksdank.cloud`
(`dankclaw/pihole/addn-hosts:37`) plus the raw port — the homepage card links
`http://comfyui.franksdank.cloud:8188` and probes `127.0.0.1:8188`
(`shared/homepage/services.json:519-521`) — **Backend:** ROCm GPU (gfx1151) —
**Docs:** https://github.com/Comfy-Org/ComfyUI
## Facts
| Item | Value |
|------|-------|
| Image | `comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui` |
| Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` |
| Upstream latest | 0.37.0 (2026-09-21) — ships **native Qwen-Image 2.1** support (CORE-423, `comfy/ldm/qwen_image21/`). The image's custom GGUF patches may be unnecessary on the next rebuild; check before re-applying. |
| Port | 8188 (host) → 8188 (container) |
| Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag. |
| Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` |
| Run | `./scripts/deploy.sh dankclaw --service comfyui` (Quadlet unit `comfyui.service`) |
| Boot-persistent | Quadlet unit `dankclaw/comfyui/quadlet/comfyui.container` → `~/.config/containers/systemd/` |
| Image | `localhost/comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui`, `Pull=never` (`dankclaw/comfyui/quadlet/comfyui.container:41-42`; `podman inspect comfyui` 2026-10-18) |
| Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` (`dankclaw/Containerfile.comfyui:9`) |
| Upstream latest | 0.37.0 — the version the running container reports (`GET http://127.0.0.1:8188/system_stats`, 2026-10-18). The release date 2026-09-21 and its **native Qwen-Image 2.1** support (CORE-423, `comfy/ldm/qwen_image21/`) *(unverified 2026-10-18)* — the image's own startup patch still rewrites detection and VAE-decode code at every boot (`dankclaw/apply_qwen21_patch.py:4,23,37`, run by `dankclaw/comfyui-start.sh:3`), so check before re-applying on the next rebuild. |
| Port | 8188 (host) → 8188 (container), published `0.0.0.0` (`dankclaw/comfyui/quadlet/comfyui.container:44`) |
| Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag — the unit runs `python main.py --listen 0.0.0.0 --port 8188` (`dankclaw/comfyui-start.sh:4`). `rocm-smi --showproductname` reports `GFX Version: gfx1151` (2026-10-18). |
| Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` (`dankclaw/comfyui/quadlet/comfyui.container:48-51`) |
| Run | `./scripts/deploy.sh dankclaw --service comfyui` (Quadlet unit `comfyui.service`; handler `deploy_comfyui`, `scripts/deploy-remote.sh:649`) |
| Boot-persistent | Quadlet unit `dankclaw/comfyui/quadlet/comfyui.container` → `~/.config/containers/systemd/`; live 2026-10-18 `systemctl --user is-enabled comfyui` → `generated`, `is-active` → `active` |
## Why the GPU works here
Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ship gfx1151 kernels. So `torch.cuda.is_available()` returns `False` and ComfyUI falls back to CPU. AMD publishes a gfx1151-tagged build, and the image above is based on it. On this APU it reports `torch.cuda.is_available() = True`, device `Radeon 8060S`, capability `(11, 5)`. See [[ComfyUI ROCm and Vulkan]] for the full verification.
Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ship gfx1151 kernels. So `torch.cuda.is_available()` returns `False` and ComfyUI falls back to CPU. AMD publishes a gfx1151-tagged build, and the image above is based on it. On this APU it reports `torch.cuda.is_available() = True`, device `Radeon 8060S Graphics` (`GET http://127.0.0.1:8188/system_stats` 2026-10-18: one device, `cuda:0 Radeon 8060S Graphics : native`, type `cuda`), capability `(11, 5)` (`dankclaw/Containerfile.comfyui:4-5`). See [[ComfyUI ROCm and Vulkan]] for the full verification.
## How the image is built
@@ -39,16 +43,22 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not
## GPU device access (rootless), and why the unit exists
The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`) and uses
The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`,
`dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses
`GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman
cannot map the group name into the user namespace. The companion udev rule
`dankclaw/udev/99-comfyui-gpu-access.rules` keeps those nodes `MODE 0666`, which
is what lets a **user-unit spawn** — carrying no supplementary groups — open
them anyway. Verified inside the running container: `torch.cuda.is_available()`
is `True` and the device is `Radeon 8060S Graphics`.
them anyway. The rule is installed on the host
(`/etc/udev/rules.d/99-comfyui-gpu-access.rules`, `ls` 2026-10-18) and the nodes
are world-writable today: `crw-rw-rw- /dev/kfd`, `crw-rw-rw-
/dev/dri/renderD128` (`ls -l` 2026-10-18). Verified through the running
service: `/system_stats` reports the CUDA-type device, i.e.
`torch.cuda.is_available()` is `True` and the device is `Radeon 8060S Graphics`.
**Incident, 2026-09-16 — why this is a Quadlet now.** ComfyUI ran from
`dankclaw/compose.yaml` through a hand-written `~/.config/systemd/user/
**Incident, 2026-09-16 (retired era) — why this is a Quadlet now.** ComfyUI ran
from `dankclaw/compose.yaml` (no longer in the repo) through a hand-written
`~/.config/systemd/user/
comfyui.service` that called `podman compose up -d`. Its container was created
*before* the GPU declarations were added to the compose service, and
`restart: unless-stopped` then kept restarting that stale container: no
@@ -58,7 +68,8 @@ fine; the container was old. A `--force-recreate` would have fixed it that day,
but the class of bug needed the unit: one owner per container name, repo-tracked,
and recreated from the source of truth on every deploy. The compose service is
deleted; `deploy_comfyui` retires any compose-era unit into `~/.AGENTTRASH/`
and removes a stale container before installing the unit.
(`scripts/deploy-remote.sh:649,660`) and removes a stale container before
installing the unit.
## Volumes
@@ -75,7 +86,10 @@ The ComfyUI code is baked into the image (not mounted). Rebuild the image to upd
Custom nodes go in `/home/dank/comfyui-data/custom_nodes` (mounted at `/opt/ComfyUI/custom_nodes`). ComfyUI loads every subdirectory there on start. Because this path is a bind mount, it shadows anything baked into the image — so custom nodes are installed host-side, not in the Containerfile.
**ComfyUI-Manager** is installed here. Install it with a one-time clone, then restart the container:
**ComfyUI-Manager** is installed here, alongside **ComfyUI-GGUF** (`ls
/home/dank/comfyui-data/custom_nodes` 2026-10-18 — the GGUF node's Python deps
are baked into the image, `dankclaw/Containerfile.comfyui:29`). Install Manager
with a one-time clone, then restart the container:
```bash
git clone https://github.com/ltdrdata/ComfyUI-Manager \
@@ -83,19 +97,29 @@ git clone https://github.com/ltdrdata/ComfyUI-Manager \
systemctl --user restart comfyui
```
ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGithub, matrix-nio, transformers, and more). To confirm it loaded, query `GET /manager/version` (returns a version like `V3.41`). Its pip installs live in the container's writable layer, so they survive a restart and a reboot but not an image rebuild; after a rebuild, Manager re-installs them on the next start.
ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGithub, matrix-nio, transformers, and more). To confirm it loaded, query `GET /manager/version` — it answers `V3.41` today (`curl http://127.0.0.1:8188/manager/version` 2026-10-18). Its pip installs live in the container's writable layer, so they survive a restart and a reboot but not an image rebuild; after a rebuild, Manager re-installs them on the next start.
## Boot persistence
The Quadlet unit carries `[Install] WantedBy=default.target` with
`Restart=always`, so the generator wires it into `default.target` and it comes
back on its own (user linger is on). Inspect it with
`systemctl --user is-enabled comfyui.service` → `generated`; an explicit
`systemctl --user enable` fails on generated units.
`Restart=always` and `RestartSec=10`
(`dankclaw/comfyui/quadlet/comfyui.container:66-70`), so the generator wires it
into `default.target` and it comes back on its own (user linger is on:
`loginctl show-user dank -p Linger` → `yes`). Inspect it with
`systemctl --user is-enabled comfyui.service` → `generated`, `is-active` →
`active` (both 2026-10-18); an explicit `systemctl --user enable` fails on
generated units.
## Healthcheck
The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `/system_stats` reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device).
The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/`
2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats`
reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The
health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`,
bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
(`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs
`http://127.0.0.1:8188/` inside the container, and the container reports
`healthy` (`podman inspect comfyui` 2026-10-18).
## Gotchas
@@ -103,8 +127,8 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `
- Do not add the `render` group by name. Rootless podman cannot map it; the udev rule plus `keep-groups` is the mechanism that works.
- **A stale container outlives a compose edit.** If the GPU declarations change, recreate the container (`deploy.sh dankclaw --service comfyui` removes the old one) — `restart: unless-stopped` will happily keep an old config alive forever.
- Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
- The base image is about 32 GB. A code update means a rebuild, because the code is baked in.
- The model collection starts empty. Add models to `/home/dank/comfyui-data/models` before generating.
- The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
- The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.
## See Also
serverAppConfigs-docs/Dankclaw/Gitea.md
+23 -18
View File
@@ -14,23 +14,24 @@ timestamp: 2026-08-26
Gitea is a self-hosted Git service with issues, pull requests, and a web UI. On Dankclaw it runs as two Quadlet-managed containers in the rootless Podman session of user `dank`. The canonical Caddy on the same host reverse-proxies its web UI at `gitea.franksdank.cloud`.
**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`) — **Proxy domain:** `gitea.franksdank.cloud` — **Docs:** https://docs.gitea.com
**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`), git SSH on `2222` (`ss -ltnp` 2026-10-18) — **Proxy domain:** `gitea.franksdank.cloud` — **Docs:** https://docs.gitea.com
## Containers
| Container | Unit | Image | Purpose |
|-----------|------|-------|---------|
| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` | Database |
| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` | Web UI, API, git (HTTP :3000, SSH :2222) |
| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` (`dankclaw/gitea/quadlet/gitea-postgres.container:18`) | Database |
| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` (`dankclaw/gitea/quadlet/gitea-server.container:22`) — answers `GET /api/v1/version` → `1.27.3` (2026-10-18) | Web UI, API, git (HTTP :3000, SSH :2222, `dankclaw/gitea/quadlet/gitea-server.container:32-33`) |
The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name. The automatic `podman` default network has DNS disabled. Do not run the Gitea containers on it.
The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name; `podman network ls` 2026-10-18 confirms `systemd-gitea dns=true` while the automatic `podman` default network is `dns=false`. Do not run the Gitea containers on it.
## Hosted repositories
| Repository | Purpose |
|------------|---------|
| `dank/serverAppConfigs` | this repo — Podman stacks + LLM fleet |
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) |
| `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) |
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* |
The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)
@@ -57,9 +58,9 @@ graph LR
## Boot persistence
User-level linger is enabled for `dank` (`loginctl enable-linger dank`). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target`. The podman generator applies it at boot; no symlink is needed.
User-level linger is enabled for `dank` (`loginctl enable-linger dank`; `loginctl show-user dank -p Linger` → `yes`, 2026-10-18). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target` with `Restart=always` (`dankclaw/gitea/quadlet/gitea-server.container:41,43-44`; `gitea-postgres.container:28,30-31`); live 2026-10-18 both report `systemctl --user is-enabled` → `generated` and `is-active` → `active`. The podman generator applies the install at boot; no symlink is needed.
The containers use the quadlet fresh-start model. A stop removes the container (`podman run --rm --replace`). The next start recreates it. The bind-mounted data survives. The canonical Caddy is managed separately by the Dankclaw stack.
The containers use the quadlet fresh-start model. A stop removes the container — the generated unit runs `podman run --name gitea-server --replace --rm …` with `ExecStopPost=-/usr/bin/podman rm -v -f -i gitea-server` (`systemctl --user cat gitea-server.service` 2026-10-18). The next start recreates it. The bind-mounted data survives. The canonical Caddy is managed separately by the Dankclaw stack.
## Secrets
@@ -90,22 +91,26 @@ Per the live quadlet units (`Volume=` lines):
| Container path | Host path | Contents |
|----------------|-----------|----------|
| `/data` | `/home/dank/Containers/gitea/data` | Git repositories, app.ini, attachments |
| `/var/lib/postgresql/data` | `/home/dank/Containers/gitea/postgres` | PostgreSQL data |
| `/data` | `/home/dank/Containers/gitea/data` | Git repositories, app.ini, attachments (`dankclaw/gitea/quadlet/gitea-server.container:34`) |
| `/var/lib/postgresql/data` | `/home/dank/Containers/gitea/postgres` | PostgreSQL data (`dankclaw/gitea/quadlet/gitea-postgres.container:23`) |
Back up `/home/dank/Containers/gitea/` (tar) or `pg_dump` the database. See the knowledge base page [[Gitea]].
## Gotchas
- **Data files are owned by `dank` (uid 1000), not root.** The quadlet units run
podman in the user manager with no `--user`/UserNS override, so container uid
1000 is literally dank's host uid — there is no subuid offset for these
containers. Edit files as the owning user with
`podman exec --user 1000 gitea-server ...`.
- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
- **Port 3000, and one owner.** The live host port is 3000, published by the `gitea` Quadlet unit. The `dankclaw/compose.yaml` that mapped 3001 was **retired on 2026-09-16** (staged in `.AGENTTRASH/`) once an ownership audit showed every container on this host reports a `PODMAN_SYSTEMD_UNIT` label — a second composition path could only spawn a duplicate fighting a unit for a name. Do not reintroduce a compose service for Gitea or the proxy.
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it. Its containers collide with the Quadlet units on ports 3000 and 2222.
- **Canonical proxy.** `gitea.franksdank.cloud` is served by the canonical Caddy **unit** (`dankclaw/caddy/quadlet/caddy.container`, repo `dankclaw/Caddyfile`), which proxies to `192.168.0.143:3000`. Do not recreate a Gitea-specific Caddy unit or publish another proxy on ports 80/443. If a legacy unit is ever installed again, the deployer stages it under `~/.AGENTTRASH/YYYYMMDD-gitea-caddy/` before starting canonical Caddy.
- **Gitea's uid 1000 is NOT dank's host uid.** The units run podman in the user
manager with no `UserNS=`/`--user` override (`podman inspect gitea-server
--format '{{json .HostConfig.IDMappings}}'` → `null`, i.e. the default rootless
map), so container uid 1000 (the `git` user) lands on host uid **100999**
(the first subuid plus 999): `ls -ln /home/dank/Containers/gitea/data` shows
`git/` and `gitea/` owned by `100999` while `ssh/`, created host-side, is
`1000` (`ls -ln` 2026-10-18). Edit app data through the container instead of
the host path: `podman exec --user 1000 gitea-server ...`.
- **The host gets rebooted.** The 2026-08-16 incident below was an `unattended-upgrades` reboot; today `unattended-upgrades` is enabled but `Unattended-Upgrade::Automatic-Reboot` is not set anywhere in `/etc/apt/apt.conf.d/` (`apt-config dump` and `grep` 2026-10-18 — the key is present only commented-out at `50unattended-upgrades:94`). Do not rely on the trigger: the Quadlet units above are what make Gitea survive any reboot.
- **Port 3000, and one owner.** The live host ports are 3000 (HTTP) and 2222 (git SSH), published by the `gitea-server` Quadlet unit (`dankclaw/gitea/quadlet/gitea-server.container:32-33`; `ss -ltnp` 2026-10-18). The `dankclaw/compose.yaml` that mapped 3001 was **retired on 2026-09-16** (staged in `.AGENTTRASH/`) once an ownership audit showed every container on this host reports a `PODMAN_SYSTEMD_UNIT` label — a second composition path could only spawn a duplicate fighting a unit for a name. Do not reintroduce a compose service for Gitea or the proxy.
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/` (`ls` 2026-10-18 shows the file there, 1,278 B). Do not run it. Its containers collide with the Quadlet units on ports 3000 and 2222.
- **Canonical proxy, and it is exempt.** `gitea.franksdank.cloud` is served by the canonical Caddy **unit** (`dankclaw/caddy/quadlet/caddy.container`, repo `dankclaw/Caddyfile`), which proxies to `192.168.0.143:3000` and is on the **exempt** list — no Anubis challenge, no Authentik gate — because git clients cannot solve a JS proof; Gitea's own login is the only gate. Do not recreate a Gitea-specific Caddy unit or publish another proxy on ports 80/443. If a legacy unit is ever installed again, the deployer stages it under `~/.AGENTTRASH/YYYYMMDD-gitea-caddy/` before starting canonical Caddy (`scripts/deploy-remote.sh:347,365-370`).
- **`/metrics` is off, so Prometheus cannot scrape Gitea.** `curl http://127.0.0.1:3000/metrics` → **404** and the container carries no metrics-related environment variable (`podman inspect gitea-server` filtered 2026-10-18; no `ENABLE_METRICS`/`PROMETHEUS` line in either quadlet). The `gitea` job (`shared/prometheus/prometheus.yml:46-49`, target `192.168.0.143:3000`, `metrics_path: /metrics`) is therefore DOWN with `server returned HTTP status 404 Not Found` (`/api/v1/targets` 2026-10-18). Gitea is covered by the homepage status probe, not by metrics.
## Incident: 2026-08-16 — down after unattended reboot
serverAppConfigs-docs/Dankclaw/Gluetun.md
+17 -14
View File
@@ -11,17 +11,17 @@ timestamp: 2026-08-28
Gluetun is a lightweight VPN client container. It connects to a VPN provider (WireGuard or OpenVPN) and shares that tunnel with other containers that attach to its network namespace.
**Host:** Dankclaw — **Docs:** https://github.com/qdm12/gluetun
**Host:** Dankclaw — **Tunnel:** WireGuard (`VPN type: wireguard`, journal 2026-10-18) to ProtonVPN US-WA per `dankclaw/searxng/quadlet/gluetun.container:6` — **Exit:** `149.22.88.19`, United States, Washington, Seattle (`journalctl --user -u gluetun`, `[ip getter]`, 2026-10-18; the journal never names the provider, so the provider attribution is the repo's) — **Docs:** https://github.com/qdm12/gluetun
## Facts (from research, primary source)
| Item | Value |
|------|-------|
| Image | `qmcgaw/gluetun:latest` |
| Capabilities | `NET_ADMIN` required |
| Device | `/dev/net/tun:/dev/net/tun` |
| Ports published | searxng's 8081 → 8080 lives here |
| Tunnel check | the `tun0` interface exists (the `:8000` server 404s on all paths) |
| Image | `docker.io/qmcgaw/gluetun:latest` (`dankclaw/searxng/quadlet/gluetun.container:19`; live 2026-10-18). Tagged `:latest`, so a redeploy can move the version — pin it deliberately if that ever matters. |
| Capabilities | `NET_ADMIN` required (`dankclaw/searxng/quadlet/gluetun.container:20`) |
| Device | `/dev/net/tun:/dev/net/tun` (`dankclaw/searxng/quadlet/gluetun.container:21`) |
| Ports published | Two, both on this container because it owns the netns: `8081 → 8080` (SearXNG) and `8082 → 8085` (Marreta) (`dankclaw/searxng/quadlet/gluetun.container:31-32`; `podman inspect gluetun --format '{{json .NetworkSettings.Ports}}'` and `ss -ltnp` 2026-10-18). Neither consumer container publishes anything. |
| Tunnel check | the `tun0` interface exists — the boot config dump logs `Network interface: tun0` (`journalctl --user -u gluetun` 2026-10-18). The control server listens on `:8000` **inside the netns only** (`.NetworkSettings.Ports` shows `"8000/tcp": null`, 2026-10-18), so nothing on the host can reach it. |
## Required Environment
@@ -32,8 +32,8 @@ environment:
WIREGUARD_PRIVATE_KEY: "${WIREGUARD_PRIVATE_KEY}" # from .env, never committed
```
- `VPN_TYPE` default is `openvpn`; WireGuard preferred when the provider supports it.
- Provider-specific variables come from the Gluetun wiki; set them in `.env`.
- `VPN_TYPE` default is `openvpn`; WireGuard preferred when the provider supports it (https://github.com/qdm12/gluetun-wiki/blob/main/setup/options/vpn.md). This deployment runs WireGuard (`WIREGUARD_*` variables, `dankclaw/searxng/quadlet/gluetun.container:24-30`).
- Provider-specific variables come from the Gluetun wiki (https://github.com/qdm12/gluetun-wiki); set them in `.env`.
## Optional DNS (from research)
@@ -44,11 +44,13 @@ environment:
| `DNS_CACHING` | `on` | cache queries |
| `BLOCK_MALICIOUS` | `on` | block malicious hostnames |
Names and defaults match the upstream wiki (https://github.com/qdm12/gluetun-wiki/blob/main/setup/options/dns.md); live confirmation 2026-10-18: `[dns] ready and using DNS server with dot upstream resolvers` and `[dns] downloading hostnames and IP block lists`.
Warning from the wiki: do NOT set `DNS_UPSTREAM_PLAIN_ADDRESSES` to a LAN/bridge address — DNS queries leak out of the tunnel.
## Attaching a Container
The searxng unit reuses gluetun's network namespace:
Three units reuse gluetun's network namespace — SearXNG and both Marreta containers:
```ini
[Container]
@@ -56,11 +58,11 @@ ContainerName=searxng
Network=gluetun.container
```
`Network=gluetun.container` also adds a systemd dependency on gluetun, so searxng starts after the tunnel.
`Network=gluetun.container` also adds a systemd dependency on gluetun, so the consumer starts after the tunnel (`dankclaw/searxng/quadlet/searxng.container:13-18`, `dankclaw/marreta/quadlet/marreta.container:29-34`, `dankclaw/marreta/quadlet/marreta-browser.container:20-25`). Live 2026-10-18 all three report `HostConfig.NetworkMode=container:03a75714…`, which is gluetun's own container id (`podman inspect gluetun --format '{{.Id}}'`) — one netns, four containers, ports published only by gluetun.
## Boot Persistence
gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio). It sets `WantedBy=default.target` and user linger is on, so the tunnel starts at boot. The WireGuard credentials live in `/opt/searxng/.env` (mode 0600). Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`.
gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio). It sets `WantedBy=default.target` and `Restart=always` (`dankclaw/searxng/quadlet/gluetun.container:37,39-40`) and user linger is on, so the tunnel starts at boot — live 2026-10-18 `systemctl --user is-enabled gluetun` → `generated`, `is-active` → `active`, `NRestarts=0` since the 01:25 boot. The WireGuard credentials live in `/opt/searxng/.env` (mode 0600 — `-rw------- dank:dank`, `ls -l` 2026-10-18). Deploy with `scripts/deploy.sh dankclaw --service gluetun` (`scripts/deploy-remote.sh:596-597`); `scripts/deploy-searxng-quadlet.sh dankclaw` is a compatibility wrapper that now forwards to `deploy.sh --service searxng`, so it does **not** deploy the tunnel.
## Verify the Tunnel
@@ -68,13 +70,14 @@ gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio).
podman exec gluetun curl -s ifconfig.me
```
The returned IP must differ from the host IP.
The returned IP must differ from the host IP. A read-only alternative is the journal: gluetun logs its own egress at start, and on 2026-10-18 it logged `[ip getter] Public IP address is 149.22.88.19 (United States, Washington, Seattle …)` — not the host's address.
## Gotchas
- The container that holds the published port must be the one with the ports mapping (gluetun).
- `restart: always` on gluetun so the tunnel reconnects after VPN drops.
- The container that holds the published port must be the one with the ports mapping (gluetun) — searxng, marreta and marreta_browser publish nothing of their own (`dankclaw/searxng/quadlet/searxng.container` and `dankclaw/marreta/quadlet/*.container` have no `PublishPort=`; `ss -ltnp` 2026-10-18 shows only gluetun's rootlessport on 8081/8082).
- `Restart=always` on gluetun (`dankclaw/searxng/quadlet/gluetun.container:37`) so the tunnel reconnects after VPN drops.
- When gluetun restarts, attached containers inherit the new network state; searxng may need a restart to re-resolve DNS.
- The gluetun health-check probe cannot open ICMP sockets: it logs `listening for ICMP packets: not permitted: you can try adding NET_RAW capability to resolve this; permanently falling back to plain DNS` (journal 2026-10-18). The unit grants `NET_ADMIN` only; the fallback works, so this is expected noise, not a fault.
## See Also
serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md
+69 -31
View File
@@ -19,64 +19,82 @@ the Hermes agent in-process from the shared `HERMES_HOME`, so it is the *same*
agent — same memory, skills, and sessions — as the [[Hermes]] gateway and the
[[Hermes-Dashboard]].
**Host:** Dankclaw — **Port:** `0.0.0.0:8787` (all interfaces) — **Ingress:** SSH tunnel for the phone app; direct LAN access also works
**Host:** Dankclaw — **Port:** `0.0.0.0:8787` (all interfaces; `ss -ltnp`
2026-10-18 shows the webui listening there in the host netns) — **Ingress:** SSH
tunnel for the phone app; direct LAN access also works (re-measured 2026-10-18
from another LAN host: `/health` → 200, `/` → 302) — **Vhost:**
`hermes.franksdank.cloud`, **exempt by construction** — no Anubis, no Authentik;
`HERMES_WEBUI_PASSWORD` is the only gate
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
| Command | `python3 /webui/bootstrap.py --foreground` |
| Source checkout | `/home/dank/Containers/hermes/webui` on Dankclaw, bind-mounted at `/webui` |
| Port | `8787`, bound to `0.0.0.0` in the **host** network namespace (`Network=host`) — LAN-reachable, not loopback-only |
| Auth | `HERMES_WEBUI_PASSWORD` (host `.env`; Hermex signs in with it) |
| Image | `localhost/hermes-agent:latest` with `Pull=never` (`dankclaw/hermes/quadlet/hermes-webui.container:35-36`) — the same image as the gateway and dashboard. The deployer keeps that alias fresh by tagging the upstream name into local storage (`podman tag docker.io/nousresearch/hermes-agent:latest localhost/hermes-agent:latest`, `scripts/deploy-remote.sh:1056`); the installed image is the 2026-09-24 build, revision `749220e` (`podman image inspect` 2026-10-18). |
| Command | `python3 /webui/bootstrap.py --foreground` (`hermes-webui.container:48`) |
| Source checkout | `/home/dank/Containers/hermes/webui` on Dankclaw, bind-mounted at `/webui` (`hermes-webui.container:47`; live dir `drwxrwxr-x dank dank`) |
| Port | `8787`, bound to `0.0.0.0` in the **host** network namespace (`Network=host` + `HERMES_WEBUI_HOST=0.0.0.0`, `hermes-webui.container:37-40`) — LAN-reachable, not loopback-only |
| Auth | `HERMES_WEBUI_PASSWORD` (from `/home/dank/serverAppConfigs/dankclaw/.env`, the unit's only `EnvironmentFile`; Hermex signs in with it) |
| Data | bind mount `/home/dank/Containers/hermes/data` → `/data`, shared with the gateway and dashboard; anonymous volume at `/opt/data` |
| Health | `HealthCmd=` in the Quadlet unit probes `http://127.0.0.1:8787/health` from inside the unit (works because the webui runs in the host netns); the deployer probes the same path from the host |
| Health | `HealthCmd=` in the Quadlet unit probes `http://127.0.0.1:8787/health` from inside the unit (`hermes-webui.container:53`; works because the webui runs in the host netns). Container state 2026-10-18: **healthy**. There is no host-side probe in the deployer — the contract is the seam `tests/hermes_webui.bats`, which reads the live unit and pins the Caddyfile upstream |
## Why a container, not a host process
The gateway's state files are owned by the container-mapped uid (`109999`,
from rootless Podman's `1 → 100000/65536` subuid range). A fresh rootless
from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` →
`dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 =
109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless
namespace on Dankclaw maps identically (verified with `podman unshare
cat /proc/self/uid_map`), so a second container using the same image reads
and writes the shared volume with exactly the ownership the gateway already
uses. A host-side process running as `dank` (uid 1000) could not read the
gateway's `0600` state files.
gateway's `0600` state files (a non-privileged read of the data dir is
`EACCES`, checked 2026-10-18).
## How it works
- `bootstrap.py` creates `/webui/.venv` on first boot, pip-installs the WebUI
and agent requirements, then **execs** `server.py` as one foreground process
(`--foreground`). systemd-free: Podman supervises it via
`restart: unless-stopped`.
- `HERMES_WEBUI_SKIP_ONBOARDING=1` keeps the first-run wizard off; a profile
(`--foreground`). systemd-free inside the container: supervision is the unit's
`Restart=always` / `RestartSec=5` (`hermes-webui.container:60-61`) — the
compose-era `restart: unless-stopped` is retired with `dankclaw/compose.yaml`.
- `HERMES_WEBUI_SKIP_ONBOARDING=1` keeps the first-run wizard off (`:45`; live
env confirms it, `podman inspect hermes-webui` 2026-10-18); a profile
already exists in `/data` from the gateway.
- The healthcheck `start_period` is 600 s because the very first boot spends
- The healthcheck `start_period` is 600 s
(`HealthStartPeriod=600s`, `:57`) because the very first boot spends
minutes pip-installing into a fresh venv.
## Remote access (Hermex on iOS — persistent SSH tunnel)
Only port `22` is forwarded to Dankclaw at the router, so the phone reaches the
Only port `22` is forwarded to Dankclaw at the router *(unverified 2026-10-18:
router configuration is not readable from this host; `sshd -T` confirms sshd
itself listens on port 22)*, so the phone reaches the
webui through an SSH **local forward** — no extra router ports, no third-party
relay (no Tailscale, no Cloudflare). Implemented 2026-08-31. The tunnel is an
*access path* for the phone, **not** a containment boundary: the server binds
`0.0.0.0:8787`, so any host that can route to `192.168.0.143` can reach the API
directly and needs only `HERMES_WEBUI_PASSWORD` to get in (verified 2026-09-02:
an unauthenticated `GET /health` from another LAN machine returned HTTP 200).
an unauthenticated `GET /health` from another LAN machine returned HTTP 200;
re-measured from a different LAN host on 2026-10-18: `/health` → 200, `/` → 302).
See [[Container Network Namespaces]] for why the webui is reachable this way and
the sibling dashboard on `9119` is not.
- Dedicated `tunnel` account on Dankclaw (`/usr/sbin/nologin` shell, password
locked). Authentication is key-only; the account can do nothing except
- Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994,
shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only
*(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
without `sudo`)*; the account can do nothing except
authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel
while the transport stays up for port forwarding.
- sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` scopes
`AllowTcpForwarding yes` to that user only (plus `PermitTTY no`,
`X11Forwarding no`).
- sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31)
scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`,
`X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are
present under `Match User tunnel` (`cat` 2026-10-18).
- Key options on the authorized key pin the blast radius:
`command="/usr/sbin/nologin",no-pty,restrict,port-forwarding,permitopen="localhost:8787"`
— a leaked tunnel key can only forward to `localhost:8787`, never act as a
general SOCKS/SSH proxy.
general SOCKS/SSH proxy *(unverified 2026-10-18: the authorized_keys file
needs `sudo` to read)*.
- On the phone (client side): create a key on-device (private key never
leaves the device), hand the **public** key to the admin who appends it to
`/home/tunnel/.ssh/authorized_keys`, then connect from the SSH client:
@@ -104,10 +122,16 @@ graph LR
./scripts/deploy.sh dankclaw --service hermes-webui
```
`deploy.sh` copies `dankclaw/hermes/quadlet/*.container` into the quadlet unit
directory and restarts `hermes.service`, `hermes-dashboard.service` and
`deploy_hermes` (`scripts/deploy-remote.sh:1034-1071`; any of
`--service hermes|hermes-dashboard|hermes-webui` routes to it,
`scripts/deploy-remote.sh:1594`) copies `chrome-devtools-protocol.container`
plus the three trio units into the quadlet unit directory, starts the CDP daemon
only if inactive (it is never restarted on deploy, so in-flight browser
sessions survive), re-tags the image alias, then restarts
`hermes.service`, `hermes-dashboard.service` and
`hermes-webui.service` in the **user** manager of `dank` (linger must be on for
them to come up at boot), so gateway/dashboard/webui restarts stay isolated from
them to come up at boot — `loginctl show-user dank -p Linger` → `yes`,
2026-10-18), so gateway/dashboard/webui restarts stay isolated from
inference deploys.
Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
`dankclaw/.env`.
@@ -118,21 +142,35 @@ Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
the repo root. Do not flip it to `:ro`.
- The webui binds `0.0.0.0:8787`, and **two consumers depend on that**: the SSH
tunnel (which dials the host's `localhost`) *and* Caddy, which fronts
`hermes.franksdank.cloud` with `reverse_proxy 192.168.0.143:8787` — a host
address rather than a container name, because this unit is host-networked and
there is no port forwarder to bridge the gap (`dankclaw/Caddyfile` records the
same constraint). So the SSH tunnel is an access path for the phone, not a
`hermes.franksdank.cloud` with **`reverse_proxy host.containers.internal:8787`**
(`dankclaw/Caddyfile:186-188`) — netavark's host address (169.254.1.2), not
the host's LAN IP. That distinction is load-bearing: this unit is
host-networked and publishes nothing, so netavark DNATs nothing, and
`192.168.0.143:8787` is REFUSED from Caddy's bridge netns while
`host.containers.internal:8787` answers (measured inside the Caddy container
2026-09-16, recorded at `dankclaw/Caddyfile:178-185` and pinned by the seam
`tests/hermes_webui.bats:52-64`, which fails any Caddyfile that reverts to the
LAN IP). The dashboard's `9119` answers on the LAN IP precisely because it IS
published. So the SSH tunnel is an access path for the phone, not a
security boundary: any LAN host that can route to `192.168.0.143` reaches the
API and needs only `HERMES_WEBUI_PASSWORD`. The webui can run agent commands,
so treat that password as a credential with command-execution blast radius.
The vhost itself carries **no** guard chain — no Anubis challenge, no
Authentik gate (measured 2026-10-18 from another LAN host:
`https://hermes.franksdank.cloud/` → 302 → `/login?next=/` and `/health` → 200,
with no authentik or bot-challenge bytes in the body); it is private only
because no public DNS record exists for the name
(`dankclaw/pihole/addn-hosts:39`, LAN-only).
To make loopback-only binding true again you must set
`HERMES_WEBUI_HOST=127.0.0.1` **and** give Caddy another way in (a
host-gateway route or a bridge-network unit) — that is an exposure change to
decide deliberately, not a silent fix. It was tried on 2026-09-16 as a silent
fix and broke the vhost with a 502 while the whole e2e suite stayed green;
`tests/e2e/specs/17-hermes-vhost.spec.ts` now asserts that vhost reaches its
app through the guard chain, so a dead upstream fails a gate instead of
hiding.
`tests/e2e/specs/17-hermes-vhost.spec.ts` now asserts the vhost reaches its
app (no 5xx, no Caddy error page, no gate page), so a dead upstream fails a
gate instead of hiding. Its file comment still names the
`192.168.0.143:8787` upstream and calls this "the only guarded vhost" — both
are stale against `dankclaw/Caddyfile:186-188`.
- The webui and the gateway share one data root (`/data` is a bind mount of
`/home/dank/Containers/hermes/data`). A config change made on one side takes
effect when the other restarts (same caveat as the dashboard).
serverAppConfigs-docs/Dankclaw/Hermes.md
+98 -30
View File
@@ -12,13 +12,33 @@ timestamp: 2026-09-15
Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
**Host:** Dankclaw — the gateway itself publishes **no port** (it runs in the
host network namespace and dials the Signal daemon at `127.0.0.1:8080`); the
user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
host network namespace; `podman inspect systemd-hermes` → `net=host`, no port
bindings, 2026-10-18). Note the **container name drift**: the unit is
`hermes.service` but the generated start line is
`podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so
`podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no
`ContainerName=` key even though its header comment says "container name:
hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
`hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A
record) — **Docs:** https://hermes-agent.nousresearch.com/docs
record; the name resolves from the Pi-hole hosts file,
`dankclaw/pihole/addn-hosts:39`) — **Docs:**
https://hermes-agent.nousresearch.com/docs
**Signal is not carrying traffic today (measured 2026-10-18).** The gateway
boots and stays up, but its own log says
`Platform 'signal' is explicitly disabled by platforms.signal.enabled: false in
config.yaml, so the credentials found in the environment (SIGNAL_HTTP_URL) will
NOT start its adapter` (`journalctl --user -u hermes` 2026-10-18), and the host
daemon it would dial is not running either: `systemctl --user
is-enabled signal-cli-daemon` → `disabled`, `is-active` → `inactive`, and
nothing listens on `:8080` (`ss -ltn`). Everything below still describes the
designed wiring; the remedy for both halves is in the log line
(`platforms.signal.enabled: true`) plus enabling the daemon unit.
**Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted
at `/data` in all three Hermes containers (gateway, webui, dashboard). Each
at `/data` in all three Hermes containers (gateway, webui, dashboard —
`hermes.container:36`, `hermes-webui.container:46`,
`hermes-dashboard.container:24`). Each
container additionally gets an anonymous volume at `/opt/data`, which is
re-created empty on every container recreation — anything that lives only there
is not durable. The tracked, sanitized configuration baseline is
@@ -28,8 +48,8 @@ is not durable. The tracked, sanitized configuration baseline is
| Item | Value |
|------|-------|
| Image | built from `NousResearch/hermes-agent` (official Dockerfile in repo) |
| Upstream latest | agent v0.21.4 / CalVer `v2026.9.21` (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not `:latest`. |
| Image | `localhost/hermes-agent:latest`, built from `NousResearch/hermes-agent` (official Dockerfile in repo); `Pull=never` in all three units (`hermes.container:28-29`, `hermes-webui.container:35-36`, `hermes-dashboard.container:19-20`), so the tag never moves at runtime. Live image: built 2026-09-24, OCI revision label `749220ef0007f8d87bd1531f1c24b0fe93816385` (`podman image inspect localhost/hermes-agent:latest` 2026-10-18) — that build is what is running. |
| Upstream latest | agent v0.21.4 / CalVer `v2026.9.21` (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not `:latest`. *(unverified 2026-10-18: the local image records only a git revision, and the webui version endpoint answers 401 without a session, so neither number can be re-derived from the host.)* |
| License | MIT |
| Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
| Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
@@ -40,16 +60,35 @@ is not durable. The tracked, sanitized configuration baseline is
- Point Hermes at the local model endpoint or an external provider through its provider config.
- The gateway runs `hermes gateway run --force -v` in the **host** network
namespace (`Network=host` in its Quadlet unit) and reaches the Signal daemon at
`http://127.0.0.1:8080`. That loopback hop works only because the gateway is in
namespace (`Exec=` + `Network=host`, `dankclaw/hermes/quadlet/hermes.container:27,37`) and reaches the Signal daemon at
`http://127.0.0.1:8080` (`SIGNAL_HTTP_URL`, `hermes.container:33`). That loopback hop works only because the gateway is in
the host netns — see [[Container Network Namespaces]]. The daemon itself is a
**host** user unit (`signal-cli-daemon.service`), not a container, and binds
`0.0.0.0:8080`, so the Signal HTTP API is reachable from the whole LAN.
**host** user unit (`~/.config/systemd/user/signal-cli-daemon.service`), not a
container. **Today it is `disabled` + `inactive` and nothing is bound to
`:8080`** (`systemctl --user is-enabled/is-active`, `ss -ltn`, both
2026-10-18), and the gateway's own config gate (`platforms.signal.enabled:
false`) keeps the adapter off even if the daemon came back — see the dated
note at the top of this page. When the daemon does run it binds
`0.0.0.0:8080`, i.e. the Signal HTTP API is reachable from the whole LAN.
- The data volume persists skills, memory, and session history.
- Hermes has two named local providers: `dankclaw-large` reaches
`http://192.168.0.143:7770/v1` and exposes `frank-dragoon`;
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
`frank-medium` (renamed from `frank-small` on 2026-09-15).
- Hermes has two named local providers (the `dankclaw-large` name is confirmed
live from the gateway log — `resolve_provider_client: named custom provider
'dankclaw-large' has no resolvable api_key`, `journalctl --user -u hermes`
2026-10-18; the base URLs below are the repo record, since
`/data/config.yaml` is not readable without `sudo`):
`dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
`frank-dragoon` — verified from the host with `curl
http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len
200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is
a rootless pasta publish and is IPv4-only);
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
2026-10-18, that endpoint serves only `frank-archon`** (`GET
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
this page used to carry for it — `frank-medium`, renamed from `frank-small` on
2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
if the live `config.yaml` still names `frank-medium` the provider is dead
weight and requests will 404 on a nonexistent model. Diff the live config
before trusting either name.
- The live `config.yaml` contains secrets and user state and is not committed;
the tracked example is the sanitized backup of its non-secret settings.
@@ -63,9 +102,9 @@ is not durable. The tracked, sanitized configuration baseline is
## Gotchas
- Hermes is under active development; pin the image tag for reproducibility.
- Hermes is under active development; pin the image tag for reproducibility. The units already pin by absence of pulling (`Pull=never`) and by building `localhost/hermes-agent:latest` locally — a rebuild, not a pull, is what changes the code.
- Model choice drives quality; the local `frank-dragoon` (Dankclaw, 7770) is the intended backend.
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth. The gateway itself publishes no port today; the surfaces that are exposed are the webui (`hermes.` vhost, exempt by construction because the app enforces its own password — see [[Hermes-WebUI]]) and the dashboard (`hermesadmin.` vhost → `192.168.0.143:9119`, app-enforced basic auth — see [[Hermes-Dashboard]]).
## Browser automation (Chrome DevTools Protocol daemon)
@@ -73,11 +112,19 @@ The agent's browser tool is an MCP client (`mcp_tool.py` in the image) that
spawns `npx -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`
per session; the browser itself lives in a dedicated daemon container, the
digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host
network, loopback-only bind on 127.0.0.1:9222). Both consumers — the Signal
gateway and the webui server — run in the host network namespace, so they can
network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows
`127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image
`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`,
`dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal
gateway and the webui server — run in the host network namespace (`podman
inspect` → `net=host` for both, 2026-10-18), so they can
dial the daemon on loopback; nothing on the LAN can reach the CDP endpoint.
`GET http://127.0.0.1:9222/json/version` answers **200** today with
`Browser: Chrome/151.0.7922.109`, `Protocol-Version: 1.3` (2026-10-18).
The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devtools`
(command `npx`, args `-y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`).
(command `npx`, args `-y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`) —
that file is mode `drwx------` uid 109999, so reading it needs `sudo` (a
non-privileged read is `EACCES`, checked 2026-10-18).
The npx package cache lives in the container layer, so a recreation (not just a
restart) pays one first-call npm download; subsequent calls are cached.
@@ -115,12 +162,15 @@ checks miss this class entirely.
## Live config edits
Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
uid 109999; all reads/edits need `sudo`). The repo file
uid 109999 — `ls -ld /home/dank/Containers/hermes/data` → `drwx------ 29
109999 109999` 2026-10-18; a non-privileged read returns `EACCES`, so all
reads/edits need `sudo`). The repo file
`dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
live — always diff the live file first; never treat the example as truth**.
1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
(in-dir `.AGENTTRASH/` also exists for removals).
(in-dir `.AGENTTRASH/` also exists for removals *(unverified 2026-10-18:
inside the `0700` data dir, unreadable without `sudo`)*).
2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
the live file are tool-generated — preserve them.
3. **Sweep stale references** across the whole data dir:
@@ -196,6 +246,12 @@ What actually breaks, and the fix:
effort: none # mandatory, see below
```
Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw
`:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider
name is recorded against — answers only `frank-archon`. The provider/model
pairing above is the shape that was landed in 2026; re-derive it from the live
config before reusing it.
- **Disable thinking for any cap-limited task.** The adapter omits reasoning
only when config says `reasoning.enabled: false`. The
`reasoning_effort: none` *shorthand* produces exactly that and leaves the
@@ -220,10 +276,13 @@ or a chatty non-title answer. Always read the raw content, not the status.
## Webui password rotation ("Hermex")
Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth/login` with JSON
`{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
on match, 401 on miss. No username field. `/health` answers 200
unauthenticated and `/` answers 302 — **never verify auth via those**; probe
unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with
`{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w
%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via
those**; probe
login with a deliberately wrong password (expect 401) and the real one
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
@@ -233,17 +292,26 @@ login with a deliberately wrong password (expect 401) and the real one
`-w %{http_code}` inline.
2. Update BOTH env files with
`sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
- `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
quadlet's `[Container] EnvironmentFile=` is what the container reads at
create; rotating only the root file leaves the running container serving
the old secret.
- `/home/dank/serverAppConfigs/.env` — the repo-root checkout's env file
(exists, `-rw------- dank dank`, 2026-10-18) but **not** loaded by this
unit: keep it in sync only so other consumers do not drift.
- `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one, for
both `[Container]` and `[Service]`**: `systemctl --user show hermes-webui
-p EnvironmentFiles` reports exactly
`/home/dank/serverAppConfigs/dankclaw/.env` (2026-10-18;
`dankclaw/hermes/quadlet/hermes-webui.container:38,62`). That is what the
container reads at create; rotating only the root file leaves the running
container serving the old secret.
3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
env; no deploy needed if the unit file itself didn't change).
4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
client at `http://192.168.0.143:8787` (webui publishes on the LAN
interface since commit `4443897`; off-network ingress is SSH tunnel only —
user `tunnel`, port **22**, key-only auth, forward remote end
interface since commit `4443897`, 2026-08-31 "publish webui on LAN interface
instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18;
off-network ingress is SSH tunnel only —
user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth
*(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*,
forward remote end
localhost:8787).
Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
serverAppConfigs-docs/Dankclaw/Marreta.md
+69 -29
View File
@@ -17,8 +17,10 @@ timestamp: 2026-09-12
Marreta is a self-hosted **content-cleaning proxy**: you hand it a URL and it
returns a stripped, readable version of the page — no paywall chrome, no
tracking, no cookie walls. It is a Laravel 13 app on the
`serversideup/php:8.4-fpm-nginx` image, with an optional headless-browser
companion for JavaScript-heavy pages.
`serversideup/php:8.4-fpm-nginx` image (`composer.json` requires
`php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` —
<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>),
with an optional headless-browser companion for JavaScript-heavy pages.
**Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained)
— **Upstream:** host-published port `8082` (inside the Gluetun netns)
@@ -28,11 +30,16 @@ companion for JavaScript-heavy pages.
- Fetches a target URL with a cascade of strategies: plain HTTP (curl), a
headless browser (Lightpanda) for JS pages, and the Internet Archive
(Wayback) as a last resort.
(Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php`
files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`,
`WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`).
- Strips paywall/consent/tracking markup and re-renders clean HTML.
- Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` /
`ADMIN_PASSWORD`) for settings and usage.
- Health endpoint at `/up` (Laravel's built-in health route).
`ADMIN_PASSWORD`) for settings and usage — live 2026-10-18
`curl http://127.0.0.1:8082/admin/login` → **200**.
- Health endpoint at `/up` (Laravel's built-in health route —
`bootstrap/app.php` passes `health: '/up'`). Live: `curl
http://127.0.0.1:8082/up` → **200** (2026-10-18).
## Topology
@@ -58,11 +65,17 @@ graph LR
| Unit | File | Image | Netns | Listens |
|------|------|-------|-------|---------|
| `marreta.service` | `dankclaw/marreta/quadlet/marreta.container` | `ghcr.io/manualdousuario/marreta:latest` | `gluetun.container` | `8085` (nginx) |
| `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` | `gluetun.container` | `9222` (CDP) |
| `marreta.service` | `dankclaw/marreta/quadlet/marreta.container` | `ghcr.io/manualdousuario/marreta:latest` (`:35`) | `gluetun.container` (`:34`) | `8085` (nginx, `NGINX_HTTP_PORT=8085`, `:37`) |
| `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) |
Both units are boot-persistent (`WantedBy=default.target`) and self-healing
(`Restart=always`), per the repo's Boot Persistence rule.
(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`,
`marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live
2026-10-18 both report `systemctl --user is-enabled` → `generated` and
`is-active` → `active`, and `podman inspect` reports marreta **healthy**
(`HealthCmd` GETs `http://127.0.0.1:8085/up`, `marreta.container:53`) and
marreta_browser `running`. Both images are `:latest`, so a redeploy can move
them.
## The port-collision resolution (why 8085)
@@ -81,8 +94,11 @@ The fix is a single env var, verified against the serversideup source:
port off the collision with no custom build.
Marreta then publishes **nothing** of its own — the netns owner (Gluetun)
publishes `8082:8085` on its behalf, exactly as it publishes `8081:8080`
for SearXNG. Caddy dials `192.168.0.143:8082`.
publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`),
exactly as it publishes `8081:8080` for SearXNG. Caddy dials
`192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the
relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085`
(2026-10-18).
## Why netns sharing, not `PROXY_URL`
@@ -93,7 +109,13 @@ browser ran outside the tunnel it would leak the host's real egress IP on
every JS-rendered page. Netns sharing is the only way to put **both** the
app and the browser on the same exit, which is what "share the same Gluetun
egress as SearXNG" requires. The browser is therefore reached on loopback
(`ws://127.0.0.1:9222`) inside the shared netns.
(`ws://127.0.0.1:9222`) inside the shared netns — upstream's own compose uses
`ws://marreta_browser:9222`, which needs no service name here because both
containers share one namespace. Live env: `BROWSER_WS_ENDPOINT=ws://127.0.0.1:9222`
(`podman inspect marreta` 2026-10-18; `marreta.container:44`). Upstream agrees
on the mechanism: `BrowserFetchStrategy.php` only calls
`BrowserFactory::connectToBrowser($wsEndpoint)` and never passes a proxy, while
`CurlFetchStrategy.php:24-25` is the only place `marreta.proxy_url` is read.
## Guard chain (behind Authentik)
@@ -101,16 +123,19 @@ egress as SearXNG" requires. The browser is therefore reached on loopback
[[Anubis]] and [[Authentik]]:
1. **Public edge** — the vhost proxies everything to `anubis:8923`
(bot challenge), with `header_up X-Real-Ip {remote_host}`.
(bot challenge), with `header_up X-Real-Ip {remote_host}`
(`dankclaw/Caddyfile:159-163`).
2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies
whole-request to `authentik-server:9000`.
whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`).
3. **Post-auth routing** — authenticated traffic returns to `http://:8017`,
where `@marreta host marreta.franksdank.cloud` routes to
`reverse_proxy 192.168.0.143:8082`.
`reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`).
**Access model: any authenticated user.** The post-auth handler carries no
group gate (unlike the homepage `/admin`, which requires the
`homepage-admins` group). Anyone who can log in via Authentik can use
`homepage-admins` group) — the Caddyfile says so in the block header
("Any authentik-authenticated user may use it (no group gate)",
`dankclaw/Caddyfile:156-157`). Anyone who can log in via Authentik can use
Marreta.
**Required provider (manual, one-time).** Before the vhost goes live, a
@@ -136,8 +161,12 @@ must change together.
## Scheme / proxy trust
Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0.1')`
and forces `https` when `APP_URL` starts with `https://`. With
`APP_URL=https://marreta.franksdank.cloud` the app honors the forwarded
— that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and
this container sets no `TRUSTED_PROXIES`, so the default is what runs
(`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL`
starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`,
`URL::forceScheme('https')`). With
`APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded
scheme and client IP through the whole chain, so there is no redirect loop
or mixed-content breakage.
@@ -145,19 +174,24 @@ or mixed-content breakage.
Three values, seeded into `/home/dank/Containers/marreta/env` (mode 0600,
dank-owned so the rootless unit can read it — the grafana convention, not
`/opt`):
`/opt`; live 2026-10-18 `-rw------- dank dank` 155 B). The seeder writes it
only when the file does not exist and `chmod 600`s it
(`seed_marreta_env`, `scripts/deploy-remote.sh:615-622`).
| Repo `.env` key | Mapped to (app) | Notes |
|-----------------|-----------------|-------|
| `MARRETA_APP_KEY` | `APP_KEY` | Laravel encryption key. **Must be the `base64:`-prefixed form**: `base64:$(openssl rand -base64 32)`. A bare base64 string (no prefix) is read as a raw key of the wrong length and every app route 500s with `Unsupported cipher or incorrect key length` at `Encrypter.php:61` (only `/up`, which skips the encrypter, still passes). Changing it after first boot invalidates encrypted session/cookie data. |
| `MARRETA_APP_KEY` | `APP_KEY` | Laravel encryption key. **Must be the `base64:`-prefixed form**: `base64:$(openssl rand -base64 32)`. A bare base64 string (no prefix) is read as a raw key of the wrong length and every app route 500s with `Unsupported cipher or incorrect key length` at `Encrypter.php:61` (laravel/framework 13.x, `src/Illuminate/Encryption/Encrypter.php:61`) while `/up`, which skips the encrypter, still passes. Today's key is well-formed: `/`, `/admin/login` and `/up` all answer 200 (`curl http://127.0.0.1:8082/…` 2026-10-18). Changing it after first boot invalidates encrypted session/cookie data. |
| `MARRETA_ADMIN_EMAIL` | `ADMIN_EMAIL` | Admin panel login (created on first boot). |
| `MARRETA_ADMIN_PASSWORD` | `ADMIN_PASSWORD` | Admin panel password. |
The quadlet maps them with `Environment=APP_KEY=${MARRETA_APP_KEY}` etc.
The `.container` files hold **no** secrets — only `${VAR}` placeholders.
(`dankclaw/marreta/quadlet/marreta.container:47-49`), loaded from the env file
via `[Service] EnvironmentFile=` (`:60`). The `.container` files hold **no**
secrets — only `${VAR}` placeholders.
The SQLite DB + cache live in the `marreta_storage` named volume at
`/var/www/html/storage/app` (podman auto-creates it and copyups the image's
`www-data` ownership so the app can write).
`/var/www/html/storage/app` (`marreta.container:52`; podman auto-creates it and
copyups the image's `www-data` ownership so the app can write). Live 2026-10-18
the volume exists at `/home/dank/.local/share/containers/storage/volumes/marreta_storage/_data`.
## Deploy
@@ -165,11 +199,11 @@ The SQLite DB + cache live in the `marreta_storage` named volume at
./scripts/deploy.sh dankclaw --service marreta
```
`deploy_marreta` seeds the env file (only if absent), installs both
`deploy_marreta` (`scripts/deploy-remote.sh:630-640`) seeds the env file (only if absent), installs both
`.container` units, ensures Gluetun is up, then restarts the browser before
the app. Because Marreta shares Gluetun's netns, the planner treats it as
part of the Gluetun netns family: a change to `gluetun.container` or
`searxng.network` also selects `marreta` (a Gluetun recreation tears the
`searxng.network` also selects `marreta` (`scripts/deploy-plan.py:295-309` — a Gluetun recreation tears the
netns down and would otherwise orphan it).
> **First deploy note:** adding `PublishPort=8082:8085` to `gluetun.container`
@@ -180,14 +214,20 @@ netns down and would otherwise orphan it).
- **Unauthenticated:** `curl -s https://marreta.franksdank.cloud/` → the
Anubis challenge page (grep body for `challenge`/`not a bot`). A 502 here
means the chain is mis-wired.
means the chain is mis-wired. Re-checked 2026-10-18: **200** with `challenge`
×5 and `not a bot` ×2 in the body — the guard chain is live.
- **Post-auth routing:** `podman exec caddy sh -c "grep -n -A3 '@marreta host' /etc/caddy/Caddyfile"`
after the deploy's reload line.
after the deploy's reload line (repo copy: `dankclaw/Caddyfile:419-422`).
- **Egress:** from inside the netns, `podman exec marreta curl -s https://ifconfig.co`
must return the VPN exit IP, not the home IP.
must return the VPN exit IP, not the home IP. The tunnel's own egress is
readable without exec: `journalctl --user -u gluetun` logged exit
`149.22.88.19` (United States, Washington, Seattle) on 2026-10-18.
- **Port:** `podman exec marreta sh -c 'curl -fsS localhost:8085/up'` → 200;
`curl http://192.168.0.143:8082/up` → 200.
- **Test seam:** `bats tests/marreta_quadlet.bats` (17 contracts).
`curl http://192.168.0.143:8082/up` → 200 (also **200** from the host
loopback, `curl http://127.0.0.1:8082/up` 2026-10-18; the container health
probe GETs `http://127.0.0.1:8085/up` and reports **healthy**).
- **Test seam:** `bats tests/marreta_quadlet.bats` (17 contracts — 17 `@test`
blocks, counted 2026-10-18).
### Live verification (deployed 2026-09-12, rev `91e9410`)
serverAppConfigs-docs/Dankclaw/Postgres.md
+73 -14
View File
@@ -9,20 +9,35 @@ timestamp: 2026-09-03
# Postgres
PostgreSQL is the database backend for Gitea. It runs on the same podman network as Gitea and holds the repository metadata, users, and issue data.
PostgreSQL is the database backend for two separate services on this host:
**Gitea** (`gitea-postgres`) and **Authentik** (`authentik-postgres`). They are two
independent clusters — two containers, two data dirs, two podman networks — and each
holds only its own app's data; neither is a shared instance. Gitea's DB holds the
repository metadata, users, and issue data; Authentik's holds the IdP objects.
**Host:** Dankclaw — **Docs:** https://www.postgresql.org/docs
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/library/postgres:16-alpine` |
| Port | 5432 (internal only) |
| Data volume | `/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data` (live bind, verified from the unit) |
| Item | `gitea-postgres` | `authentik-postgres` |
|------|------------------|----------------------|
| Image | `docker.io/library/postgres:16-alpine` (`dankclaw/gitea/quadlet/gitea-postgres.container:18`) | `docker.io/library/postgres:16-alpine` (`dankclaw/authentik/quadlet/authentik-postgres.container:17`) |
| Port | 5432, container-internal — no `PublishPort`; a TCP probe of `192.168.0.143:5432` is refused (`ss -ltn` + probe 2026-10-18) | same: nothing published, 5432 refused on the LAN IP (2026-10-18) |
| Network | `systemd-gitea`, 10.89.0.0/24 (`dankclaw/gitea/quadlet/gitea.network`, `podman network inspect systemd-gitea` 2026-10-18) | `dankclaw_default`, 10.89.1.0/24 (`authentik-postgres.container:18`; the compose-stack bridge shared with Authentik/Seerr/Grafana) |
| Data volume | `/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data` (`gitea-postgres.container:23`) | `/home/dank/Containers/authentik/postgres:/var/lib/postgresql/data` (`authentik-postgres.container:23`) |
| Healthcheck | **none** — the unit carries no `HealthCmd`, so `podman ps` reports `Up` with no health state (2026-10-18) | `pg_isready -d authentik -U authentik` every 30 s, 5 retries, 20 s start (`authentik-postgres.container:29-33`); container reports `healthy` (2026-10-18) |
| Secret source | `/home/dank/Containers/gitea/env` (mode 0600, `ls -l` 2026-10-18) | `/home/dank/serverAppConfigs/dankclaw/.env` (`authentik-postgres.container:19`) |
`shared/homepage/services.json` reflects both: its `Postgres` card reads "Authentik's
and Gitea's databases (Dankclaw). Not probeable: 5432 is container-internal."
## Configuration
Retired compose-era definition (kept for the history below — the live stack is
Quadlet, and this service ran from `compose.yaml` before the 2026 boot-persistence
cutover). Note the data path in it (`/opt/gitea/data/gitea-postgres`) no longer
exists; nothing deploys this block today.
```yaml
services:
postgres:
@@ -38,7 +53,9 @@ services:
## Gitea Connection
Gitea reaches the database over the `systemd-gitea` network (quadlet-managed live, compose-managed after the full-stack deploy):
Gitea reaches its database over the `systemd-gitea` network. The live quadlet unit
`dankclaw/gitea/quadlet/gitea-server.container:24-29` sets exactly these variables
(`Environment=DATABASE_*`), so the compose-era block below is equivalent in keys:
```yaml
environment:
@@ -49,19 +66,49 @@ environment:
DATABASE_PASSWD: "${GITEA_DB_PASSWORD}"
```
The live `app.ini` on the host overrides env vars. When you rename the database container, update its `[database] HOST` line. See [[Gitea]] Gotchas.
The live `app.ini` on the host overrides env vars: its `[database]` stanza carries
`HOST = gitea-postgres:5432`, `NAME = gitea`, `USER = gitea`
(`/home/dank/Containers/gitea/data/gitea/conf/app.ini:27-33`, 2026-10-18). When you
rename the database container, update that `[database] HOST` line. See [[Gitea]]
Gotchas.
## Layout (current)
The live `gitea-postgres` unit mounts its data at
`/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data` — the old
Both live units mount their data under `/home/dank/Containers`:
- `gitea-postgres` → `/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data`
- `authentik-postgres` → `/home/dank/Containers/authentik/postgres:/var/lib/postgresql/data`
Both directories exist on the host with mode 0700 and owner uid 100069 (the mapped
`postgres` uid of the rootless namespace) — `ls -ld` 2026-10-18. The old
`/opt/gitea/data/gitea-postgres` compose-era path is gone. Secrets come from
the host-local env file `/home/dank/Containers/gitea/env` (0600) via
`EnvironmentFile=`; `POSTGRES_PASSWORD` never enters this repo. See [[Gitea]].
host-local env files (`/home/dank/Containers/gitea/env`, 0600;
`/home/dank/serverAppConfigs/dankclaw/.env`) via `EnvironmentFile=`; the passwords
never enter this repo. See [[Gitea]] and [[Authentik]].
## Boot persistence (Quadlet)
Both units carry `Restart=always` (`gitea-postgres.container:28`,
`authentik-postgres.container:38`) and `[Install] WantedBy=default.target`
(`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed
under `~/.config/containers/systemd/`, and report `generated` / `active` with
`Linger=yes` on the host (2026-10-18), so both come back at boot.
## Healthcheck
Use `pg_isready`:
Only `authentik-postgres` has one. Its quadlet runs bare argv, not a shell string,
because the generator space-splits `HealthCmd` (the quoted `/bin/sh -c "…"` form
arrived mangled and reported a false-negative `unhealthy` forever — see the comment
in `dankclaw/authentik/quadlet/authentik-postgres.container:24-33`):
```ini
HealthCmd=pg_isready -d authentik -U authentik
HealthInterval=30s
```
`gitea-postgres` has **no** `HealthCmd`; the probe below is the compose-era snippet,
kept as the recipe to add if Gitea's DB ever needs one. Until it is in the unit,
judge `gitea-postgres` by `podman ps` state and Gitea's own health.
```yaml
healthcheck:
@@ -73,7 +120,12 @@ healthcheck:
## Gotchas
- The database password must match between the `postgres` service and Gitea's `DATABASE_PASSWD`; both come from the same `.env` variable.
- Two clusters, not one: `gitea-postgres` serves only Gitea and
`authentik-postgres` only Authentik. They are on different bridges
(`systemd-gitea` 10.89.0.0/24 vs `dankclaw_default` 10.89.1.0/24, `podman network
inspect` 2026-10-18), so neither can reach the other's data by name. Any
"one Postgres serves everything" reading of this page is wrong.
- The database password must match between the `postgres` container and Gitea's `DATABASE_PASSWD`; both come from the same env-file variable (`GITEA_DB_PASSWORD`, expanded by systemd from `/home/dank/Containers/gitea/env`).
- Do not change the Gitea `SECRET_KEY` or `INTERNAL_TOKEN` after migration — encrypted data becomes unreadable.
- Back up with `pg_dump`; see [[Gitea]] (knowledge base).
@@ -81,3 +133,10 @@ healthcheck:
- [[Gitea]]
- [[Dankclaw Stack]]
## Citations
- `dankclaw/gitea/quadlet/gitea-postgres.container`, `dankclaw/gitea/quadlet/gitea.network`, `dankclaw/gitea/quadlet/gitea-server.container`
- `dankclaw/authentik/quadlet/authentik-postgres.container`
- `shared/homepage/services.json` (the `Postgres` card)
- Host evidence 2026-10-18: `podman ps`, `podman inspect`, `podman network inspect systemd-gitea dankclaw_default`, `ss -ltn`, TCP probe of `192.168.0.143:5432`, `systemctl --user is-enabled/is-active gitea-postgres authentik-postgres`
serverAppConfigs-docs/Dankclaw/SearXNG Providers.md
+84 -36
View File
@@ -14,12 +14,15 @@ VPN tunnel, and how to unblock them in SearXNG.
**Host:** Dankclaw — **Docs:** https://docs.searxng.org
SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
SearXNG aggregates the engines bundled in the running build — **262 installed, 75
enabled** on the live instance (`GET http://127.0.0.1:8081/config`, 2026-10-18; the
"~50 engines" framing this page carried predates that count). When all of them are
queried through **one** VPN exit IP — as on
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
engine for an hour to 15 days and silently drops it from results. That is the
"Google (and sometimes a few others) went unavailable" symptom.
engine (three minutes for a 429 under the current build's defaults, up to 15 days for
a Cloudflare CAPTCHA — see the table below) and silently drops it from results. That
is the "Google (and sometimes a few others) went unavailable" symptom.
This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
are the detail if you need it.
@@ -35,20 +38,32 @@ should disappear completely.
|----------|------------------|----------------------|
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
| Science | arXiv, PubMed, Semantic Scholar | Google Scholar until independently tested; OpenAlex (`openalex` is `enabled: false` in the running build, `/config` 2026-10-18) |
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
| Images | Wikimedia Commons (ships as `wikicommons.images/videos/audio/files`, all enabled), Bing Images, Unsplash | Openverse (`enabled: false` today), Google/Brave/Startpage image engines |
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
This policy uses no API credentials and does not identify the operator to a
search provider. It is deliberately conservative: a bundled engine is not
**Live state of this table (2026-10-18, from `GET /config`):** the General/web,
Technical, News and Videos baselines hold engine-for-engine; the two rows above are
the corrections. Nothing in the deferred column is enabled. One operational note for
the deferred engines: re-enabling Google CSE on demand needs a key this host does not
hold — `/opt/searxng/.env` carries only the WireGuard variables plus
`SEARXNG_SECRET`, `SEARXNG_BASE_URL` and `KAGI_API_KEY` (key **names** read
2026-10-18; values not printed).
This policy was written to use no API credentials and to avoid identifying the
operator to a search provider — **with one carve-out made afterwards (2026-09-26):
the paid `kagi` engine, whose key is rendered into the deployed config from
`/opt/searxng/.env` (`shared/searxng/settings.yml:70`; item 4 below and
[[SearXNG]]).** It is deliberately conservative: a bundled engine is not
necessarily healthy, anonymous, or suitable for every default query.
## Root cause (one line)
All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
rate-limit it → CAPTCHA → engine **suspension** → empty results.
Google/DuckDuckGo see one datacenter IP spamming every enabled engine (75 of the 262
installed ones, `/config` 2026-10-18) and rate-limit it → CAPTCHA → engine
**suspension** → empty results.
## The suspension model (what "unavailable" actually is)
@@ -58,15 +73,23 @@ Suspended engines are the usual cause of "gone":
| Exception | Default suspension | Trigger |
|-----------|--------------------|---------|
| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
| `SearxEngineTooManyRequests` | **180 s (3 min)** | HTTP 429 |
| `SearxEngineAccessDenied` | **180 s (3 min)** | 403 / access denied |
| `SearxEngineCaptcha` | **3600 s (1 h)** | Google `/sorry/` or reCAPTCHA |
| `recaptcha_SearxEngineCaptcha` | ~7 days (604 800 s) | Google reCAPTCHA |
| `cf_SearxEngineCaptcha` | ~15 days (1 296 000 s) | Cloudflare edge CAPTCHA |
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail` 5 / `max_ban_time_on_fail` 120 |
The values are `search.suspended_times` (+ `ban_time_on_fail`) in
`settings.yml`. An engine that returns **0 results without an error** is *not*
These are the defaults of the **running commit** `12f8b6515`, read from
[`searx/settings.yml` at that commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml)
(lines 66-81). The folklore numbers this table used to carry — ~1 hour for 429, ~1 day
for access-denied and CAPTCHA — were the **older** upstream defaults and are retired;
the current build un-suspends a 429'd engine after three minutes.
The values are `search.suspended_times` (+ `ban_time_on_fail`). Our
`shared/searxng/settings.yml` overrides **none** of them (`grep -n "suspended_times\|ban_time_on_fail"`
returns nothing, 2026-10-18), so the upstream defaults quoted above are exactly what
runs. An engine that returns **0 results without an error** is *not*
suspended — that is the separate "selector drift" problem (below).
## Two failure modes
@@ -81,9 +104,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
## Fix A — Engine config (settings.yml `engines:`)
The current tracked policy enables Bing and DuckDuckGo for default general
search, gives each weight 2, disables pagination, and allows 5 seconds for the
VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
but disabled from default aggregation. This limits fan-out while preserving
search, gives each weight 2, disables pagination, and allows 5.0 seconds for the
VPN path (`shared/searxng/settings.yml:26,38-56`). CAPTCHA/429-prone Google, Brave,
and Startpage families are retained but disabled from default aggregation
(`:81-124`) — though the three `startpage*` stanzas are dead weight: the running
build ships no `startpage` engine at all (`/config` lists none, 2026-10-18), so those
overrides match nothing. This limits fan-out while preserving
explicit/on-demand selection.
Important: SearXNG's engines are configured in the **`engines:` list inside
@@ -105,7 +131,7 @@ engines:
- name: google
weight: 3
disabled: false
timeout: 5.0 # wait longer over the tunnel (global default is 2.0s)
timeout: 5.0 # wait longer over the tunnel (upstream global default is 3.0s — searx/settings.yml:178 at the running commit)
retry_on_http_error: [403, 429]
display_error_messages: true
@@ -172,8 +198,8 @@ Tune the `outgoing` block and the suspend model to survive load:
```yaml
outgoing:
request_timeout: 4.0 # current Dankclaw value; modest VPN-latency allowance
max_request_timeout: 12.0
request_timeout: 5.0 # Dankclaw's value (shared/searxng/settings.yml:26)
max_request_timeout: 15.0 # Dankclaw's cap (shared/searxng/settings.yml:27)
pool_connections: 100
pool_maxsize: 20
enable_http2: true
@@ -194,9 +220,9 @@ Notes:
that is a historical-searx concept. The functional equivalent is the suspend
schedule above (widen to back off before blocking, narrow to recover faster).
- `request_timeout` interacts with the suspend model: too tight → engines look
"unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
12 seconds as the cap; raise it only when measurements show real upstream
latency rather than 429/CAPTCHA responses.
"unresponsive"; too loose → slow pages. Dankclaw uses 5 seconds globally and
15 seconds as the cap (`shared/searxng/settings.yml:26-27`); raise it only when
measurements show real upstream latency rather than 429/CAPTCHA responses.
- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
provider's rate limit worse. A retry count of 1 is a bounded compromise for a
transient network failure.
@@ -281,7 +307,14 @@ The read-only audit found:
Google's "unusual traffic from your network" response.
- `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
healthy. This isolates the primary failures to upstream provider handling,
not application availability.
not application availability. Re-checked 2026-10-18: `/healthz` is still `200`
and `gluetun`/`searxng`/`searxng-valkey` are all `generated` + `active`.
What has moved since that audit: the container is still
`ghcr.io/searxng/searxng:latest` (`podman inspect searxng` 2026-10-18), but the
embedded version is now `2026.9.25+12f8b6515` (`GET /config`), not `1.0.0`, and
`limiter.enabled` is `true`. The exit IP was not re-measured (that would need a
query fired through the tunnel).
This deployment returns the legacy array form for `unresponsive_engines`:
@@ -293,10 +326,15 @@ This deployment returns the legacy array form for `unresponsive_engines`:
```
Do not assume the newer object form (`engine`, `error_type`, `suspended`)
until the live response has been checked. The direct published port is the
useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
hostname resolved to the WAN address during the audit and timed out from the
audit host, so it was not used as the baseline path.
until the live response has been checked — and note that the jq recipes below key on
`.engine`, which only matches the object form; against the legacy `[engine, message]`
pair they silently match nothing. *(Re-checked 2026-10-18 with a `bing`-only JSON
query: `unresponsive_engines` came back empty, so today's shape is unverified.)* The
direct published port is the useful diagnostic path because it exercises the
ProtonVPN exit. The Caddy hostname resolved to the WAN address during the audit and
timed out from the audit host, so it was not used as the baseline path; on
2026-10-18 `search.franksdank.cloud` resolves to the LAN IP from inside the network
and answers with the Anubis challenge page.
## Fix D — Verify an engine is unblocked
@@ -304,7 +342,9 @@ Three layers, cheapest first. All queries should hit the path production uses
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).
**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
(otherwise `format=json` → 403), then query a single engine:
(otherwise `format=json` → 403) — `html` and `json` are both enabled here
(`shared/searxng/settings.yml:18-20`), and a JSON query through the published port
returns 200 `application/json` today (2026-10-18). Query a single engine:
```bash
BASE="http://192.168.0.143:8081" # published port (ProtonVPN exit)
@@ -335,7 +375,10 @@ done
```
**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. `?engines=<name>` isolates a single
categories — an inventory of what is live. It answers 200 on this instance today
(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`,
`categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
[[SearXNG]]). `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.
**3. `check.searx.space`** — the hosted official checker probes every engine of
@@ -369,14 +412,17 @@ misleading).
engine (Kagi) is acceptable when the blast radius is bounded — the
`kagi` engine runs with `paging: false` and weight 2, so it spends exactly
one API credit per aggregated query, and its key never enters git
(deploy-time rendering from `/opt/searxng/.env`; see [[SearXNG]]). It is
(deploy-time rendering from `/opt/searxng/.env` into
`/opt/searxng/config/settings.yml`, where the `${KAGI_API_KEY}` placeholder is
already replaced — checked 2026-10-18; see [[SearXNG]]). It is
also immune to the shared-VPN-exit-IP blocking this page is about: it
authenticates, so reputation of the exit IP is irrelevant.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
exactly which engines are down and why. Confirm the running SearXNG version
(check whether `parsers`/`rate_limit` exist in the pinned build).
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
upstream request, caps it at 15 seconds, and retries once. This addresses
upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
(`:27`), and retries once (`:32`). This addresses
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
after measuring the resulting error and latency profile.
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
@@ -407,6 +453,8 @@ misleading).
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
- [Upstream `searx/settings.yml` at the running commit `12f8b6515`](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml) — the `suspended_times`, `ban_time_on_fail` and `request_timeout` defaults quoted above.
- [Upstream `searx/engines/kagi.py` at the same commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/engines/kagi.py) — Bearer auth, `POST https://kagi.com/api/v1/search`, `require_api_key: True`.
## See Also
serverAppConfigs-docs/Dankclaw/SearXNG.md
+121 -48
View File
@@ -17,11 +17,11 @@ SearXNG is a privacy-respecting metasearch engine. It aggregates results from ma
| Item | Value |
|------|-------|
| Image | `ghcr.io/searxng/searxng:latest` (or DockerHub mirror) |
| Port | 8081 (host, on Gluetun) → 8080 (container) |
| Config | `/etc/searxng/settings.yml` (bind mount) |
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine` |
| Formats | `html`, `json` (agent API) |
| Image | `ghcr.io/searxng/searxng:latest` (`dankclaw/searxng/quadlet/searxng.container:19`; the live container's image reference is exactly that — `podman inspect searxng` 2026-10-18). Running build: `2026.9.25+12f8b6515` (`GET /config`) |
| Port | SearXNG listens on **8080 inside Gluetun's network namespace**; the host sees **8081**, published by the gluetun unit (`dankclaw/searxng/quadlet/gluetun.container:31`; `ss -ltn` shows `*:8081`, `curl http://127.0.0.1:8081/` → 200, 2026-10-18) |
| Config | `/etc/searxng/settings.yml` (bind mount of `/opt/searxng/config`, `searxng.container:22`) |
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine`, container `searxng-valkey`, publishes nothing (see [[Valkey]]) |
| Formats | `html`, `json` (agent API) — `shared/searxng/settings.yml:18-20`; `curl "http://127.0.0.1:8081/search?q=…&format=json"` → 200 `application/json` (2026-10-18) |
## VPN Routing (critical)
@@ -37,15 +37,15 @@ Port publishing happens on the **gluetun** unit, not on searxng. See [[Gluetun]]
## Boot Persistence (Quadlet)
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `searxng/quadlet/` and install to `~/.config/containers/systemd/`.
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `dankclaw/searxng/quadlet/` and install to `~/.config/containers/systemd/`.
The units live in `dankclaw/searxng/quadlet/` and install to
`~/.config/containers/systemd/` on Dankclaw (verified live):
| Unit file | Service | Role |
|-----------|---------|------|
| `searxng.network` | generated network service | DNS-enabled bridge; the created network is named `systemd-searxng`, subnet pinned to 10.89.2.0/24 |
| `gluetun.container` | `gluetun.service` | WireGuard tunnel (env-file secrets); publishes `8081:8080` — the only published port in the trio |
| `searxng.network` | generated network service | DNS-enabled bridge (`dns_enabled: true`, `podman network inspect systemd-searxng` 2026-10-18); the created network is named `systemd-searxng`, subnet pinned to 10.89.2.0/24 (`searxng.network:16`) |
| `gluetun.container` | `gluetun.service` | WireGuard tunnel (env-file secrets); publishes `8081:8080` for SearXNG **and** `8082:8085` for [[Marreta]] (`gluetun.container:31-32`) — SearXNG's is the only port in the search trio, but the tunnel unit carries two, and both are live (`ss -ltn` 2026-10-18) |
| `searxng-valkey.container` | `searxng-valkey.service` | Valkey cache on the searxng network, alias `valkey` (see [[Valkey]]) |
| `searxng.container` | `searxng.service` | `Network=gluetun.container` — joins the tunnel's netns, so it publishes no port of its own; reached through gluetun's 8081 |
@@ -53,7 +53,7 @@ gluetun and valkey join the searxng network. searxng reuses gluetun's netns
and reaches valkey by name on that network. Each unit sets
`WantedBy=default.target`. User linger is on, so the units start at boot.
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, daemon-reloads, enables them, and starts them in dependency order.
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw` — a wrapper that execs `scripts/deploy.sh dankclaw --service searxng` (`scripts/deploy-searxng-quadlet.sh:11`). The dispatcher copies the config, installs the three units plus the network, daemon-reloads, and starts them in dependency order: network → gluetun → valkey → searxng (`scripts/deploy-remote.sh:583-593`).
## settings.yml Keys
@@ -108,51 +108,66 @@ engines:
disabled: true
```
The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas);
the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked
`secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is
censused below.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/etc/searxng` | `/opt/searxng/config` |
| `/etc/searxng/valkey` | (valkey container data) |
| Container path | Host path | Proof |
|----------------|-----------|-------|
| `/etc/searxng` (searxng) | `/opt/searxng/config` | `searxng.container:22`; live dir holds `settings.yml` + `limiter.toml` (`ls -la /opt/searxng/config` 2026-10-18) |
| `/data` (valkey container) | `/opt/searxng/valkey` | `dankclaw/searxng/quadlet/searxng-valkey.container:20` — this is a mount of the **valkey** container, not of `/etc/searxng/valkey` |
## Healthcheck
SearXNG responds on `/healthz` with 200 when ready.
SearXNG responds on `/healthz` with 200 when ready — re-checked on 2026-10-18:
`curl -s -o /dev/null -w '%{http_code} %{content_type}' http://127.0.0.1:8081/healthz`
→ `200 text/plain`. The unit itself carries no `HealthCmd`, so `podman ps` shows
`searxng` as plain `Up` (2026-10-18); the homepage card probes `127.0.0.1:8081`.
## Trusted Group (limiter.toml)
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on. The file deploys to `/opt/searxng/config/limiter.toml`, mounted at `/etc/searxng/limiter.toml`.
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on (`server.limiter: true`, `shared/searxng/settings.yml:11` — and the running instance confirms it: `GET /config` reports `limiter.enabled: true`, 2026-10-18). The file deploys to `/opt/searxng/config/limiter.toml` (`scripts/deploy-remote.sh:211`), mounted at `/etc/searxng/limiter.toml`. The deployed copy is byte-identical to the repo file today (`diff` clean, 2026-10-18).
Two lists control access:
- `pass_ip` — an IP in this list gets unrestricted access and bypasses all bot detection.
- `trusted_proxies` — if the request IP is in this list, SearXNG reads the real client IP from `X-Forwarded-For` (set by Caddy).
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`). An IP in `pass_ip` skips every bot-detection check, including the rate limiter. `deploy.sh` copies the file into place on each deploy.
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`) — `shared/searxng/limiter.toml:54-60`. An IP in `pass_ip` skips every bot-detection check, including the rate limiter: ip_lists "outranks every other method" (`limiter.toml:41-45`). `deploy.sh` copies the file into place on each deploy (`scripts/deploy-remote.sh:208-211`).
Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client IP. Only Caddy's `X-Forwarded-For` restores it. So a local subagent that hits the port directly must present a `pass_ip` source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.
The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The constant is hardcoded in Python (`searx/botdetection/ip_limit.py`). It cannot be raised from `limiter.toml`. `pass_ip` is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.
Reach SearXNG through Caddy (`search.franksdank.cloud`, guarded) for browser
use. A local subagent can hit the published port directly because its
use: the vhost challenges through `anubis:8923` (`dankclaw/Caddyfile:102-103`) and the
post-auth listener :8017 routes it to `192.168.0.143:8081` (`dankclaw/Caddyfile:361`).
A local subagent can hit the published port directly because its
slirp/loopback/LAN sources are in `pass_ip`. The searxng network
(`systemd-searxng`, 10.89.2.0/24) is in `trusted_proxies`, so Caddy's
`X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it
stays stable across reboots.
(`systemd-searxng`, 10.89.2.0/24) is in `trusted_proxies`
(`shared/searxng/limiter.toml:22-29`), so Caddy's `X-Forwarded-For` is honored. The
subnet is pinned in `searxng.network:16`, so it stays stable across reboots — that is
what keeps the `trusted_proxies` entry meaningful.
## Unblocking Google & Search Providers
The current request policy allows a modest amount of VPN latency (`4.0s` per
request, `12.0s` maximum) while limiting retries to one. This can recover slow
The current request policy allows a modest amount of VPN latency (`5.0s` per
request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting
retries to one (`:32`). This can recover slow
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
disables observed CAPTCHA/429 providers from normal aggregation, and disables
pagination to reduce upstream request volume. SearXNG **suspends** offending
engines for hours to days, which is what shows up as "Google went unavailable".
This is upstream, not local.
VPN exit IP. SearXNG queries every engine it has enabled from one Gluetun VPN exit
IP: the running build ships 262 engines and **75 are enabled**
(`curl http://127.0.0.1:8081/config` → `engines[]`, 2026-10-18). The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
The default provider policy therefore favors anonymous Bing and DuckDuckGo (both `enabled: true`
in `/config`), disables observed CAPTCHA/429 providers from normal aggregation
(google, google cse, google news, google scholar, the whole `brave.*` family:
`enabled: false` on 2026-10-18), and disables pagination for those engines to reduce
upstream request volume. SearXNG **suspends** offending engines — which is what shows
up as "Google went unavailable". The default suspension is shorter than the folklore:
see the table in [[SearXNG Providers]]. This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
@@ -165,10 +180,11 @@ root cause, the suspend-model table, the `engines:` config, and the
### Gotchas
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
- **Agent-API contract (verified live 2026-08-31):** the guarded path
(`search.franksdank.cloud`) answers non-browser API clients with Anubis'
- **Agent-API contract (verified live 2026-08-31, re-verified 2026-10-18):** the guarded
path (`search.franksdank.cloud`) answers non-browser API clients with Anubis'
JS-challenge page at HTTP **200** — a status-code-only client silently gets
HTML where it expected JSON. Check `content-type`, not just status. There is
HTML where it expected JSON (`curl` with a JSON query through the vhost returned
`200 text/html`, 2026-10-18). Check `content-type`, not just status. There is
deliberately no way for an API client through the guard (JS challenge + UA
deny-list); the agent path is the direct published port
`http://192.168.0.143:8081/search?q=…&format=json` from trusted sources —
@@ -176,38 +192,95 @@ root cause, the suspend-model table, the `engines:` config, and the
LAN by construction (no public route to the LAN IP). No second agent-only
instance is needed; a duplicate would double upstream request volume from the
same rate-sensitive VPN exit IP for zero benefit.
- The config file is root-owned (uid of the rootless podman namespace). The
official path is `deploy.sh --service searxng`: it copies the repo settings
- The config dir is owned by uid 100976 — the searxng container's mapped root inside
the rootless user namespace, not a host root account
(`ls -la /opt/searxng/config` 2026-10-18: both files `100976 100976`).
The official path is `deploy.sh --service searxng`: it copies the repo settings
and renders the `${VAR}` placeholders via `scripts/searxng-render-settings.py`
from `/opt/searxng/.env`. A hotfix can still hand-copy with `sudo cp`, but
remember the renderer: an unrendered copy silently serves the literal
from `/opt/searxng/.env` (`scripts/deploy-remote.sh:208-216`). A hotfix can still hand-copy with `sudo cp`, but remember the renderer: an unrendered copy silently serves the literal
placeholders (before 2026-09-26 the live `secret_key` literally was
`"${SEARXNG_SECRET}"` and nobody noticed).
- `base_url` must match the public address or proxy rewrites break.
- Keep the valkey cache container on the same podman network as searxng.
`"${SEARXNG_SECRET}"` and nobody noticed). Today's deployed file carries no
unrendered placeholder — the only `${` left in it is inside the header comment
(`grep -n '[$]{' /opt/searxng/config/settings.yml`, 2026-10-18).
- `base_url` must match the public address or proxy rewrites break — **the deployed
value does not**: `SEARXNG_BASE_URL` in `/opt/searxng/.env` is set so that the
rendered config reads `base_url: "http://192.168.0.143:8081/"`
(`/opt/searxng/config/settings.yml:10`, 2026-10-18), i.e. the published port on the
LAN IP, not `https://search.franksdank.cloud`. Browser use through the vhost still
works because Caddy rewrites `Host`, but absolute links SearXNG builds point at the
LAN address. The unit also passes `SEARXNG_BASE_URL` into the container
(`searxng.container:21`), and that is one of the variables SearXNG substitutes at
load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a
config change, out of scope for docs.
- Keep the valkey cache container on the network SearXNG can actually reach — see
[[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's
netns), so the network that matters is the one Gluetun and `searxng-valkey` share.
## Kagi (paid API engine)
Added 2026-09-26 (user-ordered). The stock SearXNG image ships a native
`kagi` engine module (`searx/engines/kagi.py`); it authenticates with
`Authorization: Bearer <key>` against `POST https://kagi.com/api/v1/search`.
Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native
`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`),
which sets `Authorization: Bearer <key>` (`kagi.py:103`) and issues
`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`),
and declares `require_api_key: True` (`kagi.py:63`).
- **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
`[general, web]`, shortcut `kg`, `kagi_categ: search`.
`[general, web]`, shortcut `kg`, `kagi_categ: search`
(`shared/searxng/settings.yml:66-77`); live `/config` reports `kagi` with
`enabled: true`, categories `general, web` (2026-10-18).
- **Cost control**: `paging: false` — exactly one API credit per aggregated
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0).
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0)
(`shared/searxng/settings.yml:74-77`; upstream paging default is `True`,
`kagi.py:67`, so the `false` here is a real override).
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
`deploy-remote.sh` bakes the real key into the live settings.yml via
`scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
(dank-owned, mode 600). The key never enters git and never appears in any
process argv. SearXNG itself does NOT expand env vars in settings.yml —
without the renderer the engine fails `setup()` and is dropped at load.
(owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are
the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`).
The key never enters git and never appears in any
process argv. SearXNG does not expand *arbitrary* `${VAR}` placeholders in
settings.yml — it substitutes only a fixed set of documented environment variables
(`SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `SEARXNG_VALKEY_URL`, …; see the header
comments in upstream `searx/settings.yml` at the pinned commit), and `${KAGI_API_KEY}`
is not one of them. Without the renderer the engine fails `setup()` and is dropped
at load (`kagi.py:91` returns the error when the key is empty). The deployed file
has no unrendered placeholder left (2026-10-18).
- **Immune to the VPN-exit-IP problem**: Kagi authenticates, so it does not
care about the shared Gluetun exit IP reputation (see [[SearXNG Providers]]).
- **Verified live 2026-09-26**: direct API probe through the gluetun netns
(200, `data.search[]` shape), then a live aggregated query returned
29/29 results tagged `kagi`, zero unresponsive engines.
29/29 results tagged `kagi`, zero unresponsive engines. Re-checked 2026-10-18 only
as far as the config goes (engine enabled, key rendered); no paid query was fired
for this audit.
## Live engine census (2026-10-18)
`GET http://127.0.0.1:8081/config` answers `engines[]` for the running instance
(200 today; the endpoint would be blocked on a public instance, and
`public_instance: false` here). Counted from that response:
- 262 engines in the build, **75 enabled**, categories of the enabled set led by
`other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7).
- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`,
`github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`,
`arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`,
`wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`,
`youtube`, `vimeo`, `dailymotion`, `bing videos`.
- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`,
`brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`.
- The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target
engines that **no longer exist in this build** — `/config` lists no `startpage`,
`startpage images` or `startpage news`, so those overrides are no-ops kept as
intent. *(labelled 2026-10-18)*
## Citations
- `dankclaw/searxng/quadlet/searxng.container`, `gluetun.container`, `searxng-valkey.container`, `searxng.network`
- `shared/searxng/settings.yml`, `shared/searxng/limiter.toml`
- `scripts/deploy-searxng-quadlet.sh:11`, `scripts/deploy-remote.sh:208-216,583-593`
- `dankclaw/Caddyfile:102-103,361` (guarded vhost → post-auth → `192.168.0.143:8081`)
- Host evidence 2026-10-18: `podman ps`, `podman inspect searxng`, `podman network inspect systemd-searxng`, `ss -ltn`, `curl 127.0.0.1:8081/` → 200, `/healthz` → 200, `/config` → 200 (engine census), `/search?format=json` → 200 `application/json`, `diff shared/searxng/limiter.toml /opt/searxng/config/limiter.toml` (identical), `systemctl --user is-enabled/is-active gluetun searxng searxng-valkey`
## See Also
serverAppConfigs-docs/Dankclaw/Seerr.md
+19 -12
View File
@@ -9,26 +9,28 @@ timestamp: 2026-09-02
# Seerr
Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line). Frank's household uses it to request movies/TV/music; Seerr routes approvals to Sonarr/Radarr/Lidarr/Readarr on FrankBook-Pro and pushes downloads to qBittorrent.
Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line). It is meant to route approvals to Sonarr/Radarr/Lidarr/Readarr on FrankBook-Pro and push downloads to qBittorrent. **It is not launched:** `shared/homepage/services.json` ships the card with `wip: "down"`, `showLink: false` and the description "Work in progress: not open to family users yet", and `/srv/homepage/status.json` (2026-10-18) still reports it `wip-down` even though the container is up. Treat every "Frank's household uses it" framing as future tense.
**Host:** Dankclaw — **Hostname:** `seerr.franksdank.cloud` (Caddy, guarded) — **Docs:** https://github.com/seerr-team/seerr
## Facts (verified live, 2026-09-01)
## Facts (verified live 2026-09-01, re-verified 2026-10-18)
| Item | Value |
|------|-------|
| Image | `ghcr.io/seerr-team/seerr:v3.4.1` (runs as in-image `node`, uid 1000) |
| Port | 5055 (container), published `127.0.0.1:5055` on the host (loopback-only) |
| Data | `/home/dank/Containers/seerr/data` → `/app/config` (SQLite settings store) |
| Unit | `dankclaw/seerr/quadlet/seerr.container` → `seerr.service` |
| Network | `dankclaw_default` (post-auth listener reachability) |
| Guard | Caddy TLS edge → Anubis :8923 → Authentik outpost provider `seerr-proxy` → Caddy :8017 `@seerr` handler → `seerr:5055` |
| DNS | Pi-hole addn-hosts only. The public Porkbun A record was deleted on 2026-09-02, so the name resolves on the LAN (→ 192.168.0.143) and returns NXDOMAIN on the WAN: Seerr is internal-only by construction |
| Image | `ghcr.io/seerr-team/seerr:v3.4.1` (unit pin `dankclaw/seerr/quadlet/seerr.container:32`; live container reference `podman inspect seerr` 2026-10-18). Runs as the in-image `node` user, uid 1000 (`podman top seerr user uid`) |
| Port | 5055 (container), published `127.0.0.1:5055` only (`seerr.container:34`; `ss -ltn` shows `127.0.0.1:5055` and no `0.0.0.0` bind, 2026-10-18) |
| Data | `/home/dank/Containers/seerr/data` → `/app/config` (SQLite settings store, `seerr.container:35`; the live dir exists and is owned by uid 100999 — see UID mapping) |
| Unit | `dankclaw/seerr/quadlet/seerr.container` → `seerr.service`; `Restart=always`/`RestartSec=5` (`:38-39`), `[Install] WantedBy=default.target` (`:42-43`), installed under `~/.config/containers/systemd/` and reported `generated` / `active` with `Linger=yes` (2026-10-18) — boot-persistent |
| Network | `dankclaw_default` (post-auth listener reachability); live address 10.89.1.12 on the 10.89.1.0/24 bridge (`podman inspect seerr`, `podman network inspect dankclaw_default` 2026-10-18) |
| Guard | Caddy TLS edge → Anubis :8923 → Authentik outpost provider `seerr-proxy` *(unverified 2026-10-18: the provider name lives in Authentik's own DB)* → Caddy :8017 `@seerr` handler → `seerr:5055` (`dankclaw/Caddyfile:149-150` for the challenged vhost, `:371` for the post-auth `reverse_proxy seerr:5055`) |
| DNS | Pi-hole addn-hosts only: `192.168.0.143 seerr.franksdank.cloud` (`dankclaw/pihole/addn-hosts:46`). The public Porkbun A record was deleted on 2026-09-02, so the name resolves on the LAN (→ 192.168.0.143) and not on the WAN — re-checked 2026-10-18: a public-resolver query for `seerr.franksdank.cloud` returns no A record, while `ai.franksdank.cloud` returns the WAN address. Seerr is internal-only by construction |
## Why loopback-only publishing
Seerr is browser-facing; the only sanctioned user path is the guard chain (Caddy → Anubis → Authentik SSO). `PublishPort=127.0.0.1:5055:5055` keeps the app off the LAN entirely while still letting the homepage probe agent and host-local checks reach it — the same loopback convention as other guarded hosts (see [[Homepage Design Grilling]]).
There is no health endpoint to point at: measured on 2026-10-18, `GET http://127.0.0.1:5055/` answers **307** (redirect into the app) and `GET /api/v1/version` answers **404** — Seerr does not expose that path, so do not wire it as a health probe. The homepage card probes `127.0.0.1:5055` and is marked `wip: "down"`, which is why the card stays hidden while the container is up.
## Guard chain fix (2026-09-02)
The `seerr-proxy` provider was provisioned by script with **no flows and no
@@ -43,18 +45,23 @@ provisioning rules this incident established.
## *arr wiring
Seerr reaches the *arr services on FrankBook-Pro (192.168.0.112, gluetun-NetNS stack inside the Mac's podman machine VM) by IP-based base URLs (`http://192.168.0.112:<port>`), configured inside Seerr's own web settings — not in env or quadlet. The `.lan` hostname variant was deliberately dropped: addn-hosts records are for fleet public-domain hosts, and IP bases match the Mac stack's verified LAN reachability. API keys live in each arr's config.xml on the Mac and are entered through Seerr's settings store.
Seerr reaches the *arr services on FrankBook-Pro (192.168.0.112, gluetun-NetNS stack inside the Mac's podman machine VM) by IP-based base URLs (`http://192.168.0.112:<port>`), configured inside Seerr's own web settings — not in env or quadlet. The `.lan` hostname variant was deliberately dropped: `dankclaw/pihole/addn-hosts` carries no `frankbook-pro.lan` record (verified against the file, 2026-10-18), and IP bases match the Mac stack's verified LAN reachability. *(unverified 2026-10-18: 192.168.0.112 is offline at this census — `ping` fails and the arr probes report `down` in `/srv/homepage/status.json`, so LAN reachability from Seerr could not be re-measured; the quadlet header's "resolved via the Pi-hole addn-hosts record frankbook-pro.lan" is stale for the same reason — no such record exists.)* API keys live in each arr's config.xml on the Mac and are entered through Seerr's settings store. *(unverified 2026-10-18: the keys are inside Seerr's SQLite store, not readable from the repo.)*
OIDC single-sign-on for Seerr (Authentik OIDC provider + Seerr OIDC config) remains an open option; the guard chain already authenticates the front door, so this is an in-app setting deferred to Frank's first-run session.
## UID mapping gotcha
The container runs as `node` (uid 1000 in-image). Rootless podman's continuous mapping is container uid N → host uid `100000 + N - 1`, so uid 1000 → **100999**. The first deploy chowned the data dir to 100999 + 1 (= 101000) and crash-looped with `EACCES mkdir /app/config/logs`. Rule: read the real mapping from any container's `/proc/self/uid_map` before chowning bind roots for non-root images.
The container runs as `node` (uid 1000 in-image — `podman top seerr user uid` 2026-10-18). Rootless podman's mapping on this host is `0 → 1000 (size 1)`, `1 → 100000 (size 65536)` (`/proc/<pid>/uid_map` of the running container; `podman unshare cat /proc/self/uid_map` agrees), i.e. container uid N ≥ 1 → host uid `100000 + N - 1`, so uid 1000 → **100999**. That is the owner the data tree actually carries: `ls -n /home/dank/Containers/seerr/data` shows `100999 100999` on `db/`, `logs/` and the files the app writes (2026-10-18). The first deploy chowned the data dir to 100999 + 1 (= 101000) and crash-looped with `EACCES mkdir /app/config/logs`. Rule: read the real mapping from a running container's `/proc/<pid>/uid_map` before chowning bind roots for non-root images.
## Deployment
`./scripts/deploy.sh dankclaw --service seerr`. The deployer pre-creates the bind-source dir, installs the unit, and restarts. First visit after deploy: complete the Seerr local-admin setup wizard (the guard chain bounces through Authentik first). TLS issuance for a new vhost fires at Caddy reload (dns-01) — allow ~15 s before the first HTTPS probe.
`./scripts/deploy.sh dankclaw --service seerr` (`seerr` is a registered service in `scripts/deploy.sh:143`). The deployer pre-creates the bind-source dir, installs the unit, and restarts. First visit after deploy: complete the Seerr local-admin setup wizard (the guard chain bounces through Authentik first). TLS issuance for a new vhost fires at Caddy reload (dns-01) — allow ~15 s before the first HTTPS probe.
## Citations
- Seerr image and volume contract: https://github.com/seerr-team/seerr (official docs recommend `/app/config` persistence)
- Quadlet unit: `dankclaw/seerr/quadlet/seerr.container` (image pin `:32`, loopback publish `:34`, volume `:35`, `Restart=always`/`WantedBy` `:38-43`)
- Guard chain: `dankclaw/Caddyfile:149-150` (challenged vhost) and `dankclaw/Caddyfile:371` (post-auth `reverse_proxy seerr:5055`)
- DNS: `dankclaw/pihole/addn-hosts:46`; public-resolver query for `seerr.franksdank.cloud` (no record), 2026-10-18
- Homepage state: `shared/homepage/services.json` (`Seerr` card, `wip: "down"`) and `/srv/homepage/status.json` 2026-10-18 (`wip-down`)
- Host evidence 2026-10-18: `podman ps`, `podman inspect seerr`, `podman top seerr user uid`, `ss -ltn`, `curl 127.0.0.1:5055/` → 307, `curl 127.0.0.1:5055/api/v1/version` → 404, `systemctl --user is-enabled/is-active seerr`
serverAppConfigs-docs/Dankclaw/Swap Configuration.md
+39 -25
View File
@@ -16,15 +16,17 @@ timestamp: 2026-09-12
| Item | Value |
| --- | --- |
| Swapfile | `/swap.img` |
| Size | **64 GiB** (raised from 8 GiB on 2026-09-12) |
| Filesystem | ext4 — swapfiles fully supported, no btrfs `nocow` handling needed |
| Allocation | `fallocate` (preallocated, hole-free; 530 extents) |
| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` |
| systemd unit | `swap.img.swap` — `loaded (/etc/fstab; generated)`, `active` |
| `vm.swappiness` | `60` (distro default, unchanged) |
| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`) |
| Free on `/` after | 748 GiB of 1.9 TiB |
| Swapfile | `/swap.img` (68,719,476,736 B = 64 GiB, owner root 0600, mtime 2026-09-12 21:56 — `ls -la /swap.img`) |
| Size | **64 GiB** (raised from 8 GiB on 2026-09-12); `/proc/swaps` reports 67,108,860 KiB |
| Type / priority | `file`, priority **-1** — the only swap device on the host (`swapon --noheadings` 2026-10-18: `/swap.img file 64G 187.6M -1`) |
| Filesystem | ext4 on `/dev/nvme0n1p2` (`df -T /` 2026-10-18) — swapfiles fully supported, no btrfs `nocow` handling needed |
| Allocation | `fallocate` (preallocated, hole-free; **530 extents** — `sudo filefrag -v /swap.img` → "530 extents found", 2026-10-18) |
| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` (`grep -n -i swap /etc/fstab` 2026-10-18) |
| systemd unit | `swap.img.swap` — `LoadState=loaded`, `FragmentPath=/run/systemd/generator/swap.img.swap`, `UnitFileState=generated`, `is-active` → `active` (2026-10-18) |
| `vm.swappiness` | `60` (distro default, unchanged — `sysctl vm.swappiness` 2026-10-18) |
| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`, 2026-10-18) |
| Swap in use | 187 MiB of 63 GiB (`free -h` 2026-10-18) |
| Free on `/` | **424 GiB of 1.9 TiB, 77 % used** (`df -T /` 2026-10-18). The 748 GiB that stood right after the 2026-09-12 resize is gone — ordinary fleet writes since then |
The unit is **generated from fstab**, not a hand-written unit file. fstab is the
only control point; `systemctl daemon-reload` re-reads it. Nothing in this
@@ -33,12 +35,19 @@ it is not invisible.
## Why 64 GiB
Dankclaw is shared. `frank-dragoon` alone holds **103.5 GiB of the 121.2 GiB**
host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly
**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The
engine itself warns that when that runs out, frank-dragoon and its neighbours
"can stall for minutes at a time at full CPU with no disk activity and no
output". Swap is the valve that turns an OOM kill into slow paging.
Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1`
working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose
working set was ~4.3 GiB. That is a decision-time reading, not today's: on
2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
`frank-dragoon-halogen` container accounts for **70.33 GiB**
(`podman stats --no-stream`, same day) on the current flash-server payload —
a different model and KV shape than the one measured in September.
The engine itself warned at the time that when the headroom runs out,
frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
no disk activity and no output". Swap is the valve that turns an OOM kill into slow
paging.
Swap does **not** relieve the pinned-weights problem — see the caveat below.
It protects the *co-tenants* and buys headroom for burst allocation.
@@ -84,10 +93,10 @@ under memory pressure.
## Verification
```bash
swapon --show=NAME,TYPE,SIZE,USED,PRIO # /swap.img file 64G
systemctl is-active swap.img.swap # active
grep swap /etc/fstab # line 12 intact -> boot persistence
free -h | tail -2 # Swap: 63Gi total
swapon --show=NAME,TYPE,SIZE,USED,PRIO # /swap.img file 64G 187.6M -1 (2026-10-18)
systemctl is-active swap.img.swap # active; FragmentPath=/run/systemd/generator/swap.img.swap
grep -n swap /etc/fstab # line 12 intact -> boot persistence
free -h | tail -2 # Swap: 63Gi total, 187Mi used (2026-10-18)
```
Then confirm the inference service is unaffected — a completion, not just
@@ -113,9 +122,10 @@ frank-dragoon `health=200` throughout.
pool and working set *are* anonymous. Swapping the KV pool would be far worse
than the OOM kill it prevents. If the box ever starts thrashing, lowering
swappiness (or capping the engine's cgroup) is the knob, not more swap.
- **`free` still lies after the change.** `MemAvailable` reads ~70 GiB while
the model is loaded. Trust the engine's `host memory left for everything
else` line, not `free`.
- **`free` still lies after the change.** On 2026-09-12 `MemAvailable` read ~70 GiB
while the model was loaded; on 2026-10-18 it reads **29 GiB** with 90 GiB sitting in
`buff/cache` (`free -h`). The headline number moves with the payload; trust the
engine's `host memory left for everything else` line, not `free`.
- **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
@@ -131,9 +141,13 @@ frank-dragoon `health=200` throughout.
## Trash
The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). Purge
only on explicit approval.
The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging
directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only
`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`,
`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`,
`20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`.
The old swapfile has therefore been purged; nothing on the host still holds it.
## See Also
serverAppConfigs-docs/Dankclaw/Valkey.md
+32 -11
View File
@@ -9,7 +9,7 @@ timestamp: 2026-08-28
# Valkey
Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment.
Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment — the running build (2026.9.25+12f8b6515, `/config` 2026-10-18) reads a `valkey:` settings key and has no `redis:` key (`searx/settings.yml` at commit `12f8b6515`, https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml).
**Host:** Dankclaw — **Docs:** https://valkey.io
@@ -17,13 +17,21 @@ Valkey is the cache and limiter store for SearXNG. It replaces Redis as the defa
| Item | Value |
|------|-------|
| Image | `valkey/valkey:9-alpine` |
| Port | 6379 (internal, on the SearXNG network) |
| Image | `docker.io/valkey/valkey:9-alpine` (`dankclaw/searxng/quadlet/searxng-valkey.container:19`; live container image reference `podman inspect searxng-valkey` 2026-10-18) |
| Port | 6379 container-internal — the unit publishes **no** port (`dankclaw/searxng/quadlet/searxng-valkey.container` has no `PublishPort`; `ss -ltn 2026-10-18` shows nothing on 6379 and a TCP probe to `192.168.0.143:6379` is refused), so `shared/homepage/services.json` marks the card `wip: "closed"` |
| Role | SearXNG limiter + result cache |
## Why Valkey
The official SearXNG docker-compose ships a `searxng-valkey` container. Valkey is the Redis fork under the Linux Foundation umbrella; the SearXNG project moved to it as the default cache.
Valkey is the Redis fork under the Linux Foundation umbrella, and the SearXNG project
moved to it as the default cache: the pinned build's `searx/settings.yml` exposes
`valkey: { url: … }` and has no `redis:` key
(https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml).
The `searxng-valkey` service name and the `valkey/valkey:9-alpine` image are this
repo's choice (`dankclaw/searxng/quadlet/searxng-valkey.container:16-19`), matching
the container name SearXNG's settings then point at. *(unverified 2026-10-18: the
claim that upstream's own docker-compose ships a container named `searxng-valkey`
is not checkable from this repo)*
## Configuration
@@ -38,25 +46,38 @@ Image=docker.io/valkey/valkey:9-alpine
Volume=/opt/searxng/valkey:/data
```
SearXNG connects to it via `settings.yml`:
SearXNG connects to it via `settings.yml`, which addresses the **container name**,
not the alias — the alias is a second, working name for the same endpoint:
```yaml
valkey:
url: "valkey://searxng-valkey:6379/0"
```
It sets `WantedBy=default.target`. User linger is on, so it starts at boot.
Repo source `shared/searxng/settings.yml:22-23`; the deployed render
`/opt/searxng/config/settings.yml:23` is byte-identical on this key (2026-10-18).
It sets `WantedBy=default.target` and `Restart=always`
(`dankclaw/searxng/quadlet/searxng-valkey.container:25-28`), the unit is installed at
`~/.config/containers/systemd/searxng-valkey.container`, and `systemctl --user
is-enabled` reports `generated` / `is-active` `active` with `Linger=yes`
(host check 2026-10-18), so it starts at boot.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/data` | `/opt/searxng/valkey` |
| Container path | Host path |
|----------------|-----------|
| `/data` | `/opt/searxng/valkey` (`dankclaw/searxng/quadlet/searxng-valkey.container:20`; the live dir exists, owner uid 100998 — `ls -ld /opt/searxng/valkey` 2026-10-18) |
## Gotchas
- The limiter relies on Valkey; without it SearXNG falls back to in-memory state and rate limiting is per-restart.
- Keep Valkey on the same podman network as SearXNG.
- The limiter relies on Valkey; without it SearXNG falls back to in-memory state and rate limiting is per-restart. *(unverified 2026-10-18: upstream fallback behaviour not read; the pinned default is `url: false`, i.e. no cache configured)*
- The cache is reachable because both containers share one network: `searxng-valkey`
joins `searxng.network` (`searxng-valkey.container:17`), and SearXNG itself has no
network of its own — it reuses Gluetun's netns (`searxng.container:18`), and Gluetun
is the member of `systemd-searxng` (10.89.2.0/24, `podman network inspect
systemd-searxng` 2026-10-18). Keeping the cache off that network breaks SearXNG's
limiter, not just the result cache.
## See Also
serverAppConfigs-docs/Infrastructure/Alertmanager.md
+11 -8
View File
@@ -8,7 +8,7 @@ timestamp: 2026-09-17
# Alertmanager
Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.
Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel (`shared/grafana/provisioning/dashboards/llm-fleet.json`, the "Firing fleet alerts" panel) and reached nobody.
**Host:** Dankclaw — **Port:** 9093 (loopback) — **Docs:** https://prometheus.io/docs/alerting/latest/alertmanager/
@@ -17,13 +17,14 @@ Alertmanager receives firing rules from Prometheus, groups and dedupes them, and
| Item | Value |
|------|-------|
| Image | `docker.io/prom/alertmanager:latest` |
| Published | `127.0.0.1:9093` → 9093, loopback only (see [[Observability]] → Exposure) |
| Published | `127.0.0.1:9093` → 9093, loopback only — never a wildcard (`alertmanager.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9093` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
| Unit | `dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service` |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alertmanager.container:49-54`); installed at `~/.config/containers/systemd/alertmanager.container` and reporting `generated` + `active` (`systemctl --user is-enabled|is-active alertmanager` 2026-10-18) |
| Config | `/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy |
| Data | `/opt/alertmanager/data` (silences, notification log) |
| Health | `wget --spider -q http://localhost:9093/-/healthy` |
| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` |
| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h |
| Health | `wget --spider -q http://localhost:9093/-/healthy` (`alertmanager.container:43`; the deployer waits on the same path on the published loopback, `scripts/deploy-remote.sh:532`) |
| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` (`shared/prometheus/alertmanager.yml:27-30`) |
| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h (`shared/prometheus/alertmanager.yml:21-25`) |
## Why the webhook address is `host.containers.internal`
@@ -36,7 +37,9 @@ The Signal relay is a host-side process, and the address it must bind is the one
- POST `/alertmanager` — the Alertmanager webhook handler; formats the group and sends it.
- GET `/health` — liveness for the deployer's health wait.
- Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h `repeat_interval` during a relay hiccup.
- Sends through `/usr/local/bin/signal-cli -a +12062476336 -o json send +12062476336 -m <text>` — Frank's existing account (state dir `318311`); the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
- Sends through `/usr/local/bin/signal-cli -a +12062476336 -o json send +12062476336 -m <text>` (`scripts/signal-relay.py:88-104`) — Frank's existing account (state dir `318311`, `ls ~/.local/share/signal-cli/data` 2026-10-18: `318311`, `318311.d`, `accounts.json`); the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
Delivery is measured, not assumed. The relay's journal carries successful sends for every rule in the set — `journalctl --user -u signal-relay` (2026-10-18) ends `signal-relay: sent to +12062476336: [CRITICAL] LlmEndpointDown (1 firing)`, `… [WARNING] LlmSpecDecodeOff (1 firing)`, `… [INFO] LlmPrefixCacheCollapse (1 firing)`, with matching `(1 resolved)` lines and `suppressed duplicate` where the 15-minute window collapsed a repeat. Each of those lines is printed only after `signal-cli` exits 0 (`scripts/signal-relay.py:124-126`), and `ss -ltnp` (2026-10-18) shows `python3` bound to `192.168.0.143:18093`, with `curl -s http://192.168.0.143:18093/health` → `ok`. What the phone itself shows cannot be checked from this host. *(unverified 2026-10-18: handset receipt)*
## Deployment
@@ -51,8 +54,8 @@ The Signal relay is a host-side process, and the address it must bind is the one
- The webhook URL and the relay's listen port are a **cross-file contract**: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. `tests/observability_quadlet.bats` pins both halves.
- The container address is `alertmanager:9093`, never the published port: Prometheus is in a container and the port is loopback-only.
- signal-cli account state is on the host, not in the container — a wiped `~/.local/share/signal-cli` silences delivery while every status page stays green.
- signal-cli 0.14.7's `-a` flag parses an all-digit value as a **phone number** and never falls back to the `accounts.json` name: the account's numeric name (`318311`) dies with `Invalid account (phone number)`, while `-a +12062476336` works. The message goes via `-m`, not a positional — a positional message is parsed as a second recipient (`Invalid phone number '…'`).
- A failed send is **not** marked in the dedupe map, so the next Alertmanager repeat retries it — but the relay returns 200 to the webhook regardless (a relay bug must not 500 Alertmanager), so a dead signal-cli shows up only in the relay's journal, not in Alertmanager's.
- signal-cli 0.14.7 (`signal-cli --version` on Dankclaw, 2026-10-18) parses an all-digit `-a` value as a **phone number** and never falls back to the `accounts.json` name: the account's numeric name (`318311`) dies with `Invalid account (phone number)`, while `-a +12062476336` works (`scripts/signal-relay.py:36-41`). The message goes via `-m`, not a positional — a positional message is parsed as a second recipient (`Invalid phone number '…'`).
- A failed send is **not** marked in the dedupe map (`scripts/signal-relay.py:124-129`), so the next Alertmanager repeat retries it — but a valid webhook always gets a 200 back, even when the handler raises (`scripts/signal-relay.py:155-161`: a relay bug must not 500 Alertmanager; only an unparseable payload gets a 400), so a dead signal-cli shows up only in the relay's journal, not in Alertmanager's.
## See Also
serverAppConfigs-docs/Infrastructure/Alloy.md
+21 -9
View File
@@ -8,7 +8,7 @@ timestamp: 2026-09-11
# Alloy
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). It forwards container and journal logs to Loki.
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below.
**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest
@@ -19,22 +19,24 @@ Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-
| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) |
| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) |
| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` |
| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/` |
| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) |
| Inputs | host journald socket, podman container logs, `/var/log` |
| Output | `http://loki:3100/loki/api/v1/push` |
| Successor to | Promtail (EOL 2026-03-02) |
| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) |
| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) |
| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" |
## Why Alloy and not Promtail
Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`).
## Configuration
`shared/alloy/config.alloy` is the source of truth:
- Reads journald logs from the host's `/run/systemd/journal/socket`.
- Reads container logs from `/var/lib/containers/storage/...` (podman).
- Forwards to Loki with `job` labels.
- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`).
- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`).
- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
## Volumes
@@ -47,7 +49,17 @@ Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs
## The UI port is published on purpose
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured"). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` and publishes `127.0.0.1:12345` — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider.
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
## Delivery state (2026-10-18): healthy, shipping nothing
The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
## Gotchas
serverAppConfigs-docs/Infrastructure/Anubis.md
+84 -34
View File
@@ -35,37 +35,67 @@ graph LR
|------|-------|
| Runtime | Quadlet unit `anubis.service` (`dankclaw/anubis/quadlet/anubis.container`) — boot-persistent, self-healing |
| Image | `ghcr.io/techarohq/anubis:v1.27.0`, run under a `localhost/` alias tag (see gotcha 1) |
| Ports | `:8923` internal (no published port); metrics on `:9090` reachable only inside `dankclaw_default` |
| Ports | `:8923` internal (no published port — `podman ps` shows no port mapping, 2026-10-18); metrics on `:9090` reachable only inside `dankclaw_default` (`dankclaw/anubis/quadlet/anubis.container:3,58-61`) |
| Config surface | flags via flagenv — `TARGET=http://caddy:8016` sets `--target`, `POLICY_FNAME=/policy/policy.json` sets `--policy-fname`, `COOKIE_DOMAIN=franksdank.cloud` sets `--cookie-domain` |
| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container; parsed at process start only (no reload signal exists — every policy change needs a unit restart) |
| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container (`anubis.container:52`); parsed at process start only — the deployer restarts the unit after every policy change (`scripts/deploy-remote.sh:1091`) and no reload is wired here *(unverified 2026-10-18: whether upstream Anubis answers any reload signal at all; upstream docs are themselves behind an Anubis challenge, so this could not be read)* |
| Store | memory JWTs signed by a **persistent** ED25519 key (`ED25519_PRIVATE_KEY_HEX`, host file `/home/dank/Containers/anubis/env`, mode 0600, gitea-style `${VAR}` expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot |
Policy content: an `ALLOW` rule for Anubis's own endpoints (the
`/.within.website/x/cmd/anubis/*` API and static namespace — the challenge
JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging
those endpoints makes the challenge challenge itself), an `ALLOW` rule for
the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start,
callback and the outpost's static assets must be reachable before any
session or challenge dance completes — authentik's own session gate protects
them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*,
Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends),
then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off.
Valid action verbs
at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them
(first deploy of that rule crash-looped the unit on exactly that typo;
verified against the image binary's string table). The policy schema at this
tag is JSON-only — no `(data)/` imports and no CEL; first match wins (JSON
still loads at v1.27.0 — scratch-verified 2026-09-02 before the pin bump).
Policy content, in the file's own order — first match wins, so the DENY
really does come first (`dankclaw/anubis/policy.json`, 2026-10-18): a **DENY**
for AI-scraper user-agent regexes (GPTBot, ChatGPT-User, OAI-SearchBot,
Claude-\*, anthropic-ai, Perplexity\*, CCBot, Bytespider, Amazonbot,
meta-externalagent, Google-Extended, DuckAssistBot, cohere-ai, YouBot,
KimiBot, Applebot-Extended; lines 2-7), then an **ALLOW** for Anubis's own
endpoint namespace `^/\.within\.website` (lines 8-12 — the challenge JS calls
`make-challenge`/`pass-challenge` under that prefix, and challenging those
endpoints makes the challenge challenge itself), then an **ALLOW** for the
Authentik transport prefix `^/outpost\.goauthentik\.io/` (lines 13-17 — login,
start, callback and the outpost's static assets must be reachable before any
session or challenge dance completes; authentik's own session gate protects
them), then **CHALLENGE** for everything else (`path_regex: "."` catch-all,
lines 18-22). DNSBL off (`dnsbl: false`, line 23). Live on 2026-10-18: a
`Mozilla/5.0` request to `https://www.franksdank.cloud/` returns 200 carrying
`anubis_challenge` and the `cachebuster=v1.27.0` marker, while
`-A GPTBot` returns the deny body — see gotcha 9 for why that is also a 200.
Valid action verbs in this file are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT`
is not one of them (first deploy of that rule crash-looped the unit on
exactly that typo; verified against the image binary's string table). The
local policy uses only the JSON rule form. Upstream documents the policy file
as YAML with JSON accepted because it is valid YAML "as of version v1.17.0 or
later" (https://anubis.techaro.lol/docs/admin/policies/), and import
statements exist in the 1.x line (https://github.com/TecharoHQ/anubis/releases
lists "allow wildcard matches in import statements" on the road to v1.27.0) —
so the older local statement "JSON-only, no `(data)/` imports, no CEL at this
tag" is a v1.13-era claim kept here as history and is **not** a property of
v1.27.0; whether CEL expressions are accepted at this exact tag was not
verifiable *(unverified 2026-10-18)*. What is verified is that this JSON
policy loads and is enforced (container `healthy`; 136 challenges issued in the
retained log since the 01:25 boot, scraper deny confirmed by probe).
Upgrading the pin = pull + re-tag the `localhost/` alias, restart; verify
the flag surface with `podman run --rm ... --help` and the policy against
the new tag first (done for v1.27.0).
the new tag first (done for v1.27.0). Upstream is ahead of the pin: Anubis's
own documentation site served its challenge page with
`cacheBuster=v1.28.2-0.20261012194510-…` when fetched on 2026-10-18, so
v1.28.x exists.
## Exempt by construction
These never route through Anubis (no policy bypass rule needed): plex
(plain-HTTP legacy), auth.franksdank.cloud (the IdP login itself must stay
ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
`/info/refs` and `git-*-pack` paths pass unchallenged).
These never route through Anubis (no policy bypass rule needed): `auth.`
(the IdP login itself must stay ungated, `dankclaw/Caddyfile:90-99`), `gitea`
(git smart-HTTP clients cannot solve JS challenges — `/info/refs` and
`git-*-pack` paths pass unchallenged, `dankclaw/Caddyfile:110-120`), `ai.`
OpenWebUI (own SSO + API-key auth, `dankclaw/Caddyfile:145-151`), `pihole.`
(a deliberate basicauth edge instead, `dankclaw/Caddyfile:162-171`), `hermes.`
and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
`dankclaw/Caddyfile:179-201`), `plex.` including the plain-HTTP legacy name
(`dankclaw/Caddyfile:203-238`), the media vhosts `books.`/`jelly.`
(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
it is exempt by absence — and therefore unauthenticated on the LAN at
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
(`dankclaw/Caddyfile:283-423`).
## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)
@@ -77,8 +107,12 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
path, a container created from a registry-prefixed image ref fails with
`image not known` even when the image is present in storage; the identical
argv works from a shell or a transient unit. The unit therefore runs the
image under a `localhost/techarohq/anubis:v1.13.0` alias tag — deploy
pulls the canonical ref, then `podman tag`s the alias (idempotent).
image under a `localhost/techarohq/anubis:v1.27.0` alias tag — deploy
pulls the canonical ref, then `podman tag`s the alias (idempotent;
`scripts/deploy-remote.sh:1078-1084`, and `podman inspect anubis` reports
`localhost/techarohq/anubis:v1.27.0` on 2026-10-18). The pin used to be
v1.13.0; that name is retired, and the unit sets `Pull=never`
(`anubis.container:54-56`) so a boot cannot stall on ghcr rate limits.
3. **Health probe exec form.** The image is a distroless ko-build (no shell;
entrypoint `/ko-app/anubis`). This quadlet generator joins `HealthCmd`
into one `\x20`-escaped argv element and podman then runs it shell-form —
@@ -89,9 +123,13 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not
forward_auth — Caddy's public listeners proxy to `:8923`). A true rollback
is a Caddyfile revert + redeploy of the caddy service.
5. **Deploy handler pull guard.** `deploy_anubis` guards the pull with
`podman image exists`; a cold anonymous ghcr token request can 403 on the
first fetch — retry once, the auth cache makes the second attempt local.
5. **Deploy handler pull guard.** `deploy_anubis`
(`scripts/deploy-remote.sh:1074-1092`) guards the pull with
`podman image exists ghcr.io/techarohq/anubis:v1.27.0`; a cold anonymous
ghcr token request can 403 on the first fetch — retry once, the auth cache
makes the second attempt local. The handler always ends with
`systemctl --user restart anubis.service` + a health wait, which is what
makes a policy change land (gotcha 7).
6. **JS-capable clients are the canary.** Non-JS clients (curl, bots,
scrapers) never solve challenges — they just receive the challenge page,
so every curl probe looks green while the verification step is broken.
@@ -163,15 +201,20 @@ Layer-by-layer checks (run from the host via ssh):
Tool/behavior notes: the anubis container is distroless — `podman exec anubis
sh` fails (`sh: executable not found`); inspect via its logs or the served
challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
podman IP) for ALL clients — rootless-podman published-port SNAT;
clients are indistinguishable by IP and clearance is cookie-based. Do NOT
mistake that for the bug. Homepage render: `app.js` does
challenge page. Anubis logs the Caddy container's podman address as
`x-real-ip` for ALL clients — rootless-podman published-port SNAT; clients
are indistinguishable by IP and clearance is cookie-based. Do NOT mistake
that for the bug. The address is whatever netavark gave Caddy on
`dankclaw_default` — all 138 logged `x-real-ip` values in the container's
retained log read `10.89.1.8` (2026-10-18), and it is not a pinned address,
so never fingerprint "the Caddy hop" on a literal `10.89.1.22`-style value.
Homepage render: `app.js` does
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
failure the catch sets `#summary` to `status unavailable: <err>` and the
grid stays EMPTY — which a user reads as "links gone / not loading." A
30-service status.json with a 9-service services.json renders fine
(stateByName lookup).
status.json that carries states for more services than services.json lists
renders fine (stateByName lookup); today both files carry 35 entries
(`shared/homepage/services.json`, `/srv/homepage/status.json`, 2026-10-18).
**Decision:** if every layer is healthy (challenge valid, outpost clean,
handlers 200, files render), the fault is client-side/session. STOP
@@ -194,3 +237,10 @@ curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && e
# outpost transport exempt (ALLOW rule)
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
```
Ran on 2026-10-18 from Dankclaw with `--resolve …:443:192.168.0.143`: the apex
308s to `https://www.franksdank.cloud/`; `www.` returns 200 with
`anubis_challenge` + `cachebuster=v1.27.0`; `-A GPTBot` returns 200 whose body
contains `Access Denied`; `search./outpost.goauthentik.io/login` returns 302
(exempt, not challenged). `systemctl --user is-active anubis.service` →
`active`, `is-enabled` → `generated`, container health `healthy`.
serverAppConfigs-docs/Infrastructure/Authentik.md
+243 -104
View File
@@ -11,27 +11,43 @@ timestamp: 2026-09-16
Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication with its embedded proxy outpost. Apps that
have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack,
Pi-hole) are not wrapped — they handle their own session.
application-level authentication with its proxy outpost. Not wrapped: Gitea,
OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
they handle their own session. Grafana is the exception that is both: it is a
guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
`dankclaw/Caddyfile:374-377`; live provider table below).
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`)
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
2026-10-18)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
`authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)
- **LDAP outpost:** a 4th, quadlet-deployed unit (`authentik-outpost-ldap`)
serves LDAP for Jellyfin — see the section below.
- **Stack: five quadlet units, no compose.** `authentik-postgres`,
`authentik-server`, `authentik-worker` (`deploy_authentik`,
`scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
`deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
`scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
`active`, `Restart=always`, `WantedBy=default.target`, with linger on
(`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
pre-quadlet era, retired.
- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.
## Why Authentik
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
for all configuration, an outpost that terminates auth for reverse-proxied
apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis
was removed in 2025.10, so the stack is 3 services, not 4. (The integration
mechanism changed in 2026.x — see Architecture; the research file predates
the pin.)
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was
recorded in `.docaudit/research/auth.md`, which no longer exists in the repo or
in the host checkout (`ls`, 2026-10-18) *(unverified 2026-10-18: the
comparison itself)*. In short: full OIDC with single-logout, a web GUI for all
configuration, an outpost that terminates auth for reverse-proxied apps, and a
footprint that fits on a 3-user homelab — measured idle today
`podman stats --no-stream`: postgres 233.8 MB + server 739.1 MB + worker
292.4 MB + outpost 294.3 MB + ldap outpost 46.6 MB ≈ **1.6 GB** across five
containers (the older "~1–1.5 GB" figure was the three-service stack). Redis
was removed in 2025.10; the integration mechanism changed again in 2026.x (see
Architecture).
## Architecture
@@ -46,6 +62,25 @@ graph LR
S --> PG[(postgres)]
```
Who answers was re-measured 2026-10-18 from inside the `caddy` container with a
tenant-matching `Host: search.franksdank.cloud`:
`http://authentik-server:9000/outpost.goauthentik.io/start` → **302 into the
authentication flow**, while `http://authentik-outpost:9000/…` answers **404**
for `/outpost.goauthentik.io/*` *and* for `/if/flow/default-authentication-flow/`.
So the **server process** still terminates the proxy-provider decision, exactly
as Caddy's `:8016 → authentik-server:9000` hop assumes
(`dankclaw/Caddyfile:93-98`). The `authentik-outpost` unit is installed,
enabled (`generated`) and active, and it is the DB's `authentik Embedded
Outpost` (`7e39655b-…`, nine providers attached) holding the websocket channel
to core — but it serves no HTTP surface on the paths the chain needs. The unit
header's stronger claim that the URL space "is served by the Rust worker
binary, not the server"
(`dankclaw/authentik/quadlet/authentik-outpost.container:6-14`) is **not** what
this probe shows; do not point a new listener at `authentik-outpost:9000`
without re-measuring. Note also that a probe without a tenant `Host` (e.g.
`curl http://127.0.0.1:9000/…`) 404s on paths that work — that is the
Host-routing rule below, not an outage.
Two mechanisms were disproved live (2026-08-29/30) and one is the actual
contract:
@@ -97,7 +132,7 @@ http://:8016 {
}
```
The embedded outpost inside the server process decides by `Host`:
The outpost decides by `Host`:
no session → the provider's `authentication_flow` is served in place;
session valid → the request is forwarded to the provider's `internal_host`
(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.
@@ -107,10 +142,15 @@ routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.
Live provider inventory (verified 2026-09-02), all attached to outpost
`7e39655b-…` ("authentik Embedded Outpost"), all with
`internal_host = http://caddy:8017`, all three flows set, and each linked to
an Application (see the provisioning rules below):
Live provider inventory (read from the `authentik-postgres` container with
read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
members in `authentik_outposts_outpost_providers`), all with
`internal_host = http://caddy:8017`, all carrying the **same three flow
UUIDs** (`default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
`default-provider-invalidation-flow`), and each backed by an Application
(the `slug` column below):
| Provider pk | name | external_host | app slug |
|---|---|---|---|
@@ -122,13 +162,21 @@ an Application (see the provisioning rules below):
| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |
Provider 3 (`hermes-proxy`) still exists but is dormant — its vhost was
decommissioned from Caddy on 2026-08-31. Provider 9 (`grafana`) and 12
(`Seerr OIDC`) are OAuth2 relying-party providers, not proxy providers; the
outpost does not deserialize them. Trailing slashes on `external_host` (the
runbook-recipe style) and bare origins both intercept correctly — the outpost
matches on hostname.
That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
the lockstep rule holds today. Two corrections against the 2026-09-02 table:
**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
not proxy providers, so the proxy outpost does not deserialize them: pk 9
`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
below rejects), and pk 14 `Open WebUI`
(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
`external_host` (the runbook-recipe style) and bare origins both intercept
correctly — the outpost matches on hostname.
Two provisioning rules, each verified live by an outage:
@@ -188,47 +236,65 @@ implementation diverges from upstream in ways that bit us; all verified live:
`{sub}` only — email/groups/entitlements claims cannot be delivered by this
build until the evaluator is patched or upgraded. Clients consuming this
provider must key identity off `sub`.
- Scope mappings for this provider exist (`grafana-email-scope`,
`grafana-profile-scope`, Python-syntax expressions) and will start emitting
claims the moment the evaluator bug is fixed — no config change needed then.
- Scope mappings for this provider exist — `grafana-email-scope` and
`grafana-profile-scope` are both rows in `authentik_core_propertymapping`
(read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
the evaluator bug is fixed; no config change is needed then.
## LDAP outpost (Jellyfin auth, live 2026-09-01)
A fourth unit, `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, serves LDAP for Jellyfin's LDAP-Auth
plugin. The image is pinned by its **amd64 single-platform digest**
(`ghcr.io/goauthentik/ldap@sha256:fa33bc8f522113830e1662f4c034352f9eb08e0a7d3da2ddadad9a4b33b4cbf4`):
podman 5.7.0 refuses the
`ghcr.io/goauthentik/ldap:2026.8.0` tag because the tag resolves to a
*nested* manifest list (index → per-platform manifest lists) —
`unexpectedly received a manifest list instead of a manifest for a single
image`. Resolve once with `podman manifest inspect <image> | jq -r
'.manifests[] | select(.platform.architecture=="amd64") | .digest'` and pin
that digest.
The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
single-platform digest**
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
(`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26`, matching the
running image in `podman ps`, 2026-10-18). The digest in this page before 2026-10-18
(`…fa33bc8f…`) is superseded. The reason for pinning a digest at all: podman
5.7.0 refuses the plain tag because it resolves to a *nested* manifest list
(index → per-platform manifest lists) — `unexpectedly received a manifest list
instead of a manifest for a single image` — first hit on `:2026.8.0`, still the
rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
<image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") |
.digest'` and pin that digest.
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN; Jellyfin on
Dankserver binds `192.168.0.143:389`. The outpost token travels in
`AUTHENTIK_LDAP_OUTPOST_TOKEN` from the host `.env` (git-ignored).
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
(`PublishPort=389:3389`, `636:6636`,
`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
(2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN`
from the host `dankclaw/.env` (git-ignored).
### Authentik-side objects (provisioned via `ak shell` ORM)
Read from the live database (read-only `SELECT`s against the
`authentik-postgres` container, 2026-10-18):
| Object | Value |
|---|---|
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud` |
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` |
| Service account | `svc-jellyfin-ldap` — passwordless/unusable; authenticates by token |
| Bind token | intent **`verification`**, non-expiring — its key is the LDAP bind password |
| RBAC role | `svc-jellyfin-ldap-search` carrying object-level permission `authentik_providers_ldap.search_full_directory` on the provider |
| Groups | `jellyfin_users`, `jellyfin_admins` |
| Application | `jellyfin` (slug `jellyfin`) |
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — the page previously recorded `intent=verification`; the live value is `app_password` |
| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (the page previously said slug `jellyfin`, which does not exist) |
Hard-won rules, each verified live:
Hard-won rules (the first is now contested by the live database):
- **Token intent is load-bearing.** A bind token with `intent=app_password`
is rejected by the flow's password stage with `Invalid password`;
`intent=verification` passes. The outpost's direct binder executes the
provider's authorization flow, and that password stage only accepts
verification-intent tokens as static passwords.
- **Token intent: contested, do not trust either value blindly.** This page
long asserted that a bind token with `intent=app_password` is rejected by the
flow's password stage with `Invalid password` and only `intent=verification`
passes. The live token for `svc-jellyfin-ldap` is `app_password`
(`authentik_core_token`, 2026-10-18), so either the rule is wrong or the
current Jellyfin bind path is not that token. Nothing observable settles it
today: the LDAP outpost's retained log holds only its 5-minute
`initialized direct binder` / `Update providers` config cycles and no bind
activity *(unverified 2026-10-18: no bind was performed — reproducing it
requires the bind secret)*. Re-verify with a real bind before changing the
token's intent either way.
- **Search needs `search_full_directory`.** Without the RBAC grant the
outpost returns only the bind user itself — a `(memberOf=…)` filter then
matches zero entries and the plugin logs `Found no users matching X`.
@@ -236,19 +302,38 @@ Hard-won rules, each verified live:
and retest.
- `ak shell` import path for the provider model in 2026.8:
`from authentik.providers.ldap.models import LDAPProvider`.
*(unverified 2026-10-18: importing it outside a configured Django shell fails
with `ImproperlyConfigured`, and no `ak shell` was driven against the live
auth service for this check)*
## Configuration (secrets)
Two secrets live in the host `.env` (git-ignored; placeholders in
`.env.example`):
**Four** variables live in the host env file the units read,
`/home/dank/serverAppConfigs/dankclaw/.env` (git-ignored; the quadlets expand
`${…}` from it via `EnvironmentFile`):
- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
- `AUTHENTIK_OUTPOST_TOKEN` — proxy outpost token
(`dankclaw/authentik/quadlet/authentik-outpost.container:34`).
- `AUTHENTIK_LDAP_OUTPOST_TOKEN` — LDAP outpost token
(`…/authentik-outpost-ldap.container:29`).
Only the first two have placeholders in `dankclaw/.env.example`; the two outpost
tokens have no example entry (there is no repository-root `.env.example` —
`find`, 2026-10-18). Rotating `AUTHENTIK_SECRET_KEY` fleet-wide invalidates
every outpost session (see the ORM playbook).
## First-time setup
1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
health-gated), reloads Caddy.
1. `./scripts/deploy.sh dankclaw --service authentik-server` (any of
`authentik-postgres|authentik-server|authentik-worker` runs
`deploy_authentik`, `scripts/deploy-remote.sh:980-1004`) — installs the three
core quadlets, starts postgres health-gated (`wait_container_healthy
authentik-postgres 120`), then the server (300 s health wait) and the worker.
The outposts are separate handlers: `--service authentik-outpost` and
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
dispatch at `:1579-1588`).
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: **Directory →
@@ -257,20 +342,36 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
(`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
flows set (see the provisioning pitfall above). Attach the provider to the
embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
Connect** provider, create the app with the correct redirect URI, and paste
the `client_id` / `client_secret` into the app's SSO settings.
4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
the app with the correct redirect URI, and paste the `client_id` /
`client_secret` into the app's SSO settings. The relying parties that exist
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
**Gitea has no provider row at all** (live provider table, 2026-10-18) — it
authenticates itself and stays off the chain by design
(`dankclaw/Caddyfile:110-120`).
## Operations
- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
probes this).
- **Health:** there is **no** HTTP health path on this build —
`/-/health/`, `/-/live/` and `/api/-/health/` all answer **404**, on
`127.0.0.1:9000` and through `https://auth.franksdank.cloud` alike
(`curl`, 2026-10-18). The container's own probe is Django-side:
`HealthCmd=python /manage.py check --database default`
(`dankclaw/authentik/quadlet/authentik-server.container:36`), and
`podman inspect authentik-server` reports `healthy`. A usable liveness URL is
a flow page: `https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`
→ 200 (2026-10-18). The earlier "the server healthcheck probes `/-/health/`"
line was wrong in both halves.
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
truth). The `/home/dank/Containers/authentik/data` dir holds media + static files.
- **Upgrades:** bump the image tag in the three units under
`dankclaw/authentik/quadlet/`, run `./scripts/deploy.sh dankclaw`. Authentik
migrates the schema on startup.
truth). The `/home/dank/Containers/authentik/data` dir holds media + static
files (mounted by all three app units).
- **Upgrades:** bump the tag in **all three app units** under
`dankclaw/authentik/quadlet/` and re-deploy; `authentik-outpost` carries the
same tag (it runs the same image, `Exec=worker`) and `authentik-outpost-ldap`
is **digest-pinned**, so its pin must be re-resolved for the new version (see
the LDAP section) — bumping only the tag would leave the LDAP outpost behind.
Authentik migrates the schema on startup.
## Self-serve user-settings flow (password change)
@@ -374,8 +475,11 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
`podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
after the deploy prints its reload line; confirm the reload actually ran,
don't trust exit 0.
- Outpost reachability: `podman exec authentik-server getent hosts caddy`
resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
- Outpost reachability: `podman exec authentik-server getent hosts caddy` →
`10.89.1.8 caddy.dns.podman` (2026-10-18) — that is Caddy's address on the
`dankclaw_default` podman network, the alias `internal_host` dials. It is a
netavark allocation, not a pinned address and not a compose artifact (the
compose stack is gone); never hard-code it.
**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
000 (connection refused) — the post-auth listener lives in the caddy
@@ -403,22 +507,36 @@ EOF
- The heredoc IS the program: `ak shell` reads all of stdin and execs it
(heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
one).
- Bootstrap prints JSON log lines, then
`### authentik shell (2026.8.3)` and `229 objects imported automatically…`
— pipe through `sed -n '/229 objects/,$p'` to drop the noise.
- 229 models are pre-imported. Import paths (each wrong guess costs a failed
exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
`PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
- Bootstrap prints JSON log lines, then a shell banner and an
"`NNN objects imported automatically…`" line — pipe through
`sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
the exact banner text and the 229-object count — no `ak shell` was driven
against the live auth service for this audit)*.
- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
`Application`, `ApplicationEntitlement`, `PropertyMapping` →
`authentik.core.models`; `ProxyProvider` →
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
`LDAPProvider` → `authentik.providers.ldap.models`;
`authentik.core.models.application` does NOT exist as an import path; there
is no `authentik.policies.group` module in this build.
- For live DB truth prefer the postgres container
(`podman exec authentik-postgres psql -U authentik -d authentik -At -c
"select * from …"`) — **query with `select *` and stop guessing column
names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
FieldError — `type` moved to subclass tables; the outpost `enabled` column
may not exist under that name; provider pk is `oauth2provider_ptr_id`).
*(unverified 2026-10-18: these module paths were not re-executed — importing
outside a configured Django shell fails with `ImproperlyConfigured`)*
- For live DB truth prefer the postgres container, and it works without any
credential — the local socket is trusted, so a read-only audit is just
`podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
before guessing: 2026.x renamed both tables and columns, and the models are
MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
three `*_flow_id` columns; the proxy table is
`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
table in 2026.8), the LDAP table is
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
`authentik_core_application` attaches to a provider through `provider_id`
(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
is the same story from the ORM side: `type` lives in the subclass tables.
Never select secret columns (`client_secret`, `key`, `password`) into a log.
## User provisioning playbook (user-level account)
@@ -435,33 +553,42 @@ for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
u.groups.add(Group.objects.get(name=g))
```
`path` defaults to `users` (regular). The access map (verified live
2026-10-04): there are NO per-app policy bindings on the proxy providers
(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
access is decided entirely by GROUP MEMBERSHIP consumed downstream —
`path` defaults to `users` (regular). The access map (re-checked against the
live DB 2026-10-18): **every one of the nine proxy providers carries the same
authorization flow UUID**, `default-provider-authorization-implicit-consent`
(name "Authorize Application"), i.e. there are no per-app policy bindings to
delta, so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed
downstream —
- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
regex gate) and the homepage admin panel; regular users get only
`category: user` panels from `services.json`.
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
"Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
"Open WebUI roles" property mapping → `roles` claim. Verify live with
**"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
`Open WebUI` application (`authentik_core_applicationentitlement`,
2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
expression were not read; the mapping itself is not visible under that name
in `authentik_core_propertymapping`)*. Verify live with
`u.app_entitlements(app)`.
- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
(filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
every human account).
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
debo, freya, mitch, eric — mitch (pk 22) and eric (pk 23) created
2026-10-15, kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
`<user>@franksdank.cloud` + display name +
`is_active=True` + the password set via `set_password`, membership in
`Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was
recreated from scratch on 2026-10-08 into this shape (the old row had an
empty email and no `openwebui_users`; audit first — the row carried no MFA
device, token, entitlement, or proxy session, and events survive a user
delete because `Event.user` is a denormalized JSON field, not a FK).
debo, freya, mitch, eric). The pks in this page are confirmed against
`authentik_core_user` on 2026-10-18: ridley 13, kassie 14, alex 15,
artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
and every one carries the `<user>@franksdank.cloud` address the template
prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
*(unverified 2026-10-18)*. Template: `username` +
`<user>@franksdank.cloud` + display name + `is_active=True` + the password
set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
`openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
shape (the old row had an empty email and no `openwebui_users`; audit first —
the row carried no MFA device, token, entitlement, or proxy session, and
events survive a user delete because `Event.user` is a denormalized JSON
field, not a FK).
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
@@ -479,14 +606,24 @@ assert authenticate(request=None, username="ridley", password="nope") is None
Look flows up by slug, not name:
`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
"Welcome to authentik!" — `.get(name=…)` fails; both confirmed from
`authentik_flows_flow`, 2026-10-18). Stage order, read from
`authentik_flows_flowstagebinding` the same day:
`default-authentication-identification`(10) →
`default-authentication-password`(20) →
`default-authentication-mfa-validation`(30; no-op with no enrolled device) →
`default-authentication-login`(100). With no MFA enrolled, nothing else blocks.
More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
can be null (guard with `getattr`); `LDAPProvider` has
`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
`organization` — the backing table
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
(scopes are linked through `ProviderPropertyMapping`, which is why
`filter(provider=…)` on a scope mapping is empty); `Outpost.config` is an
`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
(use `type(p).__name__`); `User` has no `.status`.
@@ -536,3 +673,5 @@ podman run --rm --network container:<container> --entrypoint /bin/busybox \
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison
*(the file is gone from the repo and from the host checkout as of
2026-10-18)*
serverAppConfigs-docs/Infrastructure/Caddy.md
+30 -23
View File
@@ -9,7 +9,7 @@ timestamp: 2026-08-29
# Caddy
Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and gates the guarded vhosts through the two-tier chain — Anubis bot-challenge first, then Authentik — rather than through HTTP basic auth (`dankclaw/Caddyfile:14-24`). Basic auth survives on exactly one vhost today: the Pi-hole admin UI (`dankclaw/Caddyfile:166-171`).
**Docs:** https://caddyserver.com/docs
@@ -20,9 +20,9 @@ Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent q
| Runtime | Quadlet unit `caddy.service` (`dankclaw/caddy/quadlet/caddy.container`) — boot-persistent, self-healing |
| Image | `franksdank/caddy-porkbun:latest` (local build, `dankclaw/caddy/Containerfile`) |
| Config | `/home/dank/Containers/caddy/Caddyfile` (bind mount, rendered by deploy) |
| Data | `/home/dank/Containers/caddy/data` (certs) and `.../config` (ACME state) |
| Data | `/home/dank/Containers/caddy/data` — Caddy's whole store: `data/caddy/certificates/` (leaf certs) and `data/caddy/acme/` (the ACME account). `.../config` holds only `config/caddy/autosave.json`; it is not where ACME state lives (host `ls`, 2026-10-18) |
| Ports | 80, 443 on 192.168.0.143 |
| TLS | Let's Encrypt, DNS-01 via the Porkbun module (one cert per vhost) |
| TLS | Let's Encrypt, DNS-01 via the Porkbun module — one certificate per name, not a wildcard: 31 per-name directories under `/home/dank/Containers/caddy/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/` (host `ls`, 2026-10-18) |
| Runs on | Dankclaw only — Dankserver has no Caddy |
The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before start. See [[Deployment Flow]].
@@ -32,13 +32,13 @@ The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeh
Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").
- **Install path on Dankclaw:** `~/.config/containers/systemd/caddy.container` (unit `caddy.service`)
- **Network:** `dankclaw_default` (the compose bridge — the Caddyfile's container-name upstreams must keep resolving; if no compose service has created the network yet at boot, quadlet creates it first and compose reuses it by name later)
- **Network:** `dankclaw_default` — the long-lived bridge (`10.89.1.0/24`, gateway `10.89.1.1`, `podman network inspect dankclaw_default`, 2026-10-18) that the old compose stack created; the Caddyfile's container-name upstreams must keep resolving on it. `dankclaw/compose.yaml` no longer exists in the repo or on the host (`ls`, 2026-10-18), so from now on the quadlet units that name this network are the only thing that creates it at boot.
- **Published ports:** 80/tcp, 443/tcp
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy writes it from the checkout's `.env`; secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all (302) inside the container
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy creates it from the checkout's `dankclaw/.env` if it is missing and then fails if either variable is absent (`scripts/deploy-remote.sh:405-412`); secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all inside the container, which answers **308** (not 302) — `curl -sI http://127.0.0.1:80/` → `308`, `Location: https://www.franksdank.cloud/` (host, 2026-10-18; `dankclaw/Caddyfile:60-62`)
- **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
- **Enable state:** `systemctl --user is-enabled caddy` reports `generated` — that is the healthy state for a quadlet. The podman generator creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start (linger covers boot); an explicit `systemctl enable` fails on generated units, so the deploy does not run it
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs and starts the unit
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` → `deploy_caddy()` (`scripts/deploy-remote.sh:383-429`) renders the Caddyfile, rebuilds the image when the Containerfile hash changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), installs the quadlet, restarts the unit, and issues `podman exec caddy caddy reload` afterwards
## The porkbun image
@@ -57,7 +57,7 @@ acme_dns porkbun {
}
```
The deploy writes both variables from the host `.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `.env.example` (and `dankclaw/caddy/env.example`) hold placeholders. The Porkbun account holds only `franksdank.cloud`.
The deploy writes both variables from `dankclaw/.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `dankclaw/.env.example` and `dankclaw/caddy/env.example` hold placeholders (there is no repository-root `env.example` — `find`, 2026-10-18). The account holds `franksdank.cloud` and not `franksdank.casa` — that asymmetry is why `plex.franksdank.casa` stays plain HTTP (`dankclaw/Caddyfile:204-207`); whether it holds any other domain is not checkable from this repo *(unverified 2026-10-18)*.
## Vhost map
@@ -65,28 +65,32 @@ Every vhost uses the flat `franksdank.cloud` convention. The full list lives in
| Upstream | Reach |
|----------|-------|
| Dankclaw services (compose network) | by container name, e.g. `comfyui:8188` |
| Dankclaw quadlet services | by published LAN port, e.g. `192.168.0.143:3000` (Gitea) |
| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM) |
| Dankclaw services on `dankclaw_default` | by container name: `authentik-server:9000`, `anubis:8923`, `grafana:3000`, `prometheus:9090`, `loki:3100`, `seerr:5055` (`dankclaw/Caddyfile:94,75,376,381,366,371`) |
| Dankclaw quadlet services on isolated networks | by published LAN port: `192.168.0.143:3000` (Gitea), `:3005` (OpenWebUI), `:8081` (SearXNG via Gluetun), `:8082` (Marreta), `:8083` (Pi-hole), `:9119` (Hermes dashboard) |
| Host-network containers on Dankclaw | `host.containers.internal:8787` (Hermes WebUI) — a host-netns bind is not a published port, so netavark's host address is the only upstream that reaches it (`dankclaw/Caddyfile:179-188`) |
| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM), `:32400` (Plex), `:3923` (Copyparty) |
| FrankBook-Pro services | by published LAN port on `192.168.0.112` (the *arr stack and qBittorrent; those ports publish on that host's Gluetun netns owner) |
The bare-IP catch-all `:80` 302-redirects to `https://dankclaw.franksdank.cloud/`. The legacy name `dankserver.franksdank.cloud` 302-redirects there too. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.
ComfyUI is **not** in this table: it has no vhost. It publishes `0.0.0.0:8188` and is reached at `comfyui.franksdank.cloud:8188` through the Pi-hole A record, with no Caddy and no authentication on the LAN (`dankclaw/comfyui/quadlet/comfyui.container:24-30`).
The bare-IP catch-all `:80` 308-redirects to `https://www.franksdank.cloud/` (`dankclaw/Caddyfile:60-62`, status measured 2026-10-18), and so do the two legacy names `dankclaw.franksdank.cloud` and `dankserver.franksdank.cloud` (`dankclaw/Caddyfile:81-88`) — `dankclaw.` is a redirect target of the past, not of `:80`. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). The version line holds: the container on Dankserver is `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (`podman ps`, 2026-10-18). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.
## Basic auth
The Caddyfile in git carries a **bare** placeholder; deploy.sh substitutes the **full** hash from the host `.env` (`CADDY_BASICAUTH_HASH`):
```
chat.franksdank.cloud {
pihole.franksdank.cloud {
basicauth {
dank REPLACE_WITH_HASH
dankuser REPLACE_WITH_HASH
}
reverse_proxy 192.168.0.143:3005
reverse_proxy 192.168.0.143:8083
}
```
After rendering, the line reads `dank $2a$14$xxxx…` (one complete hash). Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password). `tests/deployment_flow.bats` pins the render contract.
After rendering, the line reads `dankuser <hash>` (one complete hash) — `tests/deployment_flow.bats:96-99` pins the `dankuser` name and the single-hash form, and `tests/caddyfile.bats:75` pins the bare `dankuser REPLACE_WITH_HASH` template. Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; `tests/caddyfile.bats:67-72` forbids the prefix). `tests/deployment_flow.bats` pins the render contract.
The homepage (and SearXNG, Hermes, Prometheus, RomM, Copyparty) use `forward_auth` (Authentik SSO) rather than basicauth; see [[Homepage]] and [[Authentik]]. The basicauth hash above is still rendered for the vhosts that keep it (Grafana, Pi-hole).
Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile:14-24,283-423`). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), and the media/*arr vhosts. Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik (`dankclaw/Caddyfile:122-132`). See [[Homepage]] and [[Authentik]].
## Operations
@@ -102,21 +106,24 @@ The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile ch
podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
```
Certificates and ACME state live in the caddy data volume. Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01.
Certificates and ACME state live in the caddy data volume (`data/caddy/`). Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01. The store also still holds leaf certificates for names the Caddyfile no longer defines — `sonarr.`, `chat.`, `comfyui.`, `ntfy.`, `jellyfin.` (`.franksdank.cloud`) each have a directory there (host `ls`, 2026-10-18) — retired vhosts leave certificates behind, which is harmless but is the record that those names once existed.
## Gotchas
- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit — do not re-add it to `compose.yaml`.
- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit, and the repo has no `dankclaw/compose.yaml` left to re-add it to (`ls`, 2026-10-18).
- `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
- A new vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]].
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]].
- A public vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]]. The deliberate exception is an **internal-only** name: `hermes.franksdank.cloud` and `hermesadmin.franksdank.cloud` are in `addn-hosts` and the Caddyfile but have no public record, which is what keeps them LAN-only — DNS-01 issuance needs no public A record (`dankclaw/Caddyfile:173-179`).
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]]. *(unverified 2026-10-18: hairpin behaviour is only observable from a WAN-side client; nothing on the LAN or in this repo can prove it)*
- Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
- `transport http {versions h2c 1.1}` on a vhost makes Caddy send an h2c
upgrade preface; .NET/Kestrel backends (Jellyfin) reject it and Caddy turns
the failure into a 502 while the same port answers 200 over plain HTTP/1.1.
Match the transport to what the backend speaks: Jellyfin = default HTTP/1.1,
Plex PMS = TLS (PMS drops plain HTTP off-loopback, see the test suite).
Match the transport to what the backend speaks: Jellyfin = default HTTP/1.1
(`dankclaw/Caddyfile:248-251` carries no transport block), Plex PMS = TLS
(`dankclaw/Caddyfile:221-227`; PMS drops plain HTTP off-loopback — the
`tls_insecure_skip_verify` hop is measured in `tests/caddyfile.bats`, and no
vhost in the current Caddyfile sets `versions h2c`).
## See Also
serverAppConfigs-docs/Infrastructure/Grafana.md
+29 -18
View File
@@ -18,11 +18,11 @@ Grafana visualizes the metrics that Prometheus collects from every service. It r
| Item | Value |
|------|-------|
| Image | `docker.io/grafana/grafana:latest` |
| Published | `127.0.0.1:3002` → 3000, loopback only (see [[Observability]] → Exposure) |
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service` |
| Data | `/opt/grafana/data` — `grafana.db`: accounts, sessions, anything created in the UI |
| Provisioning mount | `/opt/grafana/provisioning` (read-only) — **empty as of 2026-09-11** |
| Datasource | **none configured** — see the state note in [[Observability]] |
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |
## Configuration
@@ -36,25 +36,33 @@ EnvironmentFile=/home/dank/Containers/grafana/env # systemd-side only
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
```
`seed_grafana_env` in `scripts/deploy-remote.sh` seeds that file from
`dankclaw/.env` with a single-variable filter and fails the deploy when the
value is missing. The container receives the one expanded value, never the file.
`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.
## Datasource
None is configured. Grafana cannot query anything yet; pointing it at
`http://prometheus:9090` and `http://loki:3100` (by container name, on the
`dankclaw_default` network) is outstanding work — see the state note in
[[Observability]].
The Prometheus datasource is declared in git, not clicked into `grafana.db`:
`shared/grafana/provisioning/datasources/prometheus.yml` points at
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
address Caddy proxies to, never the loopback-published port — and carries
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
alongside it by `dashboards/provider.yml` (provider `fleet`,
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
A Loki datasource is still **not** provisioned — the tree contains exactly one
datasource file — so nothing in Grafana queries logs today, which matters less
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
Verification is through the API, not the filesystem:
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.
## Healthcheck
Grafana responds on `/api/health` with 200 and `"database":"ok"`.
Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).
## Gotchas
- Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
no provider token (`token=None`), so no identity header ever reaches Grafana;
RP mode is topology-independent. The full verified contract (provider pk 9,
@@ -62,10 +70,13 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`.
Basic client auth, form-encoded token body, and the fork's userinfo-claims
bug that leaves identity at the `sub` hash only) is in
[[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana:
users auto-upsert as Viewer with `login = <sub hex>`; no email/groups/role
claims reach Grafana until the evaluator bug is fixed — do not write role
mappings that depend on those claims. The local admin account stays the only
Admin.
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
users land as Viewer with `login = <sub hex>`; no email/groups/role claims
reach Grafana until the evaluator bug is fixed — do not write role mappings
that depend on those claims. The local admin account stays the only Admin.
(The Viewer-default outcome and the `<sub hex>` login are not readable from
the host without the database or an admin password. *(unverified 2026-10-18)*)
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
for the service, podman then looks for its configuration under `/root`,
serverAppConfigs-docs/Infrastructure/Loki.md
+33 -17
View File
@@ -9,7 +9,7 @@ timestamp: 2026-09-11
# Loki
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs. It replaces the need for a full ELK stack.
Loki is the log aggregation backend of the observability stack. It is configured to store and query container and system logs, replacing the need for a full ELK stack — and as measured on 2026-10-18 the store is **empty**: the service is healthy and queryable, and it holds zero streams. See "Contents" below.
**Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest
@@ -18,43 +18,59 @@ Loki is the log aggregation backend of the observability stack. It stores and qu
| Item | Value |
|------|-------|
| Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) |
| Published | `127.0.0.1:3100` → 3100, loopback only |
| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service` |
| Mode | single-binary (all components in one process) |
| Storage | filesystem, `/opt/loki/data` |
| Auth | `auth_enabled: false` (single-tenant homelab) |
| Schema | v13 / TSDB |
| Retention | 30 days, enforced by the compactor |
| Published | `127.0.0.1:3100` → 3100, loopback only (`dankclaw/observability/quadlet/loki.container:43`; `ss -ltnp` 2026-10-18 shows no other bind) |
| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`loki.container:52-57`), installed at `~/.config/containers/systemd/loki.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active loki` 2026-10-18) |
| Mode | single-binary (all components in one process — the `Exec` line passes no `-target`, `loki.container:42`) |
| Storage | filesystem, `/opt/loki/data` → `/loki` (`loki.container:45`; `loki-config.yaml:10-16`) |
| Auth | `auth_enabled: false` (single-tenant homelab, `loki-config.yaml:4`) |
| Schema | v13 / TSDB (`loki-config.yaml:22-30`) |
| Retention | 30 days, enforced by the compactor (`shared/loki/loki-config.yaml:33` `retention_period: 720h`, `:37-44` `retention_enabled: true`) |
| Contents (2026-10-18) | **zero streams** — see "Contents" |
## Configuration
`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy. Key settings:
`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy (`scripts/deploy-remote.sh:276-279`). Key settings:
- `schema_config` v13 with `store: tsdb` and a filesystem object store.
- `compactor` with `retention_enabled: true` so logs do not grow without bound.
- `replication_factor: 1` for a single instance.
- `schema_config` v13 with `store: tsdb` and a filesystem object store (`loki-config.yaml:22-30`).
- `compactor` with `retention_enabled: true` so logs do not grow without bound (`loki-config.yaml:37-44`).
- `replication_factor: 1` for a single instance (`loki-config.yaml:16`).
## Volumes
| Container path | Host path |
|----------------|-----------|
| `/loki` | `/opt/loki/data` |
| `/etc/loki` | `/opt/loki/config` |
| `/loki` | `/opt/loki/data` (`loki.container:45`) |
| `/etc/loki` | `/opt/loki/config` (`loki.container:44`) |
## Healthcheck
Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile` as an exec-form `HEALTHCHECK` instead (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it).
Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile:29-30` as an exec-form `HEALTHCHECK` instead, probing `http://localhost:3100/ready` (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it: `dankclaw/loki/Containerfile:13-17`, `scripts/deploy-remote.sh:558-562`). The deployer waits on the same endpoint through the published loopback, `curl http://127.0.0.1:3100/ready` (`deploy-remote.sh`, the `loki)` case of `deploy_observability`), and `podman ps` reports the container `healthy` (2026-10-18).
## Log Shipping
The [[Alloy]] agent on Dankclaw ships container and journal logs to `http://loki:3100/loki/api/v1/push` — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge).
The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge). **Nothing arrives today:** Alloy's own counters read zero lines read and zero files tailed, with both file globs refused at start (`journalctl --user -u alloy` 2026-10-18); see [[Alloy]] → "Delivery state".
## Contents (2026-10-18): healthy, empty
The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
`chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
`find /opt/loki/data -type f` counted **8 files**, all WAL segments,
cluster-seed and delete-request bookkeeping, **no chunk objects**
(2026-10-18; the byte total drifts as the WAL rolls, so it is not a
useful invariant — the file census is).
- Prometheus-side health is green: the `loki` job scrapes `loki:3100/metrics` and reports up (`shared/prometheus/prometheus.yml:57-61`), and the homepage card probes `127.0.0.1:3100` (`shared/homepage/services.json:207`).
So "30-day retention" is a policy that has never had anything to delete, and no page should describe log history as available. The cause is upstream of Loki — the shipper ships nothing — and no fix is claimed here; the check that matters is a label query, not `/ready`.
## Gotchas
- Filesystem storage grows; keep compactor retention on.
- Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
- The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
- Nothing queries Loki yet: Grafana has no datasource configured. See the state note in [[Observability]].
- Nothing queries Loki: Grafana's only provisioned datasource is Prometheus (`shared/grafana/provisioning/datasources/prometheus.yml:14-21`, uid `prometheus`), and there is no Loki datasource file in that tree. Adding one is a YAML file, not a UI click — see [[Grafana]] → Datasource.
## See Also
serverAppConfigs-docs/Infrastructure/Node-Exporter.md
+13 -11
View File
@@ -10,27 +10,29 @@ timestamp: 2026-09-11
Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container.
**Hosts:** Dankclaw (unit) + Dankserver (compose service) — **Docs:** https://prometheus.io/docs/guides/node-exporter
**Hosts:** Dankclaw (unit) + Dankserver (unit) — **Docs:** https://prometheus.io/docs/guides/node-exporter
Both hosts run the same shape as a Quadlet unit: `dankclaw/observability/quadlet/node-exporter.container` and `dankserver/observability/quadlet/node-exporter.container` (the compose-only Dankserver copy died for a day and a half before anyone noticed, so it became a unit on 2026-09-16). Both report `generated` + `active` on the host and bind their own two addresses (`systemctl --user is-enabled|is-active node-exporter`, `ss -ltnp` on dankserver, 2026-10-18); `tests/observability_quadlet.bats:419` pins the Dankserver side to a unit, not compose.
## Facts
| Item | Value |
|------|-------|
| Image | `prom/node-exporter:latest` |
| Listen addresses | `127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw) |
| Network | `Network=host` + `--pid=host` |
| Listen addresses | `127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw, `ss -ltnp` 2026-10-18; Dankserver binds `127.0.0.1:9100` + `192.168.0.235:9100`) |
| Network | `Network=host` + `--pid=host` (`dankclaw/observability/quadlet/node-exporter.container:37-38`) |
| Unit | `dankclaw/observability/quadlet/node-exporter.container` → `node-exporter.service` |
| Paths | `--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` |
| Paths | `--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` (`node-exporter.container:39`) |
| Auth | none — it is an unauthenticated metrics endpoint by design |
## Why it is the one LAN listener
Every other observability service publishes on loopback. This one cannot:
Every other observability service publishes a port on loopback. This one runs in the host network namespace instead, and netavark DNATs only *published* ports — so a bridge container cannot reach a host-netns bind at the host's LAN address, and the exporter has to be reachable at an address the consumers can actually dial:
- **Prometheus** runs in a container on the `dankclaw_default` network and cannot reach the host's loopback address, so it needs the host's LAN IP; `shared/prometheus/prometheus.yml` scrapes `192.168.0.143:9100` under the `node-dankclaw` job.
- **The homepage status timer** runs on the host and probes `127.0.0.1:9100`.
- **Prometheus** runs in a container on `dankclaw_default` and scrapes `host.containers.internal:9100` (`shared/prometheus/prometheus.yml:37-39`) — the gateway address netavark delivers a bridge container's host-bound dial to. It does **not** scrape `192.168.0.143:9100`; that address is refused by construction from this container, the same rule that sends `frank-dragoon` and `frank-zealot` to the gateway. `node-dankserver` is a different machine, so its LAN address is used (`prometheus.yml:41-43`).
- **The homepage status timer** runs on the host and probes `127.0.0.1:9100` (`shared/homepage/services.json:220`).
The unit therefore binds exactly those two addresses with two `--web.listen-address` flags — never the wildcard. It has no authentication, so it is never vhosted and never published beyond the LAN.
The unit therefore binds exactly those two addresses with two `--web.listen-address` flags (`node-exporter.container:39`) — never the wildcard: `ss -ltnp` (2026-10-18) shows `127.0.0.1:9100` and `192.168.0.143:9100` and nothing else. The LAN bind stays in the unit because LAN machines scrape it directly and the seam test pins it, not because this Prometheus needs it. It has no authentication, so it is never vhosted and never published beyond the LAN.
## Mounts
@@ -46,14 +48,14 @@ The mount paths and the `--path.*` flags are a pair: changing one without the ot
| Host | Target |
|------|--------|
| Dankclaw | `192.168.0.143:9100` |
| Dankclaw | `host.containers.internal:9100` (same-host bridge container → the gateway, `prometheus.yml:39`; live target `http://host.containers.internal:9100/metrics` is up, `curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18) |
| Dankserver | `192.168.0.235:9100` |
The scrape jobs live in `shared/prometheus/prometheus.yml`; see [[Prometheus]]. The `tests/observability_quadlet.bats` seam pins the Dankclaw address to this unit's listen address so the two cannot drift apart.
The scrape jobs live in `shared/prometheus/prometheus.yml`; see [[Prometheus]]. The `tests/observability_quadlet.bats` seam pins both halves: the unit's two `--web.listen-address` values (`:103-104`) and the `node-dankclaw` target string `"host.containers.internal:9100"` (`:108-131`), so the exporter and its only Prometheus consumer cannot drift apart.
## Gotchas
- Host networking is required (or a published port); the deploy's health wait probes `127.0.0.1:9100/metrics`.
- Host networking is required (or a published port); the deploy's health wait probes `127.0.0.1:9100/metrics` (`scripts/deploy-remote.sh:536`) and the unit's `HealthCmd` the same address from inside the host namespace (`node-exporter.container:43`).
- The `/proc`, `/sys` and root mounts must be read-only.
- Each host's instance reflects only its own host; do not try to share one.
- Collectors are the image defaults — no per-collector tuning and no textfile collector for custom metrics.
serverAppConfigs-docs/Infrastructure/Observability Provisioning.md
+110 -38
View File
@@ -21,37 +21,61 @@ committed.
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
Targets and datasources reference **container names** (`loki:3100`,
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
are host-netns services, not containers. Prometheus 3 refuses HTML responses
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
exporter (see [[Stale Signal Audit]]).
Targets and datasources reference **container names** (`grafana:3000`,
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
loopback-published ports; `alloy` is **not scraped at all** (no job in the
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
`shared/homepage/services.json:233`). The LLM endpoints split by topology: the
two same-host ones (`frank-dragoon` `:7770`, `frank-zealot` `:7771`) are
scraped as `host.containers.internal:<port>` because a bridge container cannot
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
[[Stale Signal Audit]]).
## The deploy path
`scripts/deploy-remote.sh` carries two copier functions:
`scripts/deploy-remote.sh` carries one copier per config surface, all called
from `deploy_observability`:
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
- `copy_prometheus_config` → `/opt/prometheus/config/prometheus.yml` **and** `config/rules/llm-fleet.yml` (`:219-227`)
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
`shared/grafana/**` to the `prometheus`/`grafana` services, so
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
exist, the copier functions exist, and the provisioned paths match the units'
mounts.
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
`shared/grafana/**` to `grafana` (`:261-264`); changes under
`dankclaw/observability/` select the touched unit and its dependency consumer
(`:321-328`), and a change to either relay file redeploys `alertmanager`
(`:399-402`). So `deploy.sh dankclaw --service grafana` (etc.) restarts only
what the change touched. Seam tests in `tests/observability_quadlet.bats` pin
the unit files exist and are boot-persistent/self-healing (`:36-62`), that no
unit publishes on a wildcard (`:63-81`), that the copiers install each unit
(`:250`), that grafana provisioning is declared in git and installed
(`:345`), and that the provisioned paths match the units' mounts.
## Entry points
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
vantage first: container name if the exporter is a container on the same
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
if it is an HTTP probe, not a metrics endpoint.
vantage first: container name if the exporter is a container on
`dankclaw_default`; `host.containers.internal:<port>` if it is a process in the
host's own network namespace **on this same machine** (a bridge container
cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only
if it is a process on a **different** machine; a blackbox module if it is an
HTTP probe, not a metrics endpoint.
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
Rules reload with the prometheus service.
Current delivered set: **4 alert rules + 2 recording rules** —
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
**Add a datasource** — drop a YAML file into
`shared/grafana/provisioning/datasources/`. Point it at the container name of
@@ -60,7 +84,9 @@ the backend (`http://prometheus:9090`), not loopback.
**Add a dashboard** — drop the JSON into
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
unified storage, not (only) the SQLite table — grep the API, not the DB, when
verifying (see the verify section of [[Fleet Audit and Posture]]).
verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
2026-10-18: the storage-backend claim needs an admin password to confirm
against `grafana.db` and the API side by side)*
## Verify through the APIs, not the filesystem
@@ -73,34 +99,80 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
```
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and `up` for
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
listed in [[Prometheus]]. The Grafana query needs the local admin password and
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
proof of installation only — see the next paragraph.
Health answers from a stale process and provisioned resources can hide in
unified storage — a file existing on disk is not proof the running server
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
table (refused = outage, 404/HTML = config defect).
## Current delivered state (2026-09-17)
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
via `host.containers.internal`) → signal-cli. Some targets read down for
via `host.containers.internal`) → signal-cli, addressed by number
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
`curl -s http://192.168.0.143:18093/health` → `ok`, and
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
the blackbox exporter. See [[Observability]].
the blackbox exporter. See [[Prometheus]] → Targets.
**A llama.cpp endpoint is invisible unless it opts in.** `up{job="frank-zealot"}`
has been 0 for the entire retained history of the metric — the oldest sample,
2026-10-11, is already 0 — while the service itself answered `/health` 200 the
whole time. Cause: llama.cpp answers **501 Not Implemented** on `/metrics` unless
the server is started with `--metrics`, so the scrape was refused at the protocol
layer and `LlmEndpointDown` (critical) sat active across four payload generations.
Fixed 2026-10-18 by the `METRICS=1` knob in `scripts/frank-zealot.env`. Two laws
fall out of it: a target can read down beside a healthy service, so read
`lastError` from `/api/v1/targets` before believing an outage; and Alertmanager
runs here without persistence, so every restart re-stamps `startsAt` on a
long-standing alert — a "fresh" alert after a reboot is not evidence of a fresh
fault. Remaining known downs are the structural ones above (HTML-scraped probes,
Gitea metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
`frank-dragoon` has always been green.
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented**
on `/metrics` unless the server is started with `--metrics`, so the scrape was
refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing:
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
How many payload generations it spanned is not readable from any host
artifact. *(unverified 2026-10-18)*
Two laws fall out of it. A target can read down beside a healthy service, so
read `lastError` from `/api/v1/targets` before believing an outage. And rule
state lives in Prometheus' process, not on disk — a restart re-stamps
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
`--storage.path=/alertmanager` on a host volume keeps silences and the
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
`/opt/alertmanager/data`.)
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
`frank-dragoon` has always been green. The active-alert count is a
moment-in-time value and should never be documented as a constant:
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
minutes later, and was empty again by 11:56Z; the relay logged the matching
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
window on the 3090 — the repo Benchmarking Policy says stop the production
service before benching — and `:7771` answered from a manually booted candidate
arm (`systemctl is-active frank-zealot` → `failed` after
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
`TimeoutStopSec`, the known shutdown shape for these engines). So
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408
PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract.
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
reinstalls and starts it, restoring boot persistence and self-healing. Never
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
changed between two censuses ten minutes apart.
## Related
serverAppConfigs-docs/Infrastructure/Observability.md
+109
View File
@@ -5,3 +5,112 @@ description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Al
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-09-11
---
# Observability on Dankclaw
## Summary
Six containers and one host-side process: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
carrying every firing group to Signal. All of it is Quadlet-managed,
boot-persistent and loopback-only, deployed by
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
end to end; the log half is configured, healthy, and carries no data.
This body was rewritten on 2026-10-18 from live evidence: the previous version
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
compliance pass"), while sibling pages kept pointing at a "state note" on this
page that no longer existed. Every statement below carries the command or file
that proves it today.
## The stack
| Service | Container | Published | Unit | Page |
|---|---|---|---|---|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
Every unit lives in `dankclaw/observability/quadlet/`, carries
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
the same day, and `homepage-status.timer` — the root unit that probes all of
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
Dankserver runs one member of this family, node-exporter, as its own unit
(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
## Exposure
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
inside, and several carry lifecycle or query APIs. Browser access goes through
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
`anubis:8923`, and the post-auth handlers route by container name on
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see [[Node-Exporter]].
The relay's bind is the one address a bridge container can reach on the host:
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
and the podman bridge subnet `10.89.0.0/16` before the DROP
(`scripts/deploy-remote.sh:255-262`).
## Deployment
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
which installs the unit, copies that service's config
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
`copy_alloy_config`, `copy_grafana_provisioning`, plus
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
`grafana` after `prometheus`, `alloy` after `loki`
(`grafana.container:32-33`, `alloy.container:38-39`). See
[[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs.
## Current state (2026-10-18)
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
eight up targets; the four downs are structural (the `comfyui` and `services`
HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
4 alert rules + 2 recording rules (`/api/v1/rules`, matching
`shared/prometheus/rules/llm-fleet.yml`).
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
relay, and `journalctl --user -u signal-relay` records successful
`sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** One provisioned Prometheus datasource (uid
`prometheus`) and the `llm-fleet` dashboard, both installed from
`shared/grafana/provisioning/`.
* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
(8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
no label names.
See [[Alloy]] → Delivery state and [[Loki]] → Contents.
* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
structural downs have no real uptime signal), no Alertmanager vhost.
## Citations
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
serverAppConfigs-docs/Infrastructure/Prometheus.md
+32 -23
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists."
description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the provisioned Grafana datasource. Down targets today are structural, not outages."
tags:
- infrastructure
- prometheus
@@ -9,7 +9,7 @@ timestamp: 2026-09-21
# Prometheus
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.
Prometheus collects metrics from the services listed below by scraping their `/metrics` endpoints, and it is the datasource behind the provisioned `llm-fleet` dashboard in [[Grafana]] (datasource uid `prometheus`, `shared/grafana/provisioning/datasources/prometheus.yml:14-21`). node-exporter supplies host metrics from both machines.
**Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis →
Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
@@ -20,12 +20,12 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| Item | Value |
|------|-------|
| Image | `docker.io/prom/prometheus:latest` |
| Published | `127.0.0.1:9090` → 9090, loopback only (see [[Observability]] → Exposure) |
| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service` |
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
| Data | `/opt/prometheus/data` (TSDB) |
| Retention | `--storage.tsdb.retention.time=30d` |
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see [[Alertmanager]]) |
| Published | `127.0.0.1:9090` → 9090, loopback only — never a wildcard (`dankclaw/observability/quadlet/prometheus.container:50`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9090` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`prometheus.container:59-64`), installed at `~/.config/containers/systemd/prometheus.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active prometheus` 2026-10-18) |
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
## Targets
@@ -37,16 +37,25 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) |
**Three jobs read down, and they are not outages** (audited 2026-09-16):
The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18).
| Job | Target | Why it never reports up |
|-----|--------|-------------------------|
| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" |
| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape |
| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` |
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
| Job | Target | `health` | `lastError`, verbatim |
|-----|--------|----------|------------------------|
| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service.
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
A scrape cannot express "is this HTTP service answering?" — that is what a
probe is for, and the remedy is the **blackbox exporter** in probe mode
@@ -61,23 +70,23 @@ URLs.
| Container path | Host path |
|----------------|-----------|
| `/etc/prometheus` | `/opt/prometheus/config` |
| `/prometheus` | `/opt/prometheus/data` |
| `/etc/prometheus` | `/opt/prometheus/config` (`prometheus.container:51`) |
| `/prometheus` | `/opt/prometheus/data` (`prometheus.container:52`) |
## Healthcheck
Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on.
Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on (`prometheus.container:53`, `scripts/deploy-remote.sh` — the `prometheus)` case of `deploy_observability` waits on `http://127.0.0.1:9090/-/healthy`).
## Gotchas
- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `192.168.0.143:9100`, which is exactly the LAN address `node-exporter.container` binds (`Network=host`, `--web.listen-address` twice). Change one without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats` seam pins the pairing.
- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `host.containers.internal:9100` (`prometheus.yml:37-39`), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; `node-exporter.container:39` binds `127.0.0.1:9100` + `192.168.0.143:9100`. Change one side without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats:108-131` seam pins the pairing in both directions.
- The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently.
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Set a retention policy or the data volume grows without bound (30 days here).
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`), which is why this API must never be published beyond loopback.
- Set a retention policy or the data volume grows without bound (30 days here — `--storage.tsdb.retention.time=30d`, `prometheus.container:49`).
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`, `prometheus.container:49`), which is why this API must never be published beyond loopback.
## See Also
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16)
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16, commit `6a19143`, installed on the host by `scripts/deploy-remote.sh:286-300`)
- [[Node-Exporter]] — the host metrics source
- [[Observability]] — the whole stack, including what is still unconfigured
serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+41 -17
View File
@@ -16,9 +16,19 @@ priority and the exact action.
Dates: researched 2026-08. Hardware is verified on the live machine unless
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
**Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
iGPU behind :7770 — is retired. Dankclaw now serves
[[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2, port
7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on
the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18
03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18
([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of
the era they were taken in; the `ttm` ceiling and the NVMe scheduler rows were
re-checked 2026-10-18 and still hold.
## Corrections (re-verified 2026-08-15/16)
- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
- **Backend**: that era's production was **Vulkan0**, not ROCm. Measured on this box
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
@@ -26,9 +36,12 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). The law
still holds on `frank-zealot` (`CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` —
`scripts/frank-zealot.env:55-56`).
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. Today `frank-zealot`
is configured with speculation **off** (`SPEC_TYPE=none` — `scripts/frank-zealot.env:67`).
- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and
persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that
once held the service at ~15 GiB is gone.
@@ -39,21 +52,31 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
| ------ | ----- | ------ |
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` |
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` |
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve |
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
The OS accounts only ~30.5 GiB physical RAM. The TTM GPU-addressable pool
was capped at ~15.24 GiB (the real pre-2026-08-25 ceiling); that cap is now
lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
different, driver-reported figure that has always been larger than the kernel
cap.
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT). The OS now accounts
**121 GiB** (`MemTotal 127108080 kB`, 2026-10-18) and `rocm-smi
--showmeminfo vram` reports **2.0 GiB** (2,147,483,648 B) of dedicated VRAM —
the carve is gone and the GPU budget is the GTT aperture, not the SMI figure
([[Dankclaw System Specs]]).
The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and
`/etc/modprobe.d/ttm.conf` still carries the `options ttm` line (both read
2026-10-18).
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-dragoon-benchmark]].
## 1. Self-Hosted Agentic Tools
All of these connect to the local llama.cpp OpenAI-compatible endpoint
(`http://localhost:7770/v1`). None need NVIDIA.
All of these connect to the local OpenAI-compatible endpoint
(`http://127.0.0.1:7770/v1`). Today that port is **halogen-flash-server**, not a
llama.cpp server, and the address must be the IPv4 literal: `curl
http://localhost:7770/v1/models` returns nothing while
`http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18 — the rootless
podman publish is IPv4-only). None need NVIDIA.
| Tool | Use | Endpoint config | Priority |
| ---- | --- | --------------- | -------- |
@@ -67,7 +90,8 @@ All of these connect to the local llama.cpp OpenAI-compatible endpoint
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
with the current llama.cpp server and need no extra GPU.
with the endpoint on 7770 and need no extra GPU. The 7770 endpoint today is
halogen-flash-server, not llama.cpp.
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
@@ -90,8 +114,8 @@ All require root and a reboot for the modprobe change.
| Setting | Action | Effect | Priority |
| ------- | ------ | ------ | -------- |
| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done |
| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
| Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ |
| NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done |
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
@@ -99,8 +123,8 @@ All require root and a reboot for the modprobe change.
- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises,
change the env tunable.
- Consider switching the disabled `vllm-thinkingcap.service` unit off or
removing it to avoid confusion (it is loaded in a failed state).
- The `vllm-thinkingcap.service` unit this item told you to clean up is gone:
`systemctl is-enabled vllm-thinkingcap.service` → `not-found` (2026-10-18).
- Profile generation under 3-slot concurrent load before touching boot
settings, so any tuning is measured against the real workload.
serverAppConfigs-docs/LLM/Dankclaw System Specs.md
+25 -24
View File
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
## Server Overview
Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool.
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[ciru-activefpx-promptforge]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
## Hardware Specifications
### CPU
@@ -22,7 +22,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
| Cores | 16 (1 socket, 1 thread per core) |
| Max MHz | 5187.50 |
| Min MHz | 625.00 |
| NUMA Nodes | 1 (node 0, 31213 MB) |
| NUMA Nodes | 1 (node 0, 124129 MB — `Node 0 MemTotal: 127108080 kB`, read 2026-10-18) |
### GPU (Integrated)
@@ -31,8 +31,8 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
| Model | AMD Radeon 8060S (Strix Halo) |
| Device ID | 0x1586 |
| Architecture | gfx1151 (Strix Halo) |
| GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `gtt_total`; rocm-smi's smaller SVM carve-out is not the budget) |
| Physical RAM | 64 GiB (61 GiB visible; upgraded from 32 GB, corrected 2026-08-17) |
| GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `mem_info_gtt_total` = 122,402,963,456 B read 2026-10-18; rocm-smi's smaller SVM carve-out is not the budget) |
| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ([[ling-3.0-flash-trial]] §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
### NPU
@@ -41,12 +41,12 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
| Model | AMD XDNA2 (Ryzen AI) |
| Rating | 50 TOPS |
| Device | /dev/accel/accel0 |
| Driver | amdxdna 0.7 (in-tree, kernel 7.0.0-29-generic) |
| Firmware | 1.1.2.65 |
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
| Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) |
| Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) |
| Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] |
The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-zealot-gemma4-12b]]).
The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot-llama-prism-bonsai2-pq20]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot-gemma4-12b]], 2026-09-17 → 10-01).
### ROCm
@@ -54,7 +54,7 @@ The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B Q
|----------|-------|
| ROCk Module | Loaded |
| Runtime Version | 1.18 |
| ROCm Version | 7.2.4 system (/opt/rocm) + TheRock 7.15 SDK in the CIRU venv (used for the runtime) |
| ROCm Version | 7.2.4 system (`/opt/rocm/.info/version` = 7.2.4, read 2026-10-18). The TheRock 7.15 SDK that lived in the CIRU venv left the serving path when CIRU was retired 2026-08-22 |
| Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
| GPU Arch | gfx1151 (Strix Halo) |
| GPU Global Memory | ~114 GiB GTT aperture (the budget) |
@@ -63,22 +63,22 @@ The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B Q
| Property | Value |
|----------|-------|
| System RAM | 64 GiB (shared CPU + GPU; 61 GiB visible) |
| GPU-Visible (ROCm) | ~114 GiB GTT aperture (the budget) |
| Swap | 8.0 GiB |
| /dev/shm | ~15 GiB (half of RAM) |
| System RAM | 121 GiB (shared CPU + GPU; `free -g` total 121, `MemTotal 127108080 kB`, 2026-10-18) |
| GPU-Visible (ROCm) | ~114 GiB GTT aperture (`mem_info_gtt_total` 122,402,963,456 B — the budget) |
| Swap | 64 GB `/swap.img` (`swapon`, 2026-10-18; `free` reports 63 GiB) |
| /dev/shm | 61 GiB tmpfs (half of RAM, `df /dev/shm`, 2026-10-18) |
### Storage
| Filesystem | Size | Used | Avail | Use% |
|------------|------|------|-------|------|
| / (nvme0n1p2) | 1.9T | 1.3T | 545G | 70% |
| / (nvme0n1p2) | 1.9T | 1.4T | 424G | 77% |
### Network
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carries the RPC path to Dankserver since 2026-08-17. Default route via 192.168.0.1. |
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carried the RPC path to Dankserver during the 2026-08-17 pipeline-parallelism trial; no RPC listener runs today (`ss -ltn` shows no :50052, 2026-10-18). Default route via 192.168.0.1. |
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. |
| docker0 | bridge | - | 172.17.0.1 | Usually down. |
@@ -92,19 +92,20 @@ cable between claw and server is required for 10G.
| Property | Value |
|----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-29-generic |
| Python | 3.14.4 (system), 3.12.13 (CIRU venv) |
| CIRU runtime | ciru-ai/ROCmFPX v2.3 (llama-server, ROCm 7.15 SDK) |
| uv | 0.12.2 |
| Kernel | 7.0.0-38-generic (`uname -r`, 2026-10-18) |
| Boot line | `iommu=pt` since 2026-10-18 (`/proc/cmdline`, no `amd_iommu` token); eras before: `amd_iommu=off` 2026-09-11 → 10-18, IOMMU default-on before that — [[dankclaw-iommu-translation-off]] |
| Python | 3.14.4 (system, `python3 --version`, 2026-10-18). The 3.12.13 CIRU venv left with that runtime (retired 2026-08-22) |
| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[ciru-activefpx-promptforge]]); not the serving runtime |
| uv | 0.12.8 (`~/.local/bin/uv --version`, 2026-10-18) |
## Runtime Relevance
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-dragoon` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-dragoon-gufo-qwen38-27b]].
- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon-flash-server-v2-native]]) — and `frank-zealot` on 7771 ([[frank-zealot-llama-prism-bonsai2-pq20]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon-gufo-qwen38-27b]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 1.1 TB free disk space is ample for model storage.
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
## See Also
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+61 -40
View File
@@ -15,10 +15,13 @@ timestamp: 2026-09-20
## Server Overview
Dankserver is a media/library server and the `frank-medium` LLM host. The
host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and replaced with
two AMD Radeon AI PRO R9700 cards.
Dankserver is a media/library server and the `frank-archon` LLM host (radiance
1.0.13, 2×R9700 tensor-parallel — [[frank-archon-radiance-qwen38-iq4r]]; the
`frank-medium` name this page carried was retired 2026-09-20, see §Active
Services). The host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that
served the FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and
replaced with two AMD Radeon AI PRO R9700 cards (`lspci | grep -ci nvidia` → 0,
read 2026-10-18).
## Hardware Specifications
@@ -28,11 +31,11 @@ two AMD Radeon AI PRO R9700 cards.
|----------|-------|
| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3`; ~31.5 GB/s per card |
| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board |
| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3` (both bridges: `current_link_width=8`, `current_link_speed=32.0 GT/s`, read 2026-10-18); ~31.5 GB/s per card. Each card sits behind an on-board PCIe switch (`01:00.0`/`02:00.0`, `05:00.0`/`06:00.0` — "Navi 10 XL Upstream/Downstream Port of PCI Express Switch"), so the GPU endpoint itself trains **x16** Gen5 while the CPU-facing leg stays x8 |
| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board (2026-09-20 census; not re-readable today — `lspci -vv` needs root) *(unverified 2026-10-18)* |
| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
| Occupied M.2 | M.2_1 — boot NVMe (Samsung PM9C1a, DRAM-less), trained **x2** at Gen5 (63 Gb/s ≈ 7.9 GB/s) |
| Onboard | 2x Realtek RTL8126 5GbE, RTL8922AE WiFi 7, PROM21 SATA controller |
| Occupied M.2 | M.2_1 — boot NVMe (Samsung PM9C1a, DRAM-less, `04:00.0`), trained **x2** at Gen5 (`current_link_width=2`, `current_link_speed=32.0 GT/s`, `max_link_width=4`, read 2026-10-18; 63 Gb/s ≈ 7.9 GB/s) |
| Onboard | 2× Realtek RTL8126 5GbE (`0b:00.0`, `0d:00.0`), RTL8922AE WiFi 7 (`0e:00.0`), ASMedia ASM1064 SATA (`0c:00.0`) + AMD 600-series SATA (`10:00.0`); the board's "PROM21 SATA" label is not what `lspci` reports *(unverified 2026-10-18)* |
### CPU
@@ -41,7 +44,7 @@ two AMD Radeon AI PRO R9700 cards.
| Architecture | x86_64 |
| Vendor | AuthenticAMD |
| Model | AMD Ryzen 7 7700X |
| Cores | 8 cores / 16 threads (1 socket) — SMT is **on**; the BIOS defaults reset with the board swap. `llama.cpp` still runs `--threads 8 --threads-batch 8`, so this is headroom, not a change in the serving shape |
| Cores | 8 cores / **8 threads** today — SMT is **off** (`lscpu`: `CPU(s): 8`, `Thread(s) per core: 1`, `Core(s) per socket: 8`, read 2026-10-18; max observed 5575.87 MHz). The board-swap era recorded 16 threads with SMT on; BIOS defaults reset with the board swap. The `--threads 8 --threads-batch 8` llama.cpp note belongs to the retired llama.cpp serving era |
| NUMA Nodes | 1 (node 0) |
### GPUs
@@ -51,10 +54,10 @@ two AMD Radeon AI PRO R9700 cards.
| Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) |
| Architecture | gfx1201 (RDNA4) |
| VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
| Link | PCIe 5.0 **x8** each (CPU GPP `00:01.1` / `00:01.3`), no XGMI; separate IOMMU groups (15 and 20), ACS `ReqRedir+` on both root ports, so P2P hairpins at the root complex |
| Atomic ops | `Routing+` on all four bridges per chain; `ReqEn+` on both endpoints (`CAP_EXP+28.w = 0x1440`) |
| Link | PCIe 5.0, endpoint-trained **x16** each behind the on-board switch; the CPU-facing leg is x8 (`00:01.1` / `00:01.3`, both `current_link_width=8` at 32.0 GT/s, read 2026-10-18). No XGMI. Separate IOMMU groups — `03:00.0` → group **15**, `07:00.0` → group **20** (`/sys/bus/pci/devices/.../iommu_group`, read 2026-10-18) |
| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see [[Dual-R9700-Cooperation]] |
| PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2, hidden from HIP |
| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2 (rocm-smi GPU[2], 512 MiB). The retired llama.cpp trios hid it with `HIP_VISIBLE_DEVICES=0,1`; the live radiance engine instead enumerates all three AQL devices and ignores it ("3 devices [0:gfx1201 1:gfx1201 2:gfx1036], using 2 of 3, ignoring gfx1036") |
The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA
apt lists remain on disk but no NVIDIA PCI device exists.
@@ -63,29 +66,31 @@ apt lists remain on disk but no NVIDIA PCI device exists.
| Property | Value |
|----------|-------|
| System RAM | **246 GiB** (upgraded for the Lucebox DeepSeek deployment of 2026-09-26; this page said 60 GiB until 2026-09-28 — a stale figure that survived the board swap) |
| Swap | 63 GiB (`/swapfile`) |
| /dev/shm | 32 GiB |
| System RAM | **246 GiB** (`free -g` total 246, read 2026-10-18; upgraded for the Lucebox DeepSeek deployment of 2026-09-26; this page said 60 GiB until 2026-09-28 — a stale figure that survived the board swap) |
| Swap | 63 GiB (`/swapfile`, `swapon` → 64 GB file, read 2026-10-18) |
| /dev/shm | 124 GiB tmpfs (`df /dev/shm`, read 2026-10-18) |
### Storage
| Filesystem | Size | Used | Avail | Use% |
|------------|------|------|-------|------|
| / (nvme0n1p2) | 1.8T | 927G | ~822G | ~54% |
| /mnt/entropy (dm-0) | 91T | 87T | **244G** | **100%** — full; do not plan model payload here |
| / (nvme0n1p2) | 1.8T | 985G | 764G | 57% |
| /mnt/entropy (dm-0) | 91T | 87T | **67G** | **100%** — full; do not plan model payload here |
Models live under `/home/dank/models`; the serving build tree under
`/home/dank/GitPackages/llama.cpp-dflash2` (the older `charlie-ROCmFPX` tree is
still on disk), both sharing the self-contained ROCm 7.14 nightly toolchain
under `~/.local/share/rocm-7.14.0a20260624` — which also carries `librccl.so.1`
and `hipcc`, so an RCCL-backed rebuild needs no new host packages.
Models live under `/home/dank/models`; the serving stack is a digest-pinned
container image, not a host build tree — the llama.cpp trees on disk
(`/home/dank/GitPackages/llama.cpp-dflash2`, the older `charlie-ROCmFPX`, both
still present 2026-10-18) are retired-era builds. They share the self-contained
ROCm 7.14 nightly toolchain under `~/.local/share/rocm-7.14.0a20260624` (also
present 2026-10-18), which carries `librccl.so.1` and `hipcc`, so an
RCCL-backed rebuild needs no new host packages.
### Network
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard RTL8126 5GbE | 2500 Mb/s (negotiated) | 192.168.0.235, **static** | LLM endpoint (port 7770). The address `models.yml`, Prometheus and Pi-hole all point at — preserved across the swap. |
| eno2 | onboard RTL8126 5GbE, second port | 2500 Mb/s | 192.168.0.247, DHCP | **New since the swap**: cabled, and it took the lower-metric default route (101, against eno1's 102). Two live addresses and two default routes for one host — either declare it or bring the link down. |
| eno2 | onboard RTL8126 5GbE, second port | 2500 Mb/s | 192.168.0.247, DHCP | **New since the swap**: cabled, both ports up at 2500 Mb/s. The old lower-metric default route was resolved — `ip route` today shows eno1 `proto static metric 100` and eno2 `proto dhcp metric 200`, so eno1 wins. Two live addresses and two default routes remain for one host |
| wlp14s0 | WiFi 7 (RTL8922AE) | - | - | Renamed with the board (was `wlp8s0`); unused. |
### OS and Software
@@ -93,10 +98,10 @@ and `hipcc`, so an RCCL-backed rebuild needs no new host packages.
| Property | Value |
|----------|-------|
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
| Kernel | 7.0.0-31-generic (`iommu=pt`) |
| Kernel | 7.0.0-38-generic, `iommu=pt` (`uname -r`, `/proc/cmdline`, read 2026-10-18) |
| Python | 3.14.4 |
| ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
| Serving stack | mainline llama.cpp `fb27a525` at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714`, RPATH-bundled with the ROCm 7.14 nightly runtime |
| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon-radiance-qwen38-iq4r]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
hijack `libhsa-runtime64` were purged on 2026-09-15 — see
@@ -104,28 +109,44 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
### Active Services
The active inference service is `frank-medium.service`: mainline llama-server on
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
its unit is archived in `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`,
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
See [[deployment]].
The active inference service is `frank-archon.service` (`systemctl is-active` →
`active`, `is-enabled` → `enabled`, read 2026-10-18): radiance 1.0.13 in a
host-network podman container answering `curl 127.0.0.1:7770/health` →
`{"status":"ok","model":"frank-archon",…}`. See
[[frank-archon-radiance-qwen38-iq4r]].
The services this page once called active are gone: `frank-medium.service`
(llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung
ran 2026-09-16 → 09-20 and is documented in [[frank-medium-llamacpp-dflash2]];
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU
swap. Its archived unit was moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`
and the vllm-radiance replica trio to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`
— neither path exists today (checked 2026-10-18), so the archive location is
unverified. See [[deployment]].
## Hardware Relevance
- 2x 32 GB VRAM: the 27.0 GiB Q8_0 checkpoint plus the 1.06 GiB DFlash2 drafter
and the f16 KV pool for 327,680 tokens fit with the split at 32:33; live
headroom is ~6.3 GiB on ROCm0 and ~5.2 GiB on ROCm1, which is why a batch or
- 2× 32 GB VRAM: the live budget is stated per card, not by checkpoint size —
`--vram-weights-mib 25600` / `--vram-kv-mib 5120`
(`scripts/frank-archon.env:230-231`), against a card total of 31.86 GiB with
96 MiB publisher headroom (`scripts/frank-archon.env:186`). The retired
frank-medium fit (27.0 GiB Q8_0 + 1.06 GiB drafter + f16 KV for 327,680
tokens at 32:33, ~6.3/~5.2 GiB headroom) is history. Either way a batch or
context change needs a re-fit and not a guess.
- 60 GB system RAM is mostly headroom; nothing in the serving path is
CPU-resident.
- The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off
the shared-memory Raphael device.
- 246 GiB system RAM is **not** headroom: the live shape pins a 43.88 GiB host
pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`,
`scripts/frank-archon.env:184`) plus the 47.68 GiB n-gram/PLE table in RAM,
and `free -g` showed **90 GiB available** with the engine loaded (read
2026-10-18). The frank-medium claim that "nothing in the serving path is
CPU-resident" is retired with that runtime.
- The iGPU stays out of the serving path by engine choice, not by an env
var: radiance enumerates all three devices and uses 2 of 3, ignoring
gfx1036. No `HIP_VISIBLE_DEVICES` is set in the container today (`podman
inspect frank-archon`, read 2026-10-18).
## See Also
- [[frank-medium-llamacpp-dflash2]] — the active serving runbook
- [[frank-archon-radiance-qwen38-iq4r]] — the active serving runbook
- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
- [[deployment]] — the deploy flow
- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
+51 -29
View File
@@ -14,11 +14,15 @@ timestamp: 2026-09-16
# Dual R9700 Cooperation
Dankserver's two R9700s serve `frank-medium` as **one vLLM-radiance process at
tensor_parallel_size 2** ([[frank-medium-radiance-mxfp4]]). This page records why
Dankserver's two R9700s run **one tensor-parallel engine across both cards** —
today that is `frank-archon` under radiance 1.0.13, TP=2
([[frank-archon-radiance-qwen38-iq4r]]). When this page was written the endpoint
was `frank-medium`: first llama.cpp's layer split
([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20
one vLLM-radiance process at `tensor_parallel_size 2`
([[frank-medium-radiance-mxfp4]], 2026-09-20 → 09-26). This page records why
true tensor parallelism was impossible on the previous board (TUF GAMING B650-E
WIFI), and what the 2026-09-20 board swap changed. The llama.cpp layer split that
served the endpoint until that swap is [[frank-medium-llamacpp-dflash2]].
WIFI), and what the 2026-09-20 board swap changed.
## The short answer
@@ -43,9 +47,11 @@ What that bought:
and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp
`--split-mode tensor` booted and served against an RCCL-backed build — so a
collective HAS been dispatched on this host since the swap.
- **One of them is production.** TP=2 wins on every axis measured, by 2.1x on
decode and 3.6x at 4-way concurrency, so it replaced the layer split on the day
of measurement. See [[frank-medium-radiance-mxfp4]] for the deployed shape.
- **TP=2 is the production shape.** It wins on every axis measured, by 2.1x on
decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day
of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is
retired 2026-09-26, and what runs today is the iq4r publisher under radiance
1.0.13 ([[frank-archon-radiance-qwen38-iq4r]]).
- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
could not carry the DFlash2 drafter, which is the honest caveat on the
comparison below.
@@ -137,20 +143,30 @@ LAN/SATA/WiFi:
- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
- `rocm-smi --showtopo` still reports `PCIE` between the cards (Navi 48 has no
XGMI), they are still in separate IOMMU groups (15 and 20), and both root
ports have ACS `ReqRedir+`, so P2P hairpins at the root complex. Plain P2P
traffic measured 45.27 GB/s bidirectional on the *old* board — DMA was never
the gate, atomics were.
- Kernel 7.0.0-31-generic with `iommu=pt`. Both were ruled out as causes
([above](#ruled-out-empirically-do-not-re-derive)) and both are unchanged.
- `rocm-smi --showtopo` reported `PCIE` between the cards (Navi 48 has no XGMI),
they were in separate IOMMU groups (15 and 20 — re-read 2026-10-18: still 15
and 20), and both root ports had ACS `ReqRedir+`, so this page inferred P2P
would hairpin at the root complex. **That inference is superseded:** the live
engine says the opposite about the same pair — "I device: the 2 cards reach
each other's memory directly (peer to peer)" (`podman logs frank-archon`,
2026-10-15 boot). Plain P2P traffic measured 45.27 GB/s bidirectional on the
*old* board — DMA was never the gate, atomics were.
- Kernel 7.0.0-31-generic with `iommu=pt` then; this host runs 7.0.0-38-generic
with `iommu=pt` today (`/proc/cmdline`, read 2026-10-18). Both were ruled out
as causes ([above](#ruled-out-empirically-do-not-re-derive)) and this host's
IOMMU posture is unchanged.
**Still unproven.** The gate that matters is ROCclr's `checkAtomicSupport()`
reading the `kLinkAtomicSupport` link attribute; the ReqEn bit is the same cause
seen from the kernel side, but no RCCL kernel has been launched since the swap.
Until the 2-rank probe runs — or a `--split-mode row` boot that does *not* log
`internal AllReduce init failed ... falling back to meta-backend butterfly` —
"collectives work here" is a strong expectation, not a measurement.
**Still unproven (as of 2026-09-20).** The gate that matters is ROCclr's
`checkAtomicSupport()` reading the `kLinkAtomicSupport` link attribute; the ReqEn
bit is the same cause seen from the kernel side, but no RCCL kernel had been
launched since the swap. Until the 2-rank probe runs — or a `--split-mode row`
boot that does *not* log `internal AllReduce init failed ... falling back to
meta-backend butterfly` — "collectives work here" was a strong expectation, not a
measurement. **RCCL never got that measurement**; what settled the operational
question instead is production traffic: radiance's TP=2 wire moves cross-rank
all-reduces over the direct P2P path and reports its own precision — "tp wire:
LOSSY (--tp-wire wht6) at 128 KiB a message and above … Steps of at most 25 rows
are exact" (`podman logs frank-archon`, 2026-10-15 boot).
## Measured matrix
@@ -249,9 +265,13 @@ RCCL 2.27 does carry gfx1201 kernels, and that is what row 3 linked against.
**Rule for any future multi-GPU collective experiment: stop and mask the
production service first**, per the benchmarking policy, or run it on a host that
is not serving users. Since 2026-09-20 `frank-medium` is a rootless quadlet, so
that is `systemctl --user stop frank-medium && systemctl --user mask frank-medium`
on Dankserver.
is not serving users. The production service is `frank-archon`, a **root** unit,
so on Dankserver that is `sudo systemctl stop frank-archon && sudo systemctl
mask frank-archon` (`scripts/frank-archon.service`; never `disable` — that
deletes the unit file, and `mask` is what the bench policy prescribes). The
2026-09-20 wording below — `systemctl --user stop frank-medium && systemctl
--user mask frank-medium`, frank-medium being a rootless quadlet — is retired
with that service, which today reports `not-found` (read 2026-10-18).
This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live
service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a
@@ -260,7 +280,7 @@ GPU driver is a plausible cause and this host has precedent for going
unreachable until a power cycle.
## The third path, and the one in production: llama.cpp's layer split
## The third path (production in the layer-split era): llama.cpp's layer split
Everything above is about *collectives*, and vLLM cannot proceed without them
in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage
@@ -270,8 +290,9 @@ layers are distributed across the cards and the boundary activation is a
plain per-backend copy, once per forward pass. No RCCL kernel is ever launched
on the second card, so the atomics gate never comes up.
That is what `frank-medium` runs today ([[frank-medium-llamacpp-dflash2]]): one
process, one model, both cards, 262144 context. Measured against tensor split
That is what `frank-medium` ran from 2026-09-16 to 09-20
([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards,
262144 context. Measured against tensor split
on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 /
26.3 tensor (butterfly fallback, no RCCL). **That tensor number is not a
prediction for an RCCL-backed build** — it is what tensor split costs when the
@@ -282,9 +303,10 @@ The ranking therefore changed twice. On 2026-09-16: software already gets both
cards onto one model, so the hardware fix stopped being a capability question
and became a performance one — *tensor* parallelism, ~40 % decode, plus one
pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
hardware half of that question was answered by the board swap; the measurement
half has not run yet, and `GGML_HIP_RCCL` is still `OFF` in
`scripts/build-llama-rdna4.sh` for exactly the reason this page documents.
hardware half of that question was answered by the board swap; the llama.cpp
measurement half never ran, and `GGML_HIP_RCCL` still defaults `OFF` in
`scripts/build-llama-rdna4.sh:41` (re-read 2026-10-18) for exactly the reason
this page documents — moot while no llama.cpp shape is deployed on this host.
## What would change the answer
serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
+39 -8
View File
@@ -15,10 +15,21 @@ timestamp: 2026-09-03
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
>
> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today
> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625
> (`ss -ltn`, read 2026-10-18). The runtime is still installed
> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`,
> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is
> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
> the RTX 3090 ([[frank-zealot-llama-prism-bonsai2-pq20]]). See [[frank-npu]]
> for the retirement record. Everything below is the 2026-08 research record.
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-zealot` (port 52625) serves in production. See Promotion.
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.
## Verdict (2026-08-16)
@@ -46,7 +57,7 @@ Key properties, per the project README and docs:
## NPU Hardware on Dankclaw
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS. It sits beside the Radeon 8060S iGPU and shares the same 30 GiB unified memory pool.
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS (vendor spec: "NPU TOPS — Up to 50 TOPS", [amd.com product page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)). It sits beside the Radeon 8060S iGPU and shares the same unified memory pool — 30 GiB exposed to the OS in the 2026-08 reading below; the box presents **121 GiB** today (`free -g` total 121, read 2026-10-18).
Live verification on Dankclaw, 2026-08-16:
@@ -60,9 +71,9 @@ $ flm validate
[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity.
```
- Kernel 7.0.0-29-generic ships the `amdxdna` driver in-tree. The driver is loaded.
- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18).
- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`).
- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".
@@ -78,6 +89,11 @@ Budget math for this fleet:
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
2026-10-18 re-read: Dankclaw's OS now reports **121 GiB** (`free -g`) and
Dankserver **246 GiB** (`free -g`), so the half-of-RAM rule would price the NPU
budget at ~60 GiB and the 7700X box still has no NPU. The 30 GiB / 62 GB rows
are the 2026-08-16 state; the table is moot while no NPU endpoint is deployed.
Weight sizes (approximate, Q4-class):
| Model | Quant | Weights | Fits ~15 GiB budget? |
@@ -162,7 +178,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
- `/dev/accel/accel0` present. `flm validate` passes fully.
- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.)
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
@@ -241,7 +257,14 @@ Consequence: the NPU endpoint is a single lane. It suits low-frequency backgroun
## Test Runbook
Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.
This runbook targets a retired endpoint; keep it as the procedure if the NPU is
ever revived. Do not run large model loads while the production service holds
most of the unified pool — 30 GiB exposed to the OS in the 2026-08 reading,
**121 GiB** today (`free -g`, read 2026-10-18). Check free RAM first, or stop
the service the way the bench policy requires (`sudo systemctl stop
frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
otherwise restart it). The NPU shares the unified pool with everything else on
the box.
```mermaid
graph TD
@@ -387,10 +410,18 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot
of Dankclaw killed it. The fleet law today is the opposite: every deployed
container/service ships a boot-persistent, self-healing unit, and the retired
NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it
cannot come back at boot.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16).** `frank-zealot` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then
served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the
`amd_iommu=off` decision that day and the unit is masked today. The remaining
NPU content (benchmarks, runbook steps) is research on a device that is again
usable since 2026-10-18 but serves nothing.
## Citations
serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md
+17 -3
View File
@@ -74,6 +74,11 @@ cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
```
Read on Dankclaw 2026-10-18: `mem_info_vram_total` = 2147483648 (2 GiB of
dedicated carve), `mem_info_gtt_total` = 122402963456 (114 GiB) — the aperture is
the whole budget, the carve is noise, and the OS sees 121 GiB of it
([[Dankclaw System Specs]]).
A large GTT window means the "GPU memory" is the *same* LPDDR under one
memory controller: adding a stage there adds a compute engine, not capacity
or bandwidth. Only the discrete card's VRAM is additive. Fencing more
@@ -101,6 +106,13 @@ of bytes), so it is tempting. Before proposing it:
gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device`
(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268),
`--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v`
(`-ctkd`/`-ctvd`, 4111/4124) all exist in
`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp),
read 2026-10-18).
## 6. Expressing a split (llama.cpp, the usual vehicle)
```bash
@@ -110,10 +122,12 @@ of bytes), so it is tempting. Before proposing it:
- buffer types match by name at load (`ggml_backend_buft_name`), so device
names work as `-ot` values;
- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`,
`common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`;
- several backends in one binary is fine (CUDA + Vulkan is the practical
pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
build's CMakeCache tells you what needs adding;
pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs
do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the
existing build's CMakeCache tells you what needs adding;
- contiguous ranges are the WRONG split when KV bandwidth dominates: they
drag half the attention layers (and their KV) back to the slow pool.
serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md
+17 -16
View File
@@ -10,13 +10,13 @@ timestamp: 2026-08-28
## Overview
Dankserver uses NVIDIA GPU with system RAM for hybrid processing. vLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]).
## Architecture
### Model Weights
Model weights stay on GPU. vLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.
In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2).
### KV Cache
@@ -47,11 +47,16 @@ Set `--gpu-memory-utilization` based on your GPU:
| 80 GB (A100 80GB) | 0.85-0.95 |
| 80 GB (H100) | 0.9-0.95 |
Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s.
The live engine does not use this knob at all — it claims an explicit per-card
pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`,
`scripts/frank-archon.env:230-231`).
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
### Swap Space / KV Offload
Recent vLLM releases removed the `--swap-space` engine argument. Use `--kv-offloading-size` to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
Recent vLLM releases removed the `--swap-space` engine argument — `CacheConfig` on `main` has no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
| System RAM | Recommended Swap |
|------------|------------------|
@@ -69,26 +74,24 @@ Use tensor parallelism for large models:
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.85 \
--swap-space 16
--gpu-memory-utilization 0.85
```
### KV Cache Data Type
Use FP8 KV cache for Ampere and newer GPUs:
Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache).
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--swap-space 16
--gpu-memory-utilization 0.9
```
FP8 KV cache reduces memory usage by 50% compared to BF16.
## NUMA Optimization
Dankserver may have multiple NUMA nodes. Optimize with:
Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies:
```
vllm serve model \
@@ -117,10 +120,11 @@ VLLM needs CPU cores for:
Minimum physical cores = 2 + N (where N is GPU count).
For Dankserver with 1 GPU:
For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read
2026-10-18):
- Minimum 3 physical CPU cores
- Recommended 8+ physical CPU cores
- Minimum 4 physical CPU cores by the 2 + N rule
- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off)
## Performance Tuning
@@ -169,7 +173,7 @@ vllm serve model \
Check GPU memory:
```
nvidia-smi
rocm-smi
```
Check for preemption warnings in VLLM logs.
@@ -187,7 +191,6 @@ htop
```
vllm serve Qwen/Qwen3.6-35B-A3B \
--gpu-memory-utilization 0.85 \
--swap-space 16 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
@@ -198,7 +201,6 @@ vllm serve Qwen/Qwen3.6-35B-A3B \
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--swap-space 16 \
--max-num-batched-tokens 8192
```
@@ -208,7 +210,6 @@ vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--swap-space 32 \
--max-model-len 8192
```
serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md
+14 -1
View File
@@ -18,6 +18,18 @@ interchangeable:
| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
Upstream anchors, re-read 2026-10-18 against `master`:
`src/llama-context.cpp:517-526` computes `pipeline_parallel` with
`!model.has_tensor_overrides()` as a required term;
`ggml/src/ggml-backend.cpp:1089` carries the verbatim abort
`pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)`
(the `:941` anchor above is where it sat in the `37b53fd` build this was
measured on); and `common/arg.cpp` defines `-sm/--split-mode` (2821),
`-fit/--fit` (2882), `--spec-type` (4295), `--spec-draft-device` (4259),
`--spec-draft-n-max` (4161), `-fa/--flash-attn` (1758) — every flag in §4 is
real upstream, not fork-only
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)).
Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
cross-backend pair → model load fails with only `failed to load model`;
@@ -147,7 +159,8 @@ not prevent a boundary-copy fault.
## 6. Drafter KV: sized from the TARGET, bounded by its window policy
`common/speculative.cpp` builds the draft context with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — **no flag caps it.** So:
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — `common/speculative.cpp:2663` on
`master`, re-read 2026-10-18 — **no flag caps it.** So:
```
draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md
+19 -8
View File
@@ -55,7 +55,7 @@ First decide which kind of memory is in question:
|---|---|---|
| RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
| R9700s (Dankserver) | discrete | `rocm-smi` figure (64 GiB, SVM-backed) |
| R9700s (Dankserver) | discrete | `rocm-smi --showmeminfo vram` → 34,208,743,424 B = 32,624 MiB (31.86 GiB) per card, read 2026-10-18 — **not** the 64 GiB "SVM-backed" figure this row once carried ([[Dankserver System Specs]]) |
Per-process truth on NVIDIA:
`nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
@@ -66,8 +66,12 @@ co-tenants.
contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
after reboot vs 475/14.7 GiB before means the model did not come back into
the same hole), device budget, KV pool/working split, "left for everything
else". llama.cpp prints at `llama-model.cpp:1827` (per-backend model buffer)
and `llama-context.cpp:692` (compute buffer) — but **`llama-bench`
else". llama.cpp prints them at `llama-model.cpp:1827` (per-backend model buffer)
and `llama-context.cpp:692` (compute buffer) on the era's build; on today's
`master` the same `LLAMA_LOG_INFO` lines sit at `llama-model.cpp:2009`
("model buffer size =") and `llama-context.cpp:842` ("compute buffer size =")
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp),
re-read 2026-10-18) — but **`llama-bench`
suppresses loader INFO and a systemd unit did not put them in the journal**
(verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
keeping the first ~200 lines.
@@ -80,7 +84,10 @@ co-tenants.
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
"move layers to the card" converts reclaimable cache into device memory and
frees nothing for co-tenants. A served model's footprint ≠ weight size
(frank-zealot: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
(a retired frank-zealot shape: 22.6 of 24.1 GiB for an 11.29 GiB model; the card
today reads 24,576 MiB = 24.0 GiB total, `nvidia-smi --query-gpu=memory.total`,
2026-10-18, and the deployed payload is the PQ2_0 pair
[[frank-zealot-llama-prism-bonsai2-pq20]]). Do the capacity
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
- **Context lengths: derive, never remember.** `llama-bench` computes
@@ -94,9 +101,13 @@ co-tenants.
## 3. Prompt cache and state sizing
`--cache-ram` (MiB, default 8192) is HOST RAM for saved prompt states —
unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (default 32) is a
different mechanism: per-slot KV snapshots; `0` disables them.
`--cache-ram` (alias `-cram`, `common/arg.cpp:1719`) is HOST RAM for saved
prompt states — unpinned, on demand, moves no VRAM. `--ctx-checkpoints` (alias
`-ctxcp`/`--swa-checkpoints`, `common/arg.cpp:1701`) is a different mechanism:
per-slot KV snapshots; `0` disables them. Both flags are current upstream; the
two defaults quoted here (8192 MiB and 32) are the era's values and were not
re-readable in the upstream sources checked *(unverified 2026-10-18)* — print
them from the binary (`llama-server --help`) before sizing on them.
**State law** (compute before trusting any cap): a state is the serialized KV
of the whole sequence + constant recurrent state + the draft context's KV:
@@ -157,7 +168,7 @@ in the timings. Context cap for full-model residency on that shape stayed
## Related
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot-llama-prism-bonsai2-pq20]])
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
serverAppConfigs-docs/LLM/Model Swap and Rollback.md
+13 -6
View File
@@ -251,17 +251,24 @@ step a redeploy deadlocks against its own gate.
small/fast + image roles at the new endpoint; move the vision role WITH the
vision tower (one owner per capability). `shared/homepage/services.json`: a
card per endpoint, probed from the monitor's own host. Prometheus: only add
a job if the runtime actually exposes Prometheus text (halogen publishes
`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
reason written down rather than left as a permanent red).
a job if the runtime actually exposes Prometheus text — and re-measure rather
than repeat, because that answer moves. This bullet once said halogen serves
`/health`, `/cache`, `/v1/*` "and nothing else"; measured 2026-10-18 on
Dankclaw, `curl -s http://127.0.0.1:7770/metrics` returns Prometheus text
(`# HELP llamacpp:prompt_tokens_total …`, content-type
`text/plain; version=0.0.4`) while an invented path 404s — so the surface has
grown and a scrape target is expressible again.
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
<large> --service <small>` (order matters); per endpoint: `/health`,
`/v1/models` reports the alias, a chat completion returns content AND
`reasoning_content`, a `tools` request returns `tool_calls`, an image
request is answered (or refused with a named reason), the runtime's own
speculation counters move, and the unit is `enabled`/`generated` with linger
on. Read the engine's boot ledger for a silent clamp. Finally spend one
speculation counters move, and the unit is `enabled` and `active`. All three LLM
endpoints are **root** units today (`User=dank` + `WantedBy=multi-user.target` —
`scripts/frank-archon.service:30,71`, `scripts/frank-dragoon.service:8,31`,
`scripts/frank-zealot.service:8,26`), so linger is not what gates them; linger
matters for rootless quadlets. Read the engine's boot ledger for a silent clamp. Finally spend one
real request through the harness per changed role — routing labels lie; the
servers' own journals do not.
@@ -278,4 +285,4 @@ predecessor "stopped" (the port race above).
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon-radiance-qwen38-iq4r]]
serverAppConfigs-docs/LLM/PCIe Link and Atomics.md
+19 -7
View File
@@ -59,9 +59,11 @@ Gen4 usable payload after 128b/130b: **x4 7.877 GB/s, x8 15.75, x16 31.5**.
**Ceiling arithmetic:** `link time/token = active expert bytes / measured
H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
LPDDR5X on the Strix Halo). **If the link is >10x slower than the memory
behind it (x4 is 31x here), streaming loses by construction and no tuning
rescues it; doubling the link only doubles a losing number.**
LPDDR5X on the Strix Halo — 128 GB LPDDR5x-8000 on a 256-bit bus,
[amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)).
**If the link is >10x slower than the memory behind it (x4 is 31x here),
streaming loses by construction and no tuning rescues it; doubling the link
only doubles a losing number.**
Worked example (Dankclaw 2026-09-16): Gen4 x4 measured H2D **6.72 / D2H 6.60
GB/s** (85 % of spec) — pinned H2D ≈ async, pageable within 1 %, i.e.
@@ -215,15 +217,25 @@ reporting an outage (`gitea` vs `gitea-server`).
complex; x4 suffices for TP.
2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — both
R9700s on root ports `00:01.1`/`00:01.3`, every bridge `Routing+`, both
endpoints `ReqEn+`).
endpoints `ReqEn+`). Re-checked 2026-10-18 without root: both root ports
report `current_link_width=8` at `32.0 GT/s`
(`/sys/bus/pci/devices/0000:00:01.{1,3}/current_link_width`), and the live
engine states direct P2P between the cards
([[Dual-R9700-Cooperation]]).
3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
continuous batching / spec decode.
4. Independent TP=1 replicas + routing proxy — cards never share one model.
**Closed software routes (do not re-litigate):** vLLM's IPC custom allreduce
is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
12.0 → off on RDNA4 by code limit) — and patching it is insufficient because
TP init's `broadcast`/`barrier`/logits `all_gather` have no non-RCCL path. A
12.0 → off on RDNA4 by code limit) — the table is
`CUSTOM_ALL_REDUCE_MAX_SIZES` in
`vllm/distributed/device_communicators/all_reduce_utils.py`, whose keys on
today's `main` are exactly `"9.0"`, `"10.0"`, `"10.3"`, `"10.7"`
([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py),
read 2026-10-18); nothing keys on a ROCm-reported `12.0`. — and patching it is
insufficient because TP init's `broadcast`/`barrier`/logits `all_gather` have
no non-RCCL path. A
fork wrapping `.all_reduce` (e.g. radiance `RADIANCE_USE_R4D_AR` one-shot
P2P, gloo for IPC-handle exchange) does not intercept
`PyNcclCommunicator.__init__`'s warmup allreduce. NCCL symmetric memory needs
@@ -241,4 +253,4 @@ kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live gather economics
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md
+39 -14
View File
@@ -10,6 +10,14 @@ timestamp: 2026-09-03
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
> [deployment.md](../deployment.md) for the active configuration.
>
> **2026-10-18: nothing pipeline-parallel runs on the fleet.** No RPC listener
> exists on either host (`ss -ltn` shows no :50052), every artifact named below
> is gone (§Files), and the worker hardware left the box: the RTX 3090 was
> removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
> runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
> inside one process ([[Dual-R9700-Cooperation]],
> [[frank-archon-radiance-qwen38-iq4r]]).
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
@@ -54,7 +62,10 @@ helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-zealot` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. `deploy.sh dankserver` is broken until retargeted. See
head. The "deploy.sh dankserver is broken until retargeted" note is long fixed:
`./scripts/deploy.sh dankserver --service frank-archon` is the documented
invocation today (`scripts/deploy.sh:38`, dispatch case
`dankserver:frank-archon` at `scripts/deploy.sh:149`). See
[[deployment]] for the full boot-policy table.
## FA-all-quants rebuild (2026-08-17)
@@ -133,10 +144,13 @@ content, degenerate logits). Root cause chain:
A clean cache and a healthy disk produced correct output immediately
(acceptance 0.89 on the first smoke test).
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
Never let the cache directory grow unchecked; a full disk silently
corrupts it. Signature to recognize: healthy load + degenerate repeated-
token output.
Guard was `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap). That
script is gone from the repo and from the host, yet the crontab line still
runs every 10 minutes against it
(`*/10 * * * * /home/dank/vLLM/scripts/prune-rpc-cache.sh`, `crontab -l`, read
2026-10-18 — the target does not exist, so it fails silently). Never let the
cache directory grow unchecked; a full disk silently corrupts it. Signature to
recognize: healthy load + degenerate repeated-token output.
## Topology
@@ -163,9 +177,9 @@ llama-server b10433 ggml-rpc-server b10433
- **Why the head is on Dankclaw:** the model file is already there; the
worker streams its tensor share at load (no 31.5 GB copy to the server).
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
so the production b10121 build stays untouched. A version skew hangs the
RPC handshake with no error.
worker build lived in a worktree `/home/dank/llama-cpp-rpc` on Dankserver so the
production b10121 build stayed untouched (that path no longer exists, read
2026-10-18). A version skew hangs the RPC handshake with no error.
## Files
@@ -179,6 +193,14 @@ llama-server b10433 ggml-rpc-server b10433
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
**Every PP file in that table except `scripts/frank-dragoon.env` no longer
exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
`serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
`pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
`scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
head env ([[frank-dragoon-flash-server-v2-native]]).
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
1. **No `--device` on the head.** Setting `params.devices` makes
@@ -188,9 +210,9 @@ llama-server b10433 ggml-rpc-server b10433
2. **The iGPU side is ROCm by default.** Backends register in fixed order
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
decode (measured on this box).
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off; removed — the
path is gone on 2026-10-18) — set `HEAD_BIN` in the env. Trade: ROCm side =
faster prefill, ~3% slower decode (measured on this box).
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
@@ -207,7 +229,9 @@ llama-server b10433 ggml-rpc-server b10433
## Runbook (morning of 2026-08-16)
Run from any box with ssh access to both (repo path on claw:
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM` — both still exist as
stale checkouts; the deployment checkout today is `/home/dank/serverAppConfigs`
on both hosts).
```
# 0. one-time: pull the kit on both boxes
@@ -243,8 +267,9 @@ Verification checklist after each `start`:
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
1,1 split.
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
(production small service + worker share).
3. `nvidia-smi` on the server showed ~15–16 GB used by the worker
(production small service + worker share). No NVIDIA card is in that host
today — the equivalent check is `rocm-smi --showmeminfo vram`.
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
serverAppConfigs-docs/LLM/Shared Memory Architecture Dankclaw.md
+38 -14
View File
@@ -36,7 +36,7 @@ In unified memory:
### ROCm Version
vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw has ROCm 7.2.4 from AMD repos.
vLLM supports AMD GPUs with ROCm 6.3 or above *(unverified 2026-10-18)*. Dankclaw has ROCm 7.2.4 from AMD repos (`/etc/apt/sources.list.d/rocm.list` → `deb https://repo.radeon.com/rocm/apt/7.2.4 noble main`; `rocm-core 7.2.4.70204-93~24.04`, read 2026-10-18). No vLLM endpoint runs on this host today — 7770 is halogen-flash-server and 7771 is llama.cpp ([[Dankclaw System Specs]]) — so this page is reference material, not the serving path.
Check ROCm version:
@@ -83,6 +83,12 @@ Available backends for ROCm:
| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |
All seven names exist on current vLLM `main` (`AttentionBackendEnum` in
`vllm/v1/attention/backends/registry.py` —
[source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/attention/backends/registry.py),
read 2026-10-18), and `--attention-backend`, `--numa-bind`, `--numa-bind-nodes`
are real `vllm serve` flags (`vllm/engine/arg_utils.py:1098,1171-1176`).
## Special Flags for APU
### Memory Configuration
@@ -97,7 +103,7 @@ vllm serve model \
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
- `max_model_len`: Match actual context needs
- The old `--swap-space` argument was removed from current vLLM. Use `--kv-offloading-size` for CPU KV offload
- The old `--swap-space` argument was removed from current vLLM — `CacheConfig` on `main` carries no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) for CPU KV offload
### NUMA Binding
@@ -126,15 +132,25 @@ export HSA_ENABLE_SDMA=1
export HSA_FORCE_FINE_GRAIN_PCIE=1
```
### GPU Memory Limits
These three variables came out of the September research pass and were not
re-checked against current ROCm documentation *(unverified 2026-10-18)*.
Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:
### GPU "memory limits" and the gfx override
There is no memory-limit knob for the APU here. What the GPU may address is the
GTT aperture (`mem_info_gtt_total`, see §ROCm Overcommit); the firmware VGM/UMA
setting in BIOS is what sizes the *dedicated* carve-out.
`HSA_OVERRIDE_GFX_VERSION` is not a memory setting — it makes the runtime report
a different ISA version:
```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
```
This may be needed for Strix Halo compatibility.
Do not set it on this host: the stack already reports `gfx1151` natively
(`rocm-smi --showproductname` → `GFX Version: gfx1151`, read 2026-10-18). The
override exists for GPUs whose ISA the installed runtime does not know.
## Performance Considerations
@@ -144,13 +160,13 @@ Unified memory has lower bandwidth than dedicated VRAM:
| Memory Type | Bandwidth |
|-------------|-----------|
| HBM3 (A100) | 2 TB/s |
| HBM2e (A100 80 GB) | 2 TB/s |
| GDDR6 (RTX 4090) | 1 TB/s |
| LPDDR5x (Strix Halo) | ~170 GB/s |
| LPDDR5x-8000, 256-bit (Strix Halo) | 256 GB/s ([amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)) |
### ROCm Overcommit
ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture, `gtt_total=122402963456` bytes = 114 GiB; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM is 64 GiB (61 GiB visible; the box was upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture: `mem_info_gtt_total` = 122402963456 bytes = 114 GiB, read 2026-10-18 under `/sys/class/drm/card1/device/`; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM visible to the OS is **121 GiB** today (`free -g` total 121, read 2026-10-18; 64 GiB in the earlier reading on this page, upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
Lower bandwidth means slower inference. Plan for lower throughput.
@@ -195,6 +211,10 @@ Unified memory limits concurrent requests:
| Prefix Caching | Yes |
| Chunked Prefill | Yes |
The two "No" rows (Async Output, Multi-step Decoding) were transcribed from a
vLLM feature-support snapshot and not re-checked against current upstream
*(unverified 2026-10-18)*.
### Context Length
Some models on ROCm have context length limited to 4096 tokens. Check model documentation.
@@ -205,9 +225,16 @@ FP8 quantization depends on GPU architecture:
- MI300 series supports native FP8
- Radeon RX 7900 supports FP8 via software emulation
- Strix Halo (gfx1151) uses **software emulation** for FP8 (no native FP8 tensor cores); expect memory savings, not full compute speedup
- Strix Halo (gfx1151) uses **software emulation** for FP8 matmuls — RDNA 3.5 has
no FP8 tensor instructions ([FP8-ROCM](https://github.com/wabibito/FP8-ROCM));
expect memory savings, not full compute speedup. (Conflicting vendor-side
evidence exists for an FP8/INT8 WMMA path used by FSR4 on RDNA 3.5, so treat
the matmul claim as the settled half.)
For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native ROCm FP4 kernels.
For FP4 on Dankclaw the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provided native
ROCm FP4 kernels; that runtime line is **retired** (CIRU/Kairic eras,
[[ciru-activefpx-promptforge]], [[frank-dragoon-kairic-edge-qwen38-27b]]) and no
ROCmFPX build serves this host today.
## Example Commands
@@ -217,7 +244,6 @@ For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native
vllm serve Qwen/Qwen3.6-35B-A3B \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-num-batched-tokens 4096 \
--max-num-seqs 64
```
@@ -228,8 +254,7 @@ vllm serve Qwen/Qwen3.6-35B-A3B \
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.85 \
--kv-cache-dtype fp8 \
--swap-space 8
--kv-cache-dtype fp8
```
### With Long Context
@@ -238,7 +263,6 @@ vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
vllm serve Qwen/Qwen3.6-35B-A3B \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--gpu-memory-utilization 0.8 \
--swap-space 16 \
--max-model-len 16384 \
--rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
```
serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
+6 -3
View File
@@ -14,7 +14,7 @@ tags:
timestamp: 2026-09-11
---
# Dankclaw IOMMU translation off — decision record
# Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18)
**Current state (since 2026-10-18 01:24): Dankclaw boots `iommu=pt`** — identity
DMA, IOMMU armed, XDNA2 NPU bound and usable. Everything below is the record of
@@ -201,8 +201,11 @@ and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip.
boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5
GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP
depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all
answered 200. (Later the same day the pool was doubled to 401,408 positions for
slot parity — that reservation is ~10.8 GiB and the engine-side total ~79.4 GiB;
answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
slot parity was COMMITTED — `2b12716` — but as of 2026-10-18 NOT yet
redeployed: the live boot still reserves ONE 200,704-position pool (5.5 GiB),
host `frank-dragoon.env` included. When it lands, the reservation is ~10.8
GiB and the engine-side total ~79.4 GiB;
see [[frank-dragoon-flash-server-v2-native]].)
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
serverAppConfigs-docs/LLM/deployment.md
+36 -25
View File
@@ -1,7 +1,7 @@
---
type: guide
title: LLM Deployment
description: Scoped deployment and update process for the two inference services.
description: Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot).
tags:
- llm
- deployment
@@ -19,7 +19,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.
| Target | Host | Service | Port |
|---|---|---|---:|
| Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
| Dankserver | `dank@dankserver` | `frank-archon` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-dragoon` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-zealot` | 7771 |
@@ -39,7 +39,7 @@ graph LR
Run the unified deployer from the repository root:
```bash
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --service frank-dragoon
./scripts/deploy.sh dankclaw --dry-run
```
@@ -47,10 +47,14 @@ Run the unified deployer from the repository root:
Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
on the selected host. The following files select the corresponding service:
- Any file under `dankserver/frank-medium/` (the Quadlet unit, its
entrypoint and its healthcheck) selects `frank-medium` on Dankserver —
the container mounts the entrypoint and healthcheck straight out of the
checkout, so any file there changes deployed state.
- `scripts/frank-archon.env`, `frank-archon.service`, and
`serve-frank-archon.sh` select `frank-archon` on Dankserver
(`scripts/deploy-plan.py:422-428`). The quadlet-era directory
`dankserver/frank-medium/` left the tree on 2026-09-26 and plans nothing
(`scripts/deploy-plan.py:374-377`).
- `scripts/qwen-sharp-v22.4.0.jinja` selects `frank-dragoon` — a planner rule
left from the llama.cpp era when dragoon loaded it via
`--chat-template-file` (`scripts/deploy-plan.py:415-421`).
- `scripts/frank-dragoon.env`, `frank-dragoon.service`, and
`serve-frank-dragoon.sh` select `frank-dragoon` on Dankclaw.
- `scripts/frank-zealot.env`, `frank-zealot.service`, and
@@ -59,27 +63,33 @@ on the selected host. The following files select the corresponding service:
do not select any inference service.
The target-side `scripts/deploy-remote.sh` installs the selected service.
`frank-dragoon` and `frank-zealot` are **root units**: the handler relinks the
repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
handler retires its predecessors (the replica Quadlets and the retired llama
root unit), stops the service's own unit, waits for the published port to
clear, installs the `.container` and restarts the generated unit. Each
All three LLM services are **root units** whose repo unit file is relinked
into `/etc/systemd/system/` (`relink_unit`), daemon-reloaded, enabled and
restarted (`deploy-remote.sh:1158` `deploy_llm_dragoon`, `:1254`
`deploy_llm_zealot`, `:1315` `deploy_llm_archon`). The dragoon and archon
handlers wait for port 7770 to be released before the replacement starts
(`wait_port_free`; the 2026-09-16/26 cutover lessons). The archon handler
additionally hard-gates on the env file, the digest-pinned image ref, and the
payload before touching the unit (refuse rather than restart-loop). Each
handler then waits on that service's health endpoint —
`http://127.0.0.1:7770/health` for frank-dragoon and frank-medium,
`http://127.0.0.1:7771/health` for frank-zealot. The wait budget is 20 minutes
because cold model loads are slow. A failed handler does not advance its
per-service state.
`http://127.0.0.1:7770/health` for frank-dragoon (budget 1200 s) and
frank-archon (1800 s), `http://127.0.0.1:7771/health` for frank-zealot
(900 s) — and refuses a false green: the deploy is only green when the unit
itself reports a live MainPID behind the health answer
(`deploy-remote.sh:1395-1403`, `:1285`). A failed handler does not advance
its per-service state.
## Unit and artifact paths
The root units point into the stable target checkout:
```text
/home/dank/serverAppConfigs/scripts/frank-archon.service
/home/dank/serverAppConfigs/scripts/frank-dragoon.service
/home/dank/serverAppConfigs/scripts/frank-zealot.service
```
`frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
(`frank-medium.container`) rather than a unit in `scripts/`.
All three services keep their unit, env and serve script as this flat trio
under `scripts/` — there is no LLM Quadlet left (the `frank-medium`
quadlet-era directory left the tree 2026-09-26).
The unit files and serve scripts validate their model/runtime artifacts before
starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
@@ -110,15 +120,16 @@ change in that service's inputs.
If the LLM handler fails after the target pull:
1. Inspect `journalctl -u frank-dragoon -n 200` or `journalctl -u frank-zealot
-n 200` (root units), or `journalctl --user -u frank-medium -n 200`
(rootless Quadlet).
2. Confirm the model files, the runtime binary and (for frank-medium) the
container image are present and readable by `dank`.
1. Inspect `journalctl -u frank-dragoon -n 200`, `journalctl -u frank-zealot
-n 200` or `journalctl -u frank-archon -n 200` (all three are root units);
frank-dragoon's engine boot ledger is in `/home/dank/logs/frank_dragoon.log`,
frank-archon's in `podman logs frank-archon` — not in the journal.
2. Confirm the model files, the runtime binary and (for frank-archon) the
digest-pinned container image are present and readable by `dank`.
3. Retry only the affected service:
```bash
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --service frank-dragoon
./scripts/deploy.sh dankclaw --service frank-zealot
```
serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md
+20 -10
View File
@@ -1,18 +1,22 @@
# frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
cutover — see [[frank-archon-radiance-qwen38-native-fp8]]; kept as the
rollback runbook: the image + orcarouter payload stay on disk until the
window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
cutover — see [[frank-archon-radiance-qwen38-native-fp8]], itself retired
2026-10-08; the payload live today (2026-10-18) is the publisher iq4r container,
[[frank-archon-radiance-qwen38-iq4r]]). Kept as the rollback runbook: the image
is still in dank's rootless storage, the orcarouter payload is not on disk any
more)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
on the host into the local image
`localhost/frank-archon-freetoken:pr587-77a7201` by
`scripts/docker/frank-archon-freetoken.Dockerfile`. Same port, same served
alias (`frank-archon`), same 131-shard payload
(`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`). Rollback:
on the host into the local image `localhost/frank-primary-freetoken:pr587-77a7201`
(that is the tag podman reports today — the 2026-10-10 rename never retagged it)
by `scripts/docker/frank-primary-freetoken.Dockerfile`, which survives only in
the rollback tag: `scripts/docker/` is not in the tree today. Same port, same
served alias (the wire id was `frank-primary` until the 2026-10-09 rename), same
131-shard payload (`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`, since
deleted from the host). Rollback:
git tag `frank-archon-vllm-shapeG-20261003` (the vLLM trio + its env lore),
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
@@ -30,11 +34,17 @@ deterministic pass — no per-boot estimator to fight.
## The build (on dankserver, rootless podman as dank)
```bash
git checkout frank-archon-freetoken-pr587-20261005 # scripts/docker/ exists only in the tag
cd /home/dank/serverAppConfigs
podman build -t frank-archon-freetoken:pr587-77a7201 \
-f scripts/docker/frank-archon-freetoken.Dockerfile scripts/docker/
podman build -t frank-primary-freetoken:pr587-77a7201 \
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
```
(The names above are the ones the tag actually carries —
`scripts/docker/frank-primary-freetoken.Dockerfile`, per
`git ls-tree frank-archon-freetoken-pr587-20261005`. The 2026-10-10 rename
swept the docs, not the tag or the image in storage.)
Laws baked into the Dockerfile (each one cost a failure):
- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md
+28 -18
View File
@@ -18,20 +18,26 @@ timestamp: 2026-09-30
# frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
> **RETIRED 2026-10-01.** Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8
> + MTP4 (commit `2339fae` trio). The DFlash2 drafter experiment is a
> **failure**: after five patches the drafter served and drafted (12.8 tok/s
> proposed, 65,536 ctx, both TP ranks capturing FULL dflash2 cuda graphs) but
> **acceptance pinned at 1.00-1.38 on every text class including verbatim
> repetition** — the signature of a mis-wired aux-hidden-state capture, not a
> weak drafter. At ~1.2 acceptance the 8-token verification tax makes decode
> 5x SLOWER than the 9.9-10.0 no-draft baseline (1.6-1.8 tok/s measured). The
> untested alternative capture semantics: post-layer output state
> (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream. Payload
> (170 GiB) + drafter (2.2 GiB) staged to
> `/home/dank/.AGENTTRASH/20261001-glm-purge/` pending purge approval.
> KV-cache work that survives: patches 02/03/04 in `scripts/patches/` are
> generic unifier/tailpin/grouping fixes, apply-verified against the image.
> **RETIRED 2026-10-01** (era 2026-09-30 → 10-01, matching the AGENTS.md Runbook
> Index). Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 — the
> trio as of commit `2339fae`, restored by `d91379c` (2026-09-30). frank-archon
> runs [[frank-archon-radiance-qwen38-iq4r]] (radiance) today, 2026-10-18. The
> DFlash2 drafter experiment is a **failure**: after five patches the drafter
> served and drafted (12.8 tok/s proposed, 65,536 ctx, both TP ranks capturing
> FULL dflash2 cuda graphs) but **acceptance pinned at 1.00-1.38 on every text
> class including verbatim repetition** — the signature of a mis-wired
> aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the 8-token
> verification tax makes decode 5x SLOWER than the 9.9-10.0 no-draft baseline
> (1.6-1.8 tok/s measured). The untested alternative capture semantics: post-layer
> output state (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream.
> Payload (170 GiB) + drafter (2.2 GiB) were staged to
> `/home/dank/.AGENTTRASH/20261001-glm-purge/` and purged since — on 2026-10-18
> Dankserver has no GLM tree under `/home/dank/models/` and `~/.AGENTTRASH` is
> empty. KV-cache work that survives: `glm53-dflash-04-kv-groups-draft.patch` and
> `glm53-dflash-04b-kv-registry-dflash.patch` are still in `scripts/patches/`;
> patch 03 (tailpin) was deleted at the rollback (`d91379c`) and patch 02 was
> never committed. None of these apply byte-exact today — the 2026-10-10 rename
> (`55f87fc`) rewrote the retired service names inside their context lines.
> **Question this page answers.** Can the fleet serve the 320B-class
@@ -695,12 +701,16 @@ newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
## Operations
```bash
# rollback to the pre-cutover qwen FP8 + MTP4 shape (payload and R9V state
# dirs are untouched on disk):
# rollback to the pre-cutover qwen FP8 + MTP4 shape. The trio is in git; the
# qwen FP8 payload is NOT on disk any more, so re-fetch it first. The R9V state
# dirs are still on the host (/home/dank/r9v-state/):
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-archon
# payload inventory (34 shards + 2 extras, weight-map index):
# payload inventory (34 shards + 2 extras, weight-map index) — in the GLM era
# this was scripts/download-frank-primary.sh (REPO=Intel/GLM-5.3-Flash-W4A16-AutoRound,
# EXPECTED_SHARDS=34, git show f2bcf84). The same-named script today fetches
# Qwen/Qwen3.8-Flash-Next-FP8 (scripts/download-frank-archon.sh:21):
bash scripts/download-frank-archon.sh --verify-only
# live telemetry (needs the profiler window; with_stack=false is mandatory —
@@ -794,5 +804,5 @@ toggling on the mini itself).
| state | where |
| --- | --- |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`, payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`; payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` was deleted from the host (2026-10-18: `/home/dank/models/` has `StillDeadcode`, `rad`, `tpl.jinja`) — rollback = re-fetch |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-archon-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
serverAppConfigs-docs/LLM/frank-archon-llamacpp-qwen38-flash-next-q8.md
+27 -18
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
description: ACTIVE frank-archon runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
description: RETIRED 2026-09-26 → 09-27 frank-archon runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment; superseded 2026-09-27, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
tags:
- frank-archon
- dankserver
@@ -21,7 +21,9 @@ timestamp: 2026-09-26
# frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
The ACTIVE dankserver LLM endpoint (:7770, alias **frank-archon**):
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood from
2026-09-26 to 2026-09-27 (retired; the port today serves
[[frank-archon-radiance-qwen38-iq4r]]):
unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
@@ -240,22 +242,29 @@ traffic mix with `frank-archon-expert-activity.py <pid>` (or
## Rollback
The lucebox DS4.1 material is fully retained on the host: image
`localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era
repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
[[frank-archon-lucebox-ds4]]). Restore = move the `.container` back,
uninstall the root unit (`systemctl disable --now frank-archon` — the
link, not the file; never `systemctl disable` a unit whose file must
survive), `daemon-reload`, start the user quadlet. The Qwen payload
(~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a
refetch.
The lucebox DS4.1 image `localhost/lucebox-rocm:gfx1201` is still in dank's
rootless storage; the 378 GiB payload is not (Dankserver's `/home/dank/models/`
holds only `StillDeadcode`, `rad` and `tpl.jinja` as of 2026-10-18), and the
quadlet-era repo files were deleted by `8500600` rather than trashed — there is
no `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
[[frank-archon-lucebox-ds4]]). Restore = recover the `.container` from
`git show 8500600^`, stop the root unit (`systemctl stop frank-archon`; the
fleet law forbids `systemctl disable` on these hosts — it removes the unit
entry, which here is the symlink into the checkout), `daemon-reload`, start the
user quadlet. The Qwen payload (~179 GiB) is no longer on disk either, so a
forward re-swap is a re-fetch plus a redeploy.
## Fleet context
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
b11030-mix binary family on gfx1151.
- [[frank-zealot-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
3090; the only audio-capable endpoint.
- The harness contract lives in `/home/blank/.omp/agent/models.yml`
(`llamacpp (dankserver)` provider, `contextWindow 262144`,
`maxTokens 65536`, `input: [text, image]`).
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — the era's Dankclaw :7770, the same
b11030-mix binary family on gfx1151 (retired 2026-09-28; frank-dragoon runs
[[frank-dragoon-flash-server-v2-native]] today).
- [[frank-zealot-gemma4-12b]] — the era's Dankclaw :7771, mainline llama.cpp on
the 3090, and then the fleet's only audio-capable endpoint. Both are history:
frank-zealot serves [[frank-zealot-llama-prism-bonsai2-pq20]] today and no
fleet endpoint takes audio (the audio path left with Gemma4 on 2026-10-01).
- The harness contract lives in `/home/blank/.omp/agent/models.yml`. The era's
`llamacpp (dankserver)` numbers (`contextWindow 262144`, `maxTokens 65536`,
`input: [text, image]`) are long gone — that provider reads
`contextWindow: 200000`, `maxTokens: 16384`, `input: [text, image]` as of
2026-10-18.
serverAppConfigs-docs/LLM/frank-archon-lucebox-ds4.md
+18 -11
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-archon Lucebox DeepSeek V4 Runbook
description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); payload and image retained on the host as the rollback target.
description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); the image is still in rootless storage but the payload is gone from the host, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
tags:
- frank-archon
- dankserver
@@ -21,9 +21,12 @@ timestamp: 2026-09-26
> [!warning] RETIRED 2026-09-26 (same day it went live)
> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
> served for hours; the rollback material (image `localhost/lucebox-rocm:
> gfx1201` and the 378 GiB payload) is retained on the host, and the
> quadlet-era files live in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/`.
> served for hours; the image `localhost/lucebox-rocm:gfx1201` is still in
> dank's rootless storage, but the 378 GiB payload was deleted from the host
> (`ls /home/dank/models/` on Dankserver, 2026-10-18: `StillDeadcode`, `rad`,
> `tpl.jinja` only), and the quadlet-era repo files were deleted by `8500600`
> (2026-09-26) — they survive only in git history, not in a `.AGENTTRASH`
> directory (`dankserver/.AGENTTRASH/` holds only `20260916-vllm-radiance`).
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
2026-09-26: the
@@ -61,8 +64,8 @@ mode here, not an error.
```
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
# Three local modifications (repo-tracked at
# dankserver/frank-archon/lucebox-gfx1201.patch):
# Three local modifications (tracked at the time as
# dankserver/frank-primary/lucebox-gfx1201.patch, deleted by 8500600):
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
# base already ships them in /opt/rocm; the apt repo does not serve
# them) and the three runtime math libs.
@@ -98,7 +101,8 @@ fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
## Deployed flags (2026-09-26)
From `dankserver/frank-archon/frank-archon-entrypoint.sh` (the unit
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (deleted by `8500600`;
recover it with `git show 8500600^:dankserver/frank-primary/frank-primary-entrypoint.sh`) — the unit
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
flag set and cannot express the DS4 flags):
@@ -126,20 +130,23 @@ metadata — the fleet contract is unchanged from frank-medium.
1. Image built and model fetched (gates in `deploy_llm_archon`).
2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
unit (`frank-medium`), waits for :7770 to free, installs
`dankserver/frank-archon/quadlet/frank-archon.container`, starts
`dankserver/frank-primary/quadlet/frank-primary.container` (also deleted by
`8500600`; recover it from `git show 8500600^`), starts
`frank-archon.service`, waits for health (1800 s start period — first
start page-ins the dense weights and builds the graphs).
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
call, and an Open WebUI completion.
4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
checkpoint payload stays on disk.
checkpoint payload did not stay — it is gone from the host (see Rollback).
## Rollback
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
systemctl --user start frank-medium`) and stop/mask frank-archon. The
`localhost/radiance-vllm:0.13.0-e060895` image and the
`/models/Qwen3.8-27B-PARO-MXFP6` checkpoint remain on disk.
`localhost/radiance-vllm:0.13.0-e060895` image is still in dank's rootless
storage; the `/models/Qwen3.8-27B-PARO-MXFP6` checkpoint is not — Dankserver's
`/home/dank/models/` holds only `StillDeadcode`, `rad` and `tpl.jinja` as of
2026-10-18, so this rollback needs a re-fetch of the payload.
## Verification (2026-09-26)
serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md
+19 -11
View File
@@ -3,26 +3,32 @@ title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
service: frank-archon
host: dankserver
port: 7770
status: active
status: retired
deployed: 2026-09-27
supersedes: "[[frank-archon-r9v-radiance-qwen38-uncensored]]"
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, q8]
---
# frank-archon — R9V radiance-vLLM Q8_0 (dankserver, ACTIVE)
# frank-archon — R9V radiance-vLLM Q8_0 (dankserver, RETIRED 2026-09-27 → 09-30)
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
The Dankserver LLM endpoint **was** the **R9V radiance-vLLM fork** (retired
2026-09-30, when [[frank-archon-glm53-flash-w4a16]] took the port; the engine
live today, 2026-10-18, is radiance with
[[frank-archon-radiance-qwen38-iq4r]])
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
/ `0000:07:00.0`) as `frank-archon` on **:7770**.
/ `0000:07:00.0`) as `frank-primary` on **:7770** (the `frank-archon` wire alias
arrived with the 2026-10-09 rename).
Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
direction. Rollback target: [[frank-archon-r9v-radiance-qwen38-uncensored]]
— the uncensored state dir (`/home/dank/r9v-state/uncensored/`, `ready:
true`), its frozen config (`restore-config-20260927.env`) and payload stay
on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.
direction. Rollback target at the time:
[[frank-archon-r9v-radiance-qwen38-uncensored]] — its state dir
(`/home/dank/r9v-state/uncensored/`, with `setup.json` and
`restore-config-20260927.env`) is still on the host as of 2026-10-18, but the
IQ4_XS payload itself is not (`/home/dank/models/` holds only `StillDeadcode`,
`rad`, `tpl.jinja`), so that flip now needs a re-fetch.
## Why the swap
@@ -269,9 +275,11 @@ quoting through ssh+zsh layers.
`scripts/frank-archon.env` `R9V_STATE_FILE` back at
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
(adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
uncensored state dir and payload are intact on disk.
- **Models on the wire**: served alias `frank-archon`; thinking channel
is `reasoning` (vLLM); tool parser `qwen3_coder`.
uncensored state dir is intact on disk; the IQ4_XS payload is not (it was
deleted in the 2026-10-17 cleanup), so this rollback re-fetches first.
- **Models on the wire**: served alias `frank-primary` (the StarCraft wire
names arrived 2026-10-09, `7c77326`); thinking channel is `reasoning` (vLLM);
tool parser `qwen3_coder`.
## Known costs (accepted at cutover)
serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-uncensored.md
+14 -8
View File
@@ -13,23 +13,29 @@ tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, retired]
# frank-archon — R9V radiance-vLLM (dankserver, RETIRED)
> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover,
> [[frank-archon-r9v-radiance-qwen38-q8]]. Kept as the rollback runbook:
> the uncensored state dir, frozen config and payload are intact on disk;
> the rollback procedure lives in that page's Ops section.
> [[frank-archon-r9v-radiance-qwen38-q8]] (itself retired 2026-09-30; the live
> frank-archon runbook today, 2026-10-18, is
> [[frank-archon-radiance-qwen38-iq4r]]). The uncensored state dir and frozen
> config are still on disk (`/home/dank/r9v-state/uncensored/` — `setup.json`,
> `restore-config-20260927.env`); the IQ4_XS **payload is not** — it was deleted
> in the 2026-10-17 cleanup. The rollback procedure lives in the Q8_0 page's Ops
> section, with a re-fetch as step one.
The Dankserver LLM endpoint is the **R9V radiance-vLLM fork**
The Dankserver LLM endpoint **was** the **R9V radiance-vLLM fork**
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
`0000:03:00.0` / `0000:07:00.0`) as `frank-archon` on **:7770**.
`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770** (the
`frank-archon` wire alias arrived 2026-10-09).
Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
Rollback target: [[frank-archon-llamacpp-qwen38-flash-next-q8]] — the
llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
wiring; host unit was the relinked one) and the GGUF payload stays on disk,
so a rollback is a redeploy.
wiring; host unit was the relinked one); the GGUF payload is no longer on
disk (Dankserver's `/home/dank/models/` holds only `StillDeadcode`, `rad`,
`tpl.jinja` as of 2026-10-18), so that rollback is a re-fetch plus a redeploy.
## Production shape (repo-tracked)
@@ -38,7 +44,7 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs |
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary` (the wire id became `frank-archon` on 2026-10-09), container `frank-archon-r9v`, GPU BDFs |
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md
+22 -11
View File
@@ -1,6 +1,10 @@
# frank-archon — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
**Status: RETIRED 2026-10-05 → 2026-10-08** — replaced by the Swift1.5 w4dn8
conversion ([[frank-archon-radiance-swift15-w4dn8]]), itself retired 2026-10-15;
the payload live today (2026-10-18) is the publisher iq4r container,
[[frank-archon-radiance-qwen38-iq4r]] (`scripts/frank-archon.env:110-111`). ·
**Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
libr4d kernels for gfx1201, no Python in the serving path) driving the
@@ -10,9 +14,14 @@ digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
(`frank-archon.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
same served alias (`frank-archon`), same container identity
(`frank-archon` = the unit name). Rollback: git tag
`frank-archon-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
`localhost/frank-archon-freetoken:pr587-77a7201`, payload
`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`.
`frank-archon-freetoken-pr587-20261005` (FreeToken trio + Dockerfile; the
Dockerfile lives in the tag as `scripts/docker/frank-primary-freetoken.Dockerfile`
— `scripts/docker/` is not in the tree today), image
`localhost/frank-primary-freetoken:pr587-77a7201` (that is the tag on the host —
the 2026-10-10 rename never retagged it), payload
`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`, which is **no longer on
disk** (`ls /home/dank/models/` on Dankserver 2026-10-18: `StillDeadcode`, `rad`,
`tpl.jinja` only).
## Why radiance
@@ -126,9 +135,10 @@ serve script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=140000`) refuses to
start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
verified read-back; raising at runtime is harmless and allocates nothing).
PERSISTENCE: the sysfs write does not survive reboot — add
`ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
reboot.
PERSISTENCE: the sysfs write does not survive reboot, and the kernel-cmdline
`ttm.pages_limit=52428800` was never added — the unit now raises the knob at
every boot instead (`scripts/frank-archon.service:44`,
`ExecStartPre=+…/scripts/ttm-preflight.sh --raise`).
**Per-rank VRAM** at TP2: expert slice 56.4 GiB total across VRAM + host
pool; trunk ~3.75 GiB/rank; token embedding vocab-parallel (~1.18 GiB/rank
@@ -280,12 +290,13 @@ Post-cutover checklist (each is observable, not assumed):
| Layer | Path |
|---|---|
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-archon-freetoken-pr587-20261005 -- scripts/frank-archon.env scripts/frank-archon.service scripts/serve-frank-archon.sh scripts/docker/ tests/frank_archon_unit.bats` + redeploy; image `localhost/frank-archon-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-archon-freetoken-pr587-20261005 -- scripts/frank-archon.env scripts/frank-archon.service scripts/serve-frank-archon.sh scripts/docker/ tests/frank_archon_unit.bats` + redeploy; the image in storage is `localhost/frank-primary-freetoken:pr587-77a7201`, and the payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` must be re-fetched — it is gone from the host |
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-archon-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` + `--host-pool-mib 12288` — the documented fallback if the native container ever proves unbootable |
| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` — **this became the live payload on 2026-10-15** ([[frank-archon-radiance-qwen38-iq4r]]), deployed with `--host-pool-mib 44928` (`scripts/frank-archon.env:184`), not the 12,288 MiB first sketched here |
Retire the FreeToken-era orcarouter payload to `~/.AGENTTRASH/` only after
the rollback window closes (trash policy, never `rm`).
The FreeToken-era orcarouter payload has since been deleted from the host
(`ls /home/dank/models/` on Dankserver, 2026-10-18) — there is nothing left to
trash, and the FreeToken rollback needs a re-fetch.
## Links
serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md
+7 -4
View File
@@ -547,14 +547,16 @@ full-height conversations at once, and answer each one exactly. It does.
### Wire rename + the bench drop-in trap (2026-10-09)
`FRANK_ARCHON_SERVED_NAME` moved `frank-archon` → **frank-archon**; the
service, unit, scripts and env file keep the old name. Radiance is the only
`FRANK_ARCHON_SERVED_NAME` moved the served id `frank-primary` →
**frank-archon** (commit `7c77326`, 2026-10-09); the service, unit, scripts and
env file kept `frank-primary` until the 2026-10-10 rename (`55f87fc`) brought
them to `frank-archon` too. Radiance is the only
fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
**requested** model name in chat responses — so a chat probe that returns 200
proves nothing about the served id. Always read `/v1/models`.
The first deploy after the edit came up green and `/v1/models` still said
`frank-archon`. Cause: a campaign leftover drop-in
`frank-primary`. Cause: a campaign leftover drop-in
`/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09)
overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the
staging worktree `/home/dank/serverAppConfigs-swift15`. Every
@@ -563,7 +565,8 @@ staging tree: green health gate, old configuration, silently. Verified before
removing: the staging `serve-frank-archon.sh` and `frank-archon.service`
are byte-identical to the checkout, its `frank-archon.env` differed only by
the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`.
drop-in was staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`
(that dated dir is gone — the trash dir is empty on Dankserver as of 2026-10-18).
After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
26071/3543).
serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md
+44 -1
View File
@@ -94,6 +94,9 @@ the pool: **two long conversations can no longer share the endpoint** (two of
≤100k can). That is acceptable here because frank-dragoon carries the `slow`
(medium-thinking) role and the `security-reviewer` subagent, not the
interactive workhorse (that is frank-archon).
Frank lifted exactly that concurrency limit on 2026-10-18 — see *Pool
doubled for slot parity* below; that lift is COMMITTED but not yet deployed,
so the sentences above remain the live truth until the next redeploy.
**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot
printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
@@ -112,6 +115,43 @@ that long prompts read.
`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk
stays pinned. Disk-backing the weights is not a lever on this format.
## Pool doubled for slot parity — committed 2026-10-18, awaiting redeploy
`HALOGEN_KV_POOL_POSITIONS` **200,704 → 401,408** (commit `2b12716`), Frank's
call: frank-dragoon's total in-flight context was **half** frank-zealot's —
the llama.cpp endpoint runs one 401,408-position pool split into two 200,704
conversations, while dragoon ran one 200,704-position pool behind its two
slots, so one maximum-length request filled it and any second long
conversation had to stay under ~100k (commit message;
`scripts/frank-dragoon.env:26-27`, `:183-186`).
- **The per-request ceiling does NOT move.** `HALOGEN_CTX` stays 200,704, the
native wall stays 262,144 (`HALOGEN_ROPE_YARN` law), and `models.yml`
`contextWindow` stays **155,648** — the pool is the shared budget behind the
slots, not the window. The serve seam enforces `HALOGEN_CTX ≤
HALOGEN_KV_POOL_POSITIONS ≤ HALOGEN_KV_SLOTS × HALOGEN_CTX`
(`scripts/serve-frank-dragoon.sh:85-95`); 401,408 is `slots × ctx`, the top
of the legal band, and exactly frank-zealot's pool number — parity, not
approximation.
- Expected reservation **~10.8 GiB** at ~28 KiB/position (preflight formula,
`scripts/frank-dragoon.env:70-72`); the measured points beside it are 5.5
GiB at 200,704 (boot ledger 2026-10-18), 7.2 GiB reserved / 7.4 GiB billed
at 262,144, and 14.6 GiB at the 2026-10-06 shape's 524,288
(`scripts/frank-dragoon.env:49-52`, `:74-76`).
- That RAM comes out of the n-gram table's page cache — FLAGS.md law: RAM the
pool takes is RAM that table loses.
**Deploy state as of 2026-10-18: NOT yet deployed.** The live container
(booted 2026-10-18 01:25) still runs `HALOGEN_KV_POOL_POSITIONS=200704` —
verified in its `flash_serve` `/proc/<pid>/environ` and in the host-deployed
`frank-dragoon.env` — and the 2026-10-18 boot ledger reserves `the KV pool,
200704 positions (about 5.5 GiB)` with `74.1 GiB in all`
(`/home/dank/logs/frank_dragoon.log`). Until the next
`./scripts/deploy.sh dankclaw --service frank-dragoon`, the boot ledger keeps
printing `2 slots over ONE 200704-position KV pool`; the first boot at the
new pool must show `ONE 401408-position KV pool` and the ~10.8 GiB
reservation — read that line before quoting 401,408 as live anywhere.
## Measured A/B — same battery, both boots (2026-10-17)
Behavior probes ran against the live service (tagged `swift15` / `v2abl`);
@@ -267,6 +307,9 @@ restart; `webui.db.bak-20261017b` holds the pre-cutover row).
**What the cut cost:** with pool == ctx one maximum-length request fills the
pool, so two 200k conversations never coexist (two of ≤100 k do), and the
harness window is now 155,648 — the 180 k compaction target of the 262,144 era
is unreachable because the window itself is smaller. **What it bought:** 1.7
is unreachable because the window itself is smaller. (The pool == ctx
concurrency limit is superseded — on redeploy only — by *Pool doubled for slot
parity* above; the window math stands, the pool never moves the ceiling.)
**What it bought:** 1.7
GiB back as page cache for the 47.68 GiB n-gram table (9.1 GiB against the
2026-10-06 shape's 524,288 pool), and a smaller prompt-cache entry.
serverAppConfigs-docs/LLM/frank-npu.md
+31 -17
View File
@@ -15,11 +15,17 @@ timestamp: 2026-09-11
# frank-npu — the retired FLM NPU endpoint
> **RETIRED 2026-09-11.** The endpoint is gone by operator decision, together
> with the IOMMU configuration it required — see
> [[dankclaw-iommu-translation-off]]. The service unit, env file and serve
> script are staged in `serverAppConfigs/.AGENTTRASH/` (suffixed `.flm-npu`),
> and the host unit is stopped and masked.
> **RETIRED 2026-09-11 by operator decision** — and still retired: there is no
> NPU endpoint service on Dankclaw today. The IOMMU configuration it required
> has since been reversed twice — see
> [[dankclaw-iommu-translation-off]]: the live 2026-10-18 posture is
> `cat /proc/cmdline` → `… ro quiet splash iommu=pt …` (no `amd_iommu` token),
> `amdxdna` is bound and `/dev/accel/accel0` exists, so the hardware is
> reachable again — revival is tracked as Gitea issue #3. The service unit, env
> file and serve script are staged in `serverAppConfigs/.AGENTTRASH/`
> (suffixed `.flm-npu`), and the host unit is still masked
> (`systemctl is-enabled frank-npu` → `masked`, read 2026-10-18). Port 7771
> belongs to [[frank-zealot-llama-prism-bonsai2-pq20]] today.
The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
@@ -56,9 +62,16 @@ upstream project on this exact silicon. That single flag is the whole decision:
NPU sits in IOMMU group 26 `identity` with PASID width 10, `/dev/accel/accel0`
is `crw-rw---- root:render 261,0` and `dank` is in `render` — so a revival
quadlet needs `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`, the
ComfyUI `/dev/kfd` pattern. Kernel versions to avoid: **6.18–6.18.7** disables
SVA outright. Full census in [[dankclaw-iommu-translation-off]]
§ *2026-10-18 execution*.
ComfyUI `/dev/kfd` pattern. Re-read live 2026-10-18: `lspci -k -s c6:00.1` →
`Kernel driver in use: amdxdna`; `journalctl -k -b` → `Initialized
amdxdna_accel_driver 0.7.0 for 0000:c6:00.1` and `Load firmware
amdnpu/17f0_11/npu_7.sbin`; `/sys/kernel/iommu_groups/26/type` → `identity`;
`ls -l /dev/accel/accel0` → `crw-rw---- 1 root render 261, 0` (created
2026-10-18 01:25); `id -nG dank` contains `render`. Kernel versions to avoid:
**6.18–6.18.7** disables SVA outright *(unverified 2026-10-18 — upstream
reports place the amdxdna/Strix-Halo breakage at 6.18.4+ and fixed by 6.19.2,
not at a clean 6.18–6.18.7 window)*. Full census in
[[dankclaw-iommu-translation-off]] § *2026-10-18 execution*.
- The operator took the vendor's prefill figure on the September evidence, so the
NPU endpoint went — and it stays retired because nobody has asked for a model
on it, not because the hardware is unreachable.
@@ -71,11 +84,11 @@ image path is halogen's vision tower on `frank-dragoon`.
| consumer | setting | now |
|---|---|---|
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-dragoon` (halogen vision tower) |
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-zealot` |
| harness (`~/.omp/agent/models.yml`) | `modelRoles.vision` | `llamacpp (dankclaw)` / `frank-dragoon` |
| harness | `modelRoles.tiny` | `llamacpp (dankserver)` / `frank-zealot` |
| harness | `modelRoles.task` | `llamacpp (dankclaw)` / `frank-dragoon:medium` |
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-dragoon` (halogen vision tower) *(unverified 2026-10-18: `/opt/hermes` is gone from Dankclaw, so the live `config.yaml` could not be read)* |
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-zealot` *(unverified 2026-10-18: same)* |
| harness (`~/.omp/agent/config.yml`) | `modelRoles.vision` | moved again: `llamacpp (dankclaw-3090)/frank-zealot:xhigh` (`config.yml:6`) |
| harness | `modelRoles.tiny` | `llamacpp (dankclaw-3090)/frank-zealot:off` (`config.yml:10`) |
| harness | `modelRoles.task` | `llamacpp (dankclaw-3090)/frank-zealot:off` (`config.yml:11`) |
The `flm (dankclaw-npu)` provider block was removed from the harness model
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
@@ -84,10 +97,10 @@ config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
| Field | Value |
|---|---|
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`) on the XDNA2 NPU |
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`; `dpkg -l fastflowlm` on 2026-10-18 still reports `fastflowlm 1.0.1`) on the XDNA2 NPU |
| Service | `frank-npu.service` (systemd **system** unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
| Files (archived) | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
| Port | `7771` on `0.0.0.0` |
| Port | `7771` on `0.0.0.0` (that port is frank-zealot's today) |
| Power mode | `FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything |
| Context | `FLM_CTX_LEN=131072` (128K, the verified maximum on this box) |
| Vision/audio | `FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions` |
@@ -132,8 +145,9 @@ The same root cause explained the older
consequence of IOMMU-off, not a GPU fault.
**The standing rule this produced — "never set `iommu=off` on this host again" —
has been superseded**: the NPU it protected is retired by decision, and
`amd_iommu=off` is now intentional. See
was superseded twice**: `amd_iommu=off` was the intentional posture from
2026-10-16 to 2026-10-18, and the host has since flipped back to `iommu=pt`
(live `cat /proc/cmdline`, 2026-10-18: `… ro quiet splash iommu=pt …`). See
[[dankclaw-iommu-translation-off]] for the current posture and how to revert it.
## Client wiring (historical)
serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md
+20 -2
View File
@@ -13,6 +13,16 @@ This page records the full redo of the inference fleet on 2026-08-17.
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
The old scripts moved to `old/scripts/` in this repository.
> **HISTORY — the topology below is two fleet generations dead.** Today the
> fleet is three endpoints: [[frank-archon-radiance-qwen38-iq4r]]
> (`192.168.0.235:7770`, radiance 1.0.13, 2×R9700 TP=2),
> [[frank-dragoon-flash-server-v2-native]] (`192.168.0.143:7770`, halogen
> flash-server 0.16.2 on the iGPU) and
> [[frank-zealot-llama-prism-bonsai2-pq20]] (`192.168.0.143:7771`, llama.cpp
> PrismML fork build 10754 on the RTX 3090 — the card that sat in Dankserver
> until 2026-09-15). No vLLM and no ROCm endpoint serves anything; there is no
> pipeline parallelism and no NPU endpoint.
## New Topology
| Service | Host | Runtime | Model | Port |
@@ -20,6 +30,12 @@ The old scripts moved to `old/scripts/` in this repository.
| frank-dragoon | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-zealot | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
Names as the 2026-08-17 commit wrote them: `frank-default-large` on Dankserver
(`git show d9af7f1:scripts/frank-default-large.env:20,22` → `ALIAS=frank-default-large`,
`LISTEN_PORT=7770`) and `frank-default-small` on Dankclaw
(`git show d9af7f1:scripts/frank-default-small.env:16,18` → `ALIAS=frank-default-small`,
`LISTEN_PORT=52625`); the Protoss ids date from 2026-10-06 → 10-10.
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
tensor split, no NPU.
@@ -58,7 +74,8 @@ tensor split, no NPU.
## Context Math (frank-dragoon, RTX 3090)
KV per token: 16 attention layers x 4 KV heads x 256 dim.
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
K q8_0 = 1088 B per layer (4 KV heads × 256 dim × 1.0625 B), V f16 = 2048 B per
layer, ×16 layers = 50176 B/token total.
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
(CUDA context, compute graph, scratch, draft KV) - headroom.
@@ -150,7 +167,8 @@ on gfx1151; newer vLLM releases may already include it.
- `models.json` (pi): two providers — `llamacpp (dankserver)` at
http://192.168.0.235:7770/v1 and `vllm (dankclaw)` at
http://192.168.0.143:52625/v1.
http://192.168.0.143:52625/v1. (The harness config today has three providers
and no vLLM: `~/.omp/agent/models.yml:14,94,173`.)
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
Dankserver endpoint.
serverAppConfigs-docs/LLM/frank-zealot-benchmark.md
+12 -1
View File
@@ -13,9 +13,20 @@ tags:
# frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning
> **HISTORY (runs 2026-08-13 and 2026-08-25, Dankserver's RTX 3090).** That
> card was pulled from Dankserver on 2026-09-15 (`lspci | grep -ci nvidia` → 0
> there on 2026-10-18 — [[Dankserver System Specs]]) and today drives
> frank-zealot on **Dankclaw**: [[frank-zealot-llama-prism-bonsai2-pq20]]
> (`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off, no
> `--n-cpu-moe` — the model fits the card outright). What survived the payloads:
> decode is ubatch-invariant, CUDA prefill climbs monotonically to ubatch 2048,
> and the logical `--batch-size` is neutral — the live env still ships
> 4096/2048 (`scripts/frank-zealot.env:44-45`).
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
`--n-cpu-moe` and `--ubatch-size` values for `frank-zealot`.
`--n-cpu-moe` and `--ubatch-size` values for the llama.cpp rung then served on
that card (the `frank-zealot` name dates from 2026-10-06/10-09).
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md
+29 -15
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
description: "The active frank-zealot runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
description: "The RETIRED frank-zealot runbook (2026-09-17 → 10-01) — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-zealot
@@ -16,20 +16,26 @@ timestamp: 2026-09-17
# frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
> **ACTIVE since 2026-09-17.** Replaces [[frank-zealot-qwen38-gsq-rco]]
> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment, also the
> rollback target). Same port (7771), same alias (`frank-zealot`), same
> client contract surface.
> **RETIRED 2026-09-17 → 2026-10-01** (AGENTS.md Runbook Index). It replaced
> [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) and
> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment).
> Same port (7771), same alias (`frank-small` — this endpoint only carried the
> `frank-zealot` name from 2026-10-06/10-09), same client contract surface.
> :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML
> fork build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP
> off); the audio input this page documents left the fleet with this payload.
## Identity
- **Endpoint:** `frank-zealot` at `192.168.0.143:7771` (Dankclaw).
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) —
`git show e23df8b:scripts/frank-small.env` → `ALIAS=frank-small`.
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
collapses prefill ~12x on CUDA.
collapses prefill ~12x on CUDA. Host check 2026-10-18: that binary reports
`0.4.1-dev (build 1, commit 37b53fd)` and its `CMakeCache.txt` reads
`GGML_CUDA_FA_ALL_QUANTS:BOOL=ON`.
- **Service shape:** root systemd unit `scripts/frank-zealot.service`
(`User=dank`), launched by `scripts/serve-frank-zealot.sh` with tunables from
`scripts/frank-zealot.env`.
@@ -78,25 +84,29 @@ the >12.8 GiB floor does not apply.
(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
52M params), which also carries an audio encoder. The fleet's image path
lives here. Audio works: `input_audio` content parts transcribe verbatim
(verified 2026-09-17 with a 16k mono WAV).
(verified 2026-09-17 with a 16k mono WAV). Fleet-wide audio ended with this
payload: no fleet endpoint takes audio input today.
## Payload
The main GGUF, the BF16 mmproj, and the MTP sidecar live under
`/home/dank/models/Gemma4-12B-QAT`. The serve script refuses to start if any
The main GGUF, the BF16 mmproj, and the MTP sidecar lived under
`/home/dank/models/Gemma4-12B-QAT`; they were staged to
`.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on 2026-10-02 and
that stage dir is gone as of 2026-10-18, so the payload needs a re-fetch. The serve script refuses to start if any
is missing (a missing file is an unfinished download, and the unit's restart
policy would otherwise turn it into a crash loop that reads like a runtime
fault).
## Client contract
- **Alias:** `frank-zealot` (the `id` at `/v1/models`).
- **Alias:** `frank-small` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned even though this build defaults
it on, so a future build that flips the default cannot silently take tools
away).
- **Vision + audio:** supported.
- **Vision + audio:** both supported in this era; audio is gone fleet-wide
today (`input: [text, image]` — `~/.omp/agent/models.yml:214`).
- **Sampling:** greedy by default (fleet contract); a client that sends a
temperature gets its own, and the harness applies per-model overrides from
`models.yml`.
@@ -110,8 +120,12 @@ pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
## Rollback
Payload and runtime for [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-zealot-vllm-gemma4]]
is the same-day vLLM experiment (also a rollback target).
Qwen3.8-27B-GSQ-RCO IQ3_S) are **not on disk any more** (the 2026-10-18 census
of `/home/dank/models/` has no GSQ-RCO directory and the era fetch script is
not in the tree) — that rollback re-downloads from ISTA-DASLab.
[[frank-zealot-vllm-gemma4]] is the same-day vLLM experiment; its weights are
gone too, though its derived image is still in rootless storage
(`localhost/vllm/vllm-openai:frank-small`, id `8dad46d79c46`).
# Citations
serverAppConfigs-docs/LLM/frank-zealot-llama-gemma4-q4km-200704.md
+11 -6
View File
@@ -1,9 +1,13 @@
# frank-zealot — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
**RETIRED 2026-10-07** — same day it went live. Superseded by
**RETIRED 2026-10-07** — same day it went live (AGENTS.md Runbook Index).
Superseded by
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
2 × 200,704 with exact KV); the payload stays at
`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the
2 × 200,704 with exact KV); :7771 today runs
[[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off). The payload
stays at `/home/dank/models/Gemma4-26B-A4B/` as the rollback (all three files
present in the 2026-10-18 host census). This page keeps the
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
the drafter-cost boots, both of which outlive the model.
@@ -12,8 +16,8 @@ the drafter-cost boots, both of which outlive the model.
|File|Path|Bytes|
|---|---|---|
|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,796,015,520 on disk; the engine reports 16,780,192,888|
|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|~1.195 GiB|
|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|~0.252 GiB|
|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|1,194,827,776 B on disk (2026-10-18) = 1.11 GiB|
|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|251,937,728 B on disk (2026-10-18) = 0.23 GiB|
Repo: `HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP`
(base `google/gemma-4-26B-A4B-it`). Downloaded 2026-10-07 by plain
@@ -81,7 +85,8 @@ costs 1,041 + 106 = **1,147 MiB**, so the f16-K raise is **+2,161 MiB**. The
`--cache-type-v` flag is honoured, not inert.
The llama.cpp master build at commit `5fc4f3c` supports the arch
(`src/models/gemma4.cpp`, `LLM_ARCH_GEMMA4 = "gemma4"`). Note: `grep -c`
(`src/models/gemma4.cpp` and `LLM_ARCH_GEMMA4 = "gemma4"` — both present in
`/home/dank/GitPackages/llama-master` as read on 2026-10-18, `src/llama-arch.cpp:59`). Note: `grep -c`
arch names on the packed binary returns 0 for EVERY arch (including the
`qwen3_5` it serves) — check the source tree, not the binary.
serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q5km.md
+17 -8
View File
@@ -16,13 +16,19 @@ timestamp: 2026-10-03
# frank-zealot — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
> **RETIRED 2026-10-03** — superseded by
> **RETIRED 2026-10-02 → 2026-10-03** — superseded by
> [[frank-zealot-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
> decode is the measured price of the higher rung on this bandwidth-bound
> card. This file remains the rollback reference and the owner of the
> measured boot-pair table.
> measured boot-pair table. :7771 today runs
> the PrismML 27B era of [[frank-zealot-llama-prism-bonsai2-pq20]] (build 10754,
> 2 × 200,704 q8_0 KV, MTP off) — retired 2026-10-17 → 10-18 as a rollback
> target; the active row today is build **10777** on
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 3 × 262,144 q8_0 KV, MTP off by
> measurement (AGENTS.md Runbook Index). The Q5_K_M
> payload is still on disk (`stat -c %s` 2026-10-18 → 6876124704).
> Its own history: one quant step down from [[frank-zealot-llama-mimo-q6k]]
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
@@ -36,7 +42,7 @@ timestamp: 2026-10-03
|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`git show e5893dc:scripts/frank-small.env` → `ALIAS=frank-small`; `frank-zealot` only from 2026-10-06/10-09)|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-zealot.sh`|
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
@@ -102,9 +108,11 @@ The per-slot window is the whole price, in both directions:
it reaches the ceiling instead of discovering it.
- **frank-dragoon's tallest pre-compaction prompt now fits here.** Compaction
for frank-dragoon sessions runs on this endpoint (`compactionModel` in
`models.yml`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
`models.yml`, still true 2026-10-18 at `~/.omp/agent/models.yml:107`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
tokens to spare. The 2026-10-02 window trim had restored the fit for
tokens to spare (2026-10-18: frank-dragoon's `contextWindow` is 155,648 —
`~/.omp/agent/models.yml:131` — so its compaction input is smaller still).
The 2026-10-02 window trim had restored the fit for
frank-archon's smaller 140,492-token trigger; the 2026-10-03 reslice widens
the fit to frank-dragoon itself. Nothing routes compaction automatically —
the point is the ceiling no longer forbids it.
@@ -137,7 +145,8 @@ The per-slot window is the whole price, in both directions:
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
ghost), and the cross-model DFlash draft
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf` — no longer on
disk: the 2026-10-18 census of `/home/dank/models/` has no `DFlash-Qwen35/`) is an
optimization candidate, not a default — acceptance has to be measured on this
traffic first. The bats seam fails if a draft flag ships unannounced.
@@ -146,10 +155,10 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|Target|How|
|---|---|
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-zealot.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — the deployed rung **2026-10-03 → 10-05** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_ZEALOT_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and **that stage dir no longer exists** (`ls /home/dank/.AGENTTRASH/`, 2026-10-18), so re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
## When it misbehaves
serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k-2x200704.md
+8 -5
View File
@@ -16,21 +16,24 @@ timestamp: 2026-10-03
> **RETIRED 2026-10-05 (evening).** Superseded by
> [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). :7771 today runs
> [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
> `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off). Two facts for the trail:
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
> contract Frank set here (f16 K + f16 V, "no further quantization on this
> endpoint") was superseded the same evening by the Swift swap's explicit
> q8_0/q8_0 order. This page stays as the rollback and the measured
> boot-pair record. Payload on disk: `/home/dank/models/MiMo-Q6K/`.
> boot-pair record. Payload on disk (`stat -c %s`, 2026-10-18: 7793710624):
> `/home/dank/models/MiMo-Q6K/`.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-zealot`, `/v1` OpenAI surface |
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small` on the wire in this era (`git show e5893dc:scripts/frank-small.env` → `ALIAS=frank-small`; the id became `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09), `/v1` OpenAI surface |
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (f16 KV needs no flash-attn option; the disk build's `CMakeCache.txt` reads `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree has no CUDA-side guard for the option — `ggml/src/ggml-cuda/fattn.cu:438-449`) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
@@ -149,7 +152,7 @@ so the harness compacts before llama.cpp answers a longer prompt with a 400.
|---|---|
|Q5_K_M at the same shape|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — the 2026-10-02 staging dir `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` is **gone** (`ls /home/dank/.AGENTTRASH/` on 2026-10-18 lists only the 20261005/06/07/10/17 batches), so this arm needs `old/download-frank-zealot-sglang-int8.sh`; the `sglang-cu128` venv is still on disk|
## When it misbehaves
serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k.md
+17 -10
View File
@@ -19,17 +19,20 @@ timestamp: 2026-10-01
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
> target.
> target (on disk 2026-10-18, `stat -c %s` → 7793710624). :7771 today runs
> [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
> `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704, q8_0 K + q8_0 V, MTP off).
>
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
> exposes per-tensor. That contract is what the current deployment kept.
> exposes per-tensor. The per-tensor lever stayed; this pairing did not — the
> live endpoint bills q8_0 K + q8_0 V (`scripts/frank-zealot.env:55-56`).
## Identity
|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`ALIAS` reached `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09, commit `7c77326`)|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
@@ -68,15 +71,17 @@ Frank's KV contract: **K unquantized (f16), V `q8_0`** —
(measured — this is not a fault).
- Tools: `--jinja` + server-side tool parsing, verified
`tool_calls` round-trip 2026-10-01 23:0x.
- **Text-only now.** The GGUF carries no vision projector; the
in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
`input: [text]`.
- **Text-only in this era.** The GGUF carries no vision projector; the
in-checkpoint tower existed only in the SGLang safetensors, so `models.yml`
carried `input: [text]`. Vision returned with the next payloads: the endpoint
is `input: [text, image]` today (`~/.omp/agent/models.yml:214`).
## No spec decode (yet)
The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
Q8_0) is an optimization-loop candidate, not a default — `--spec-type
Q8_0 — that path is no longer on disk; the 2026-10-18 census of
`/home/dank/models/` has no `DFlash-Qwen35/`) is an optimization-loop candidate, not a default — `--spec-type
draft-dflash` needs acceptance measured on this traffic first. The bats
seam fails if a draft flag ships unannounced.
@@ -92,9 +97,11 @@ cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
cmake --build build-cuda -j 16 --target llama-server
```
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
stay on disk as the rollback: repoint the unit's serve script and env from
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) is still on disk; the
INT8 AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
is not — it was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`
on 2026-10-02 and that stage dir is gone as of 2026-10-18, so this rollback
re-fetches with `old/download-frank-zealot-sglang-int8.sh`: repoint the unit's serve script and env from
git history (`git show frank-dragoon-gufo-q4kxl-20261001` era is BEFORE this;
use the parent of the 2026-10-01 night swap commit).
serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md
+23 -6
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
description: "The ACTIVE frank-zealot runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (RETIRED, rollback target)"
description: "The frank-zealot runbook of 2026-10-07 → 2026-10-17 (RETIRED — rollback target) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
tags:
- llm
- frank-zealot
@@ -14,7 +14,22 @@ timestamp: 2026-10-07
# frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
Live since 2026-10-07, replacing the
**RETIRED 2026-10-07 → 2026-10-17** (AGENTS.md Runbook Index), superseded by
[[frank-zealot-llama-prism-bonsai2-pq20]] (27B PQ2_0, 2 × 200,704 q8_0 KV, build
10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771
today is llama.cpp PrismML build **10777** on
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
measurement (AGENTS.md Runbook Index active row; `scripts/frank-zealot.env`).
Transient 2026-10-18: :7771 is inside a bench window — the unit is `failed`
since 03:14 and a hand-booted process answers the port; restore with
`./scripts/deploy.sh dankclaw --service frank-zealot`.
This era is the rollback target: the payload is still on disk
(`stat -c %s /home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf`
→ 7359259008, read 2026-10-18) with its master CUDA build. Same census: the
10754 rollback payload `Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` is
present at 7206168928 with its Q8_0 mmproj at 629246976, and
`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2675410720 —
nothing this era or its successors need has been purged. It replaced the
[[frank-zealot-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
@@ -26,7 +41,7 @@ conversation slot and unquantized KV back.
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` — `--version` on the disk build reads `0.5.0-dev (build 1, commit 5fc4f3c)` (built 2026-10-01); its `CMakeCache.txt` says `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree carries no CUDA-side guard for the option (`ggml/src/ggml-cuda/fattn.cu:438-449` dispatches Q4/Q5 KV unconditionally), so quantized KV never required it; `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
@@ -98,7 +113,9 @@ safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
**past** its window). Weights load in ~6 s from page cache; `/health` answers
`{"status":"ok"}` and `/v1/models` serves the alias `frank-zealot`.
**Wire rename (2026-10-09):** `ALIAS` moved `frank-zealot` → `frank-zealot`,
**Wire rename (2026-10-09):** `ALIAS` moved `frank-secondary` → `frank-zealot`
(commit `7c77326`; the service-file rename to the `frank-zealot.*` names
followed 2026-10-10, commit `55f87fc`),
so the served id matches the harness model id and the family-facing name.
Consumers re-keyed in the same change: OpenWebUI (`model` row, `access_grant`
row, `openai.api_configs` index-2 `model_ids`) and `models.yml` (already
@@ -120,7 +137,7 @@ The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
thinking" is satisfied by the 200,704 slot.
**Compaction hub**: `models.yml` routes `compactionModel` for all three
endpoints here. With `--parallel 2` a compaction request no longer serializes
endpoints here — still true 2026-10-18 (`~/.omp/agent/models.yml:25,107,185` all name `llamacpp (dankclaw-3090)/frank-zealot`). With `--parallel 2` a compaction request no longer serializes
behind a live generation (the Gemma4 era's `--parallel 1` law), and the
200,704 slot still fits the worst-case compaction input (~133k tokens).
serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md
+24 -9
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
description: "The active frank-zealot runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
description: "The frank-zealot runbook of 2026-10-05 → 2026-10-07 (RETIRED — rollback target; live from 2026-10-05 evening, renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
tags:
- llm
- frank-zealot
@@ -17,8 +17,21 @@ timestamp: 2026-10-06
# frank-zealot — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
> **ACTIVE since 2026-10-05 (evening); reshaped three times on
> 2026-10-06, renamed from frank-small the same night.** A 27B replaces
> **RETIRED 2026-10-06 → 2026-10-07** (AGENTS.md Runbook Index; cut over
> 2026-10-05 ~23:26, reshaped three times on 2026-10-06, renamed from
> frank-small the same night). Superseded by
> [[frank-zealot-llama-qwen35-9b-q6k-2x200704]]; the production shape on :7771
> today is llama.cpp PrismML build **10777** on
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
> measurement (AGENTS.md Runbook Index active row); the 27B PQ2_0 / 2 × 200,704
> shape of [[frank-zealot-llama-prism-bonsai2-pq20]] is that era's rollback
> target. Transient 2026-10-18 bench window on :7771 — the unit is `failed`
> since 03:14 and a hand-booted process answers the port; restore with
> `./scripts/deploy.sh dankclaw --service frank-zealot`).
> **Rollback target**: the payload is still on disk —
> `stat -c %s` on 2026-10-18 gives 10442827392 for
> `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf`
> and 931145888 for its `mmproj-BF16.gguf`. A 27B replaces
> the 9B on the same 24 GiB card: KV at FP8 class, vision on. The
> morning order (drop MTP, ubatch 512, TWO 128k contexts) bought the
> second slot — 1,397 MiB free. The evening order swapped to the
@@ -41,8 +54,8 @@ timestamp: 2026-10-06
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-zealot`, `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary` on the wire until the 2026-10-09 rename to `frank-zealot` (commit `7c77326`), `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (quantized V needs `--flash-attn on`; it does NOT need `GGML_CUDA_FA_ALL_QUANTS` — see the KV section) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
@@ -78,9 +91,11 @@ The order supersedes the same-day f16/f16 rule, which was written for the
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
pinned: quantized V requires it, and the build's
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than
a silent f16 fallback.
pinned: quantized V requires it. The era note claiming the build carried
`GGML_CUDA_FA_ALL_QUANTS=ON` does not describe the binary on disk: its
`CMakeCache.txt` reads `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` (binary built
2026-10-01 22:35) and the option is vestigial in that tree — `ggml/src/ggml-cuda/fattn.cu:438-449`
dispatches the Q4/Q5 KV cases unconditionally.
**Two slots + drafter, by arithmetic and by order (2026-10-06 morning /
evening, the 2 × 131,072 era).** The morning order bought the second slot
@@ -320,7 +335,7 @@ recipes:
second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
**frank-zealot** (87721ab — scripts, unit, seam, runbooks,
**frank-secondary** (87721ab — "rename frank-small → frank-secondary", scripts, unit, seam, runbooks,
identifiers; historical references to the OLD FreeToken/CUDA
frank-small, and to the Dankserver service that became frank-medium,
stay frank-small) and the per-slot window cut 131,072 → 114,688 to
serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md
+8 -1
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
description: "The RETIRED frank-zealot runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
description: "The RETIRED frank-zealot runbook (2026-09-16 → 09-17) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-zealot
@@ -13,3 +13,10 @@ tags:
- vision
timestamp: 2026-09-16
---
**RETIRED 2026-09-16 → 2026-09-17** (AGENTS.md Runbook Index); superseded by
[[frank-zealot-gemma4-12b]], and :7771 today runs
[[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
`Ternary-Bonsai-2-27B-PQ2_0.gguf`). The payload is **not** a rollback target any
more: the 2026-10-18 census of `/home/dank/models/` has no GSQ-RCO directory, so
restoring this shape means re-downloading from ISTA-DASLab.
serverAppConfigs-docs/LLM/frank-zealot-sglang-mimo-v26.md
+22 -16
View File
@@ -17,18 +17,22 @@ timestamp: 2026-10-01
# frank-zealot — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
> **RETIRED 2026-10-01, hours after cutover** (AGENTS.md Runbook Index). Replaced by
> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. :7771
> today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork
> build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off).
> The venv (`/home/dank/GitPackages/sglang-cu128/`) is still on disk; the INT8
> payload was staged to
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
> is purged.
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and
> **that stage batch is gone** (`ls /home/dank/.AGENTTRASH/`, 2026-10-18 —
> only the 20261005/06/07/10/17 batches remain), so this runtime needs
> `old/download-frank-zealot-sglang-int8.sh` to re-stage.
## Identity
- **Endpoint:** `frank-zealot` at `192.168.0.143:7771` (Dankclaw).
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) — the id
became `frank-secondary` on 2026-10-06 and `frank-zealot` on 2026-10-09.
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
@@ -53,9 +57,10 @@ timestamp: 2026-10-01
one request. A single request may grow to the whole pool — a
170,019-token prompt was verified at the 4-slot shape — the grid is
the two-way concurrency contract. **That compaction arithmetic is
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
180,224, which 183,501 tokens already exceeds — see *What three slots
cost* in [[frank-zealot-llama-mimo-q5km]].
specific to a 262,144-token slot**: the shape that followed on 2026-10-02
ran three slots of 180,224, which 183,501 tokens already exceeds — see *What
three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 runs two 200,704
slots today.
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
dim x 1 B), pool 9.31 GiB.
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
@@ -125,7 +130,8 @@ change lands.
## Payload
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
files lived under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`
(absent from the 2026-10-18 census — see the banner),
all LFS sha256-verified by `old/download-frank-zealot-sglang-int8.sh`
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
cut over to llama.cpp + GGUF on 2026-10-01 night, and
@@ -134,7 +140,7 @@ serve script refuses to start with a missing shard.
## Client contract
- **Alias:** `frank-zealot` (the `id` at `/v1/models`).
- **Alias:** `frank-small` (the `id` at `/v1/models`).
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
as llama.cpp; models.yml sets `reasoningContentField:
@@ -166,10 +172,10 @@ serve script refuses to start with a missing shard.
## Rollback
[[frank-zealot-llama-mimo-q5km]] is the live shape two steps forward; the
practical rollback today is [[frank-zealot-llama-mimo-q6k]] (payload at
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
Gemma4-12B QAT payload, was staged to
[[frank-zealot-llama-mimo-q5km]] was the live shape two steps forward; the
practical rollback at the time was [[frank-zealot-llama-mimo-q6k]] (payload at
`/home/dank/models/MiMo-Q6K/`, still on disk 2026-10-18).
This era's own rollback, the llama.cpp Gemma4-12B QAT payload, was staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
2026-10-02; its unit/env/serve trio remains in `old/`.
serverAppConfigs-docs/LLM/frank-zealot-vllm-gemma4.md
+38 -24
View File
@@ -17,10 +17,13 @@ timestamp: 2026-09-19
# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
> back to [[frank-zealot-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
> the same day it landed — this page is the rollback target. Same port
> (7771), same alias (`frank-zealot`), same client contract surface — the
> **RETIRED 2026-09-19 (same-day revert) — rollback target only in git.**
> This vLLM deployment was cut
> back to [[frank-zealot-gemma4-12b]] (llama.cpp — itself retired
> 2026-10-01; :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]])
> the same day it landed. Same port
> (7771), same alias (`frank-small` — the name `frank-zealot` dates from
> 2026-10-06/10-09), same client contract surface — the
> runtime, the model payload, the drafter and the service shape (root
> systemd unit → rootless Quadlet) all changed.
@@ -28,12 +31,12 @@ timestamp: 2026-09-19
| what | value |
|---|---|
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-zealot` (`8dad46d7…`) |
| unit | **rootless Quadlet** `dankclaw/frank-zealot/quadlet/frank-zealot.container` (root systemd unit retired 2026-09-19) |
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46` — still in rootless storage on the 2026-10-18 `podman images` census; the tag carries the era name) |
| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (era path; deleted by the revert commit `374041c` — root systemd unit retired 2026-09-19) |
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` |
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` |
| port | 7771 → 8000 (alias `frank-zealot`; `frank-dragoon` keeps 7770) |
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` (absent from the 2026-10-18 census of `/home/dank/models/` — re-fetch to use this rollback) |
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` (absent from the same census) |
| port | 7771 → 8000 (alias `frank-small` in this era; `frank-dragoon` keeps 7770) |
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
@@ -113,7 +116,8 @@ trade for the native w4a16 format + audio + CUDA graphs.)
## The image
`dankclaw/frank-zealot/Dockerfile` (repo) — 16 comment lines + `FROM
`dankclaw/frank-small/Dockerfile` (era path, deleted by the revert commit
`374041c`; recover with `git show 374041c^:dankclaw/frank-small/Dockerfile`) — 16 comment lines + `FROM
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
@@ -144,9 +148,9 @@ not a hardware mystery.
## Deployed config
Repo truth: `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
Repo truth **for this era** (the files carry llama.cpp knobs today): `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
(mounted into the container at `/scripts/`, which is the entrypoint) +
`dankclaw/frank-zealot/quadlet/frank-zealot.container`. Key values:
`dankclaw/frank-small/quadlet/frank-small.container` (era path, deleted at `374041c`). Key values:
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
@@ -169,7 +173,9 @@ assistant, or the vllm binary are missing.
- Audio → 200, a 440 Hz WAV tone described correctly (native audio
encoder; verified in the derived image).
- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
`input: [text, image, audio]`.
`input: [text, image, audio]` — era values; the endpoint reads 200,704 /
16,384 / `[text, image]` today (`~/.omp/agent/models.yml:206,212,214`), and
audio left the fleet with this payload on 2026-10-01.
## Boot ledger
@@ -186,27 +192,35 @@ container while it loads.
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
must be absent. 8.6 is correct.
- **Audio 400/500** — the container is running the *base* image, not the
derived `:frank-zealot` (missing `av`). Check the deployed image digest
is `8dad46d7…`.
derived `:frank-small` (missing `av`). In this era, check the deployed image
id is `8dad46d7…` (`podman images`, 2026-10-18: it still exists, unused —
nothing on :7771 runs in podman today).
- **CUDA not visible in container** — a `Device=` line or a lib mount
dropped out of the unit after an edit; the unit is the source of truth,
the host-side generated unit is not.
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet; `podman
logs` covers only the current `--replace --rm` instance).
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet in this era;
`podman logs` covers only the current `--replace --rm` instance). The unit is
a ROOT unit today: `journalctl -u frank-zealot`.
## Rollback
[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
mmproj and MTP sidecar are still on the host under
`/home/dank/models/Gemma4-12B-QAT/`, and the retired root unit is at
`scripts/.AGENTTRASH/frank-zealot.service.llama.cpp`. Restoring it means
reinstalling that unit and stopping the Quadlet (one owner per port).
mmproj and MTP sidecar are **no longer on the host** under
`/home/dank/models/Gemma4-12B-QAT/` (2026-10-18 census: the directory is
gone — it was staged to `.AGENTTRASH/20261002-retired-llm-payloads/` and that
stage dir is gone too), so this rollback re-fetches
`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` and restores the era
trio from git (`git show 374041c:scripts/frank-small.service`,
`…:scripts/frank-small.env`, `…:scripts/serve-frank-small.sh`); the era quadlet
lived at `dankclaw/frank-small/quadlet/frank-small.container`. One owner per
port: the llama.cpp unit is root, so stop the container unit before starting it.
## References
- `scripts/serve-frank-zealot.sh` — the actual exec line
- `dankclaw/frank-zealot/Dockerfile` — the audio-derived image
- `dankclaw/frank-zealot/quadlet/frank-zealot.container` — the unit
- `scripts/serve-frank-zealot.sh` — the actual exec line (today: llama.cpp; the
vLLM exec line is in `git show 374041c^:scripts/serve-frank-small.sh`)
- `dankclaw/frank-small/Dockerfile` — the audio-derived image (era path, deleted at `374041c`)
- `dankclaw/frank-small/quadlet/frank-small.container` — the unit (era path, deleted at `374041c`)
- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
+146 -66
View File
@@ -10,7 +10,7 @@ tags:
- stt
- vllm-omni
- kokoro
timestamp: 2026-09-22T00:00:00Z
timestamp: 2026-10-18T00:00:00Z
---
# Local TTS and Voice Cloning
@@ -19,7 +19,10 @@ Research brief (2026-09-22) on creating a new local TTS voice from sample
files, the open-weight voice-cloning landscape, the full local AI audio
stack, and a recommended deployment for the fleet. Every claim is cited in
[Citations](#citations). The fleet has no audio input today (that capability
left with the retired NPU endpoint), so this is greenfield.
left with the retired NPU endpoint, returned with the Gemma4 audio projector on
2026-09-17, and left again with that retirement on 2026-10-01 — audio is still
greenfield: every fleet endpoint declares `input: [text, image]`
(`~/.omp/agent/models.yml:63,149,214`).
## Summary
@@ -31,20 +34,26 @@ left with the retired NPU endpoint), so this is greenfield.
built-in voice list (a custom pack needs a patch).
- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
(Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT)**, **VoxCPM2
(Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
- **OMP integration — definitive:** OMP's `tts` tool supports exactly three
backends (`providers.tts` = `auto|local|xai|deepinfra`, verified in the
installed binary): local Kokoro ONNX, xAI Grok Voice cloud, DeepInfra cloud.
**There is no generic OpenAI-compatible TTS endpoint in OMP today.** Two real
paths for a cloned voice: (a) xAI console custom voice (free, cloud, US-only)
— works today; (b) patch OMP (open source) with a ~50-line OpenAI-TTS backend
pointing at a local `vLLM-Omni` server.
- **Recommended stack:** `vllm/vllm-omni` serving **Qwen3-TTS-1.7B-Base** on the
3090 (OpenAI-compatible `POST /v1/audio/speech` + `POST /v1/audio/voices`
voice upload), **Parakeet TDT 1.1B** for English STT, **Silero VAD** embedded,
**Chatterbox** as the ultra-light English-only alternative. All fit in <8 GB
VRAM and co-locate with the existing LLM services.
(Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT weights)**,
**VoxCPM2 (Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
- **OMP integration (re-verified 2026-10-18):** the speech tool is gated by
`speechgen.enabled` and resolves models through the **`speech` role and its
fallback chain** (`modelRoles.speech`; this workstation runs
`local/kokoro`). Three APIs: local Kokoro ONNX (always WAV/PCM16), `xai-tts`,
and **`openai-speech` — a generic OpenAI-compatible `/v1/audio/speech`
client** (MP3 or WAV, `voice_id` forwarded only when explicitly set, 15 000
characters max; DeepInfra is now just a model on that API). The 2026-09-22
finding that OMP has *no* configurable OpenAI TTS endpoint is **retired**: a
local server needs a speech-role entry, not a patch.
- **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue
#2 rules out every GPU option (the 3090 has carried frank-zealot since
2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF
engine on **8880** behind a voice-registry shim on **8890**
(`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI
`/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on
2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or
8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality
number has been measured for this shape**, so this page states none.
## Can Kokoro learn a new voice?
@@ -70,18 +79,25 @@ running a modern zero-shot cloner on the same GPU.
## 2026 voice-cloning landscape (permissive-licensed, single-GPU)
Ranked for a 3090 homelab, English-primary, personal voice:
Ranked (2026-09-22) for a 3090 homelab, English-primary, personal voice; the
2026-10-18 fleet decision below moved this work off the GPU entirely:
1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference;
110–500 M; ~1–2 GB VRAM; CPU-capable (Nano). *Top pick for a personal
English voice.* No official server (library + Gradio; wrap in FastAPI).
1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference
(the README's own example is `your_10s_ref_clip.wav`); 110 M (Nano) to 0.5 B
(Turbo/Multilingual); ~1–2 GB VRAM *(unverified 2026-10-18)*; Nano is
documented for CPU — "3x faster than realtime on 8 CPU cores". *Top pick for
a personal English voice.* No server in the README (library use only; wrap
it yourself); every generated file carries Resemble's PerTh watermark.
2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
vLLM-Omni serving with `POST /v1/audio/voices` upload (closest to "named
voice" semantics), official fine-tuning code. *Top pick for a server-grade,
vLLM-Omni serving with a `POST /v1/audio/voices` upload *(unverified
2026-10-18 — the reachable docs do not list it)*, the closest thing to
"named voice" semantics, official fine-tuning code. *Top pick for a
OpenAI-compatible deployment.*
3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
bi-streaming, FastAPI/WebSocket server.
4. **GLM-TTS (MIT)** — best published CER (0.89), RL emotion control, zh/en.
bi-streaming, FastAPI and gRPC servers.
4. **GLM-TTS (MIT model weights, Apache-2.0 code)** — CER 0.89 after
multi-reward GRPO (down from 1.03), RL emotion control, zh/en; the prompt
audio shipped in the repo is marked research-only.
5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
@@ -97,30 +113,36 @@ released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
## OMP integration (definitive)
OMP's `tts` tool (hidden by default; enable with `/tts`) supports exactly three
backends, selected by `providers.tts`:
The tool ships hidden and is enabled by `speechgen.enabled`
(`~/.omp/agent/config.yml:222-223`); the harness docs list it as "Generate
speech". Models come from the **`speech` role and its fallback chain**, not
from a hardcoded backend list (`modelRoles.speech`,
`~/.omp/agent/config.yml:16` = `local/kokoro` today):
| Backend | Endpoint (hardcoded in binary) | Voices |
| API | Endpoint | Notes (from the installed bundle, 2026-10-18) |
|---|---|---|
| `local` | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` | built-in list (`tts.localVoice`) |
| `xai` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + any custom voice ID |
| `deepinfra` | `POST https://api.deepinfra.com/v1/openai/audio/speech` | `voice_id` when set |
| local-inference | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` (Apache-2.0) | no network, **always WAV/PCM16**, voice from the built-in list (`tts.localVoice`) |
| `xai-tts` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + custom voice IDs; MP3 or WAV |
| `openai-speech` | `POST <provider baseUrl>/audio/speech` | the generic OpenAI-compatible client — MP3 or WAV, `voice_id` forwarded **only when explicitly set**, 15 000-character cap on `input`; DeepInfra is one model entry on it (`https://api.deepinfra.com/v1/openai`) |
There is **no generic OpenAI-compatible TTS endpoint** — `deepinfra` is a
hardcoded URL, not a configurable base URL, and OMP's `models.yml` custom
providers (LLM chat only) do not wire into the `tts` tool.
The 2026-09-22 finding — "**no generic OpenAI-compatible TTS endpoint** …
`deepinfra` is a hardcoded URL, not a configurable base URL" — is **retired**:
`openai-speech` is that configurable client, selected by putting a model on the
`speech` role. Still true from the original audit: the local Kokoro backend
accepts only its built-in voice list, so a custom pack needs a patch.
Paths for a Frank-cloned voice:
Paths for a Frank-cloned voice (2026-10-18):
1. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → `tts` tool.
Free (≤30 voices), cloud, US-only.
2. **Local, full stack — small OMP patch.** OMP is open source; the DeepInfra
backend is a plain HTTP POST to an OpenAI-compatible speech endpoint. A
~50-line `openai` backend (configurable `base_url` + `voice`) lets OMP call
any local server (vLLM-Omni, Kokoro-FastAPI, Chatterbox wrapper).
3. **No-patch workaround.** OMP's `bash` tool can `curl` the local TTS server
ad hoc.
1. **Local server + speech role, no patch.** Register the local engine (or the
8890 voice shim) as a provider model on the `openai-speech` API and point
`modelRoles.speech` at it; the shim turns a voice *name* into a reference
clip and forwards it as base64 `reference_audio`
(`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18.
2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech
tool. Free (≤30 voices), cloud, US-only.
3. **No-patch workaround.** The `bash` tool can `curl` the local TTS server ad
hoc, and the harness `tts` device writes a speech file to a path you give it.
4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.
## Full local AI audio stack
@@ -136,34 +158,77 @@ Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
faster-whisper/whisperX).
**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
official Docker `vllm/vllm-omni` CUDA and `vllm/vllm-omni-rocm` ROCm, e.g.
`:v0.28.0`) is the one server exposing the standard OpenAI speech API
(`POST /v1/audio/speech` + `POST /v1/audio/voices` upload for Qwen3-TTS) across
the 2026 lineup. CUDA needs compute ≥7.0 (3090 = 8.6 ✓); ROCm validated on
gfx942 (the fleet already runs vLLM on the R9700 gfx1201 ✓). Others:
Kokoro-FastAPI (Docker, :8880, OpenAI-shaped), CosyVoice FastAPI/WS, F5-TTS
Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 `tools/api_server.py` (own
format, not OpenAI-shaped).
Apache-2.0) publishes the official images `vllm/vllm-omni` (CUDA) and
`vllm/vllm-omni-rocm` (ROCm), both tagged `v0.28.0` (Docker Hub tag lists,
2026-10-18), and serves Qwen3-TTS (`docs/models/supported_models.md`). It
serves the OpenAI speech path `POST /v1/audio/speech`; the
`POST /v1/audio/voices` upload route is *(unverified 2026-10-18)* — the docs
reachable today do not document it. Its gfx942/gfx950 validation note is a
footnote scoped to MiniMax H3 diffusion, **not** to TTS, and the fleet runs no
vLLM anywhere: the R9700s carry radiance as frank-archon
(`scripts/serve-frank-archon.sh:2`). It is also no longer "the one server" with
the OpenAI shape — llama.cpp's own `llama-tts-server` has it too (next
section). Others: Kokoro-FastAPI (Docker, :8880, Apache-2.0), CosyVoice
**FastAPI and gRPC** (`runtime/python/fastapi`, `runtime/python/grpc`; the
README documents no WebSocket server), F5-TTS Gradio/Triton, GPT-SoVITS
WebUI/api_v2, Fish S2 `tools/api_server.py` (own format, not OpenAI-shaped).
**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
reference-clip standard); Demucs (`htdemucs`) for vocal separation before
cloning; the Perth watermark is built into Chatterbox.
## llama.cpp-native TTS serving (verified 2026-10-18)
The CPU shape does not need a Python server at all:
- **`llama-tts-server`** ([`tc-mb/llama.cpp-omni`](https://github.com/tc-mb/llama.cpp-omni),
MIT, "Omni inference in C/C++") is a standalone OpenAI-compatible TTS server
that runs without an LLM model (build target `llama-tts-server`).
`POST /v1/audio/speech` takes `model` (`"voxcpm"`, also accepts `"voxcpm2"`),
`input`, `voice` — documented **"reserved"**, so do not build named-voice
semantics on it — `response_format` (`wav`/`pcm`), `reference_audio`
(base64 WAV for cloning), `seed` 42, `cfg_value` 2.0, `inference_timesteps`
10, `max_steps` 200, `temperature` 1.0. `POST /v1/audio/speech/stream`
returns chunked WAV; `POST /v1/voxcpm2/init` hot-loads a runtime from
`base_lm` + `acoustic` GGUFs with `n_gpu_layers` (`-1` = all);
`GET /v1/audio/speech/models` lists what is loaded
(`tools/omni/voxcpm2/README.md`:232, 321-354).
- **Mainline `llama-tts`** (ggml-org/llama.cpp) has `--tts-speaker-file` for a
speaker reference with **Qwen3-TTS** and with **Kyutai Pocket-TTS** — the
latter requires it ("the model produces almost no audio without it"), and
Pocket-TTS converts one `languages/<name>` directory, never the repo root
(`tools/tts/README.md`:20-58; `kyutai/pocket-tts` is CC-BY-4.0).
- **VoxCPM2 GGUF provenance:** `openbmb/VoxCPM2` publishes **PyTorch weights
only** (`model.safetensors`, no `.gguf` sibling — HF API 2026-10-18). The
GGUFs the fork loads come from
[`DennisHuang648/VoxCPM2-GGUF`](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF)
(`VoxCPM2-BaseLM-F16.gguf`, `VoxCPM2-BaseLM-Q8_0.gguf`,
`VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own
`tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`.
Nothing has been measured on this fleet for any of these engines — no RTF, no
latency, no MOS. That absence is the finding, not an omission.
## Recommended deployment (Dankclaw)
TTS models are 0.1–2 B (<8 GB VRAM); STT (Parakeet 1.1 B) is CPU-fast. Keep it
all on Dankclaw (3090 = primary GPU, CUDA = best vLLM-Omni support); the
Dankserver R9700s stay dedicated to frank-medium.
As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's,
so no card may be asked to co-host TTS. The shape is one engine plus a thin
voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl
--user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or
8890.
| Service | Image / install | Port | GPU | Notes |
| Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---|
| TTS (primary, cloning) | `vllm/vllm-omni:v0.28.0` · `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8881 --gpu-memory-utilization 0.35` | 8881 | 3090 (~6 GB) | OpenAI `/v1/audio/speech`; upload Frank's voice once via `/v1/audio/voices`; 3 s cloning; 10 langs; Apache-2.0 |
| TTS (light, English) | Chatterbox (MIT) in a small FastAPI container | 8880 | 3090 (~1 GB) / CPU | 10 s ref clip; wrap `chatterbox.generate` in ~30 lines |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | CPU | CC-BY-4.0; WER 1.39% |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | CPU / iGPU | MIT |
| VAD | Silero (embedded) | — | CPU | MIT |
| TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot |
| Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |
Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM=
http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT
keep clear of the fleet's LLM ports 7770/7771.
## Voice-creation workflow (the "from sample files" answer)
@@ -190,17 +255,22 @@ Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
- **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the
TTS engine shares the card with frank-zealot; move to the R9700s only if TTS
throughput demand grows.
- **Keep TTS off the GPUs.** The 2026-09-22 advice to "co-locate on the 3090 at
`--gpu-memory-utilization ~0.35`" is retired: that card belongs to
frank-zealot (2026-10-17) and the R9700s to frank-archon, and Gitea issue #2
measures both as full. Run TTS on CPU; revisit only if the operator frees a
card on purpose.
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
internal address, never the public hostname.
## Citations
- OMP TTS tool: https://omp.sh/docs/tools/tts · installed binary
`@oh-my-pi/pi-coding-agent/dist/cli.js` (endpoints, voices, model IDs)
- OMP speech tool: https://omp.sh/docs/tools ("Generate speech", gated by
`speechgen.enabled`; the old `https://omp.sh/docs/tools/tts` path returns 404
as of 2026-10-18) · installed binary
`@oh-my-pi/pi-coding-agent/dist/cli.js` (the `openai-speech` API, the
15 000-character cap, the `voice_id` forwarding rule, local Kokoro WAV-only)
- Kokoro: https://github.com/hexgrad/Kokoro ·
https://huggingface.co/spaces/hexgrad/Kokoro-TTS
- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
@@ -212,6 +282,16 @@ Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
https://huggingface.co/zai-org/GLM-TTS
- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
https://huggingface.co/openbmb/VoxCPM2
- llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni ·
https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md
- mainline `llama-tts`:
https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md ·
Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)
- VoxCPM2 GGUFs (not published by OpenBMB):
https://huggingface.co/DennisHuang648/VoxCPM2-GGUF
- GLM-TTS license split: model card `mit` (HF API 2026-10-18) vs code
`LICENSE` (Apache-2.0):
https://raw.githubusercontent.com/zai-org/GLM-TTS/main/LICENSE
- F5-TTS: https://github.com/SWivid/F5-TTS
- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base