Compare commits

..
Author SHA1 Message Date
Frank 0367244d2a fix(deploy): client allowlist was missing cadvisor, gpu-exporter, prowlarr
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.

Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
2026-10-21 00:15:21 -07:00
dank 656d9567d5 Merge pull request 'chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)' (#62) from chore/plex-1435 into main 2026-10-21 00:07:28 -07:00
Frank 0b4bc60a38 chore(plex): bump pin to 1.43.5.11029-6bf8af3f0-ls329 (latest)
ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029
is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex
releases behind. Digest-verified at pin time: the tag's manifest digest
equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the
same law the page records. Quadlet restart runs PMS's usual one-minor
schema migration on first boot; the deployer's /identity HTTPS gate
tolerates the first-boot migration up to 600 s.

Rollback: revert this commit (one line).
2026-10-21 00:06:53 -07:00
dank a2011f4eb6 Merge pull request 'feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet' (#61) from feat/gpu-stats into main 2026-10-20 23:50:11 -07:00
Frank b02647b992 feat(observability): per-GPU metrics for the mixed AMD/NVIDIA fleet
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.

scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.

It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
2026-10-20 23:42:26 -07:00
dank b35269bfa5 Merge pull request 'docs(archon): four-slot ledger, resolver law from source, probe self-calibration' (#60) from wt/20261020-213526 into main 2026-10-20 23:30:27 -07:00
Frank 1aeb7af6df docs(probe): price the corpus chars/token ratio at run time
The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the
repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate
400 from the engine's own window check — which reads as a capacity-contract
failure on a healthy endpoint. One ~4k-token request now prices the ratio, and
the leg sizes to 97 % of the deployed window.
2026-10-20 23:26:53 -07:00
Frank cd0be97b39 docs(vault): the four-slot ledger, and the resolver's funding law from the source
The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB
(78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions,
`--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host`
8.99 GiB a rank.

The KV Capacity law is rewritten from the pinned source instead of inference:
`core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`,
`core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never
funding), and `core/place/planner.cpp:100-104` makes the expert floor fall
one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool
bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %,
never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling
case, not a mystery. Slot-cost law carries the four-slot price (state +
checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot
era is one row in Eras with its rollback commit. Date stamps in the trio
comments corrected to 2026-10-20.
2026-10-20 23:26:53 -07:00
dank 7993a336bf Merge pull request 'feat(frank-archon): four decode slots over a >=700k-token KV pool' (#59) from wt/20261020-213526 into main 2026-10-20 23:14:35 -07:00
Frank 1bb6399524 feat(frank-archon): four decode slots over a >=700k-token KV pool
Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it
exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is
800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of
evidence on the endpoint page), which clears 700k, and the VRAM has to come
from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with
it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly
plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60
GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to
19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the
leftover returned to the experts.

Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per
rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when
the router picks a host-resident expert. The boot ledger's vram_kv line is the
referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and
this redeploys.

TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 +
prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB =
161,848 MiB), and the bats ledger guard now counts the checkpoint store because
it grows with the slot count. The two env-value assertions in that seam are
bumped with the values they guard.
2026-10-20 23:10:05 -07:00
dank 31b71a16d6 Merge pull request 'feat(observability): per-container metrics via cadvisor + Containers dashboard' (#58) from feat/containers-observability into main 2026-10-20 23:08:23 -07:00
dank a4edaa7d9d Merge pull request 'feat(frank-archon): arm --prefill-interleave at 1' (#57) from wt/20261020-213526 into main 2026-10-20 22:55:58 -07:00
Frank aa75b45543 feat(frank-archon): arm --prefill-interleave at 1
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.

BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.

Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
2026-10-20 22:52:31 -07:00
Frank 31908c623f feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.

cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.

cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.

Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
2026-10-20 22:51:42 -07:00
Frank b1fc1113c6 docs(vault): replace an invented --checkpoint-policy name with the three that exist
The replay law named `--checkpoint-policy floor`; config.cpp admits exactly
geometric-backoff (default), tip-only, keep-all and refuses anything else at
boot, so the recorded arm could never be deployed. Replaced with the real
semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric
ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live
ledger line -- so a turn diverging d behind the tip can resume at tip-3d:
diverge at 120k, replay 21.7k); retention is not the write path, so keep-all
buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4
GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room);
and shortening --checkpoint-interval cannot fix the deep case -- the coarsest
ladder scale is interval x 2^k under the depth, so the deepest gap stays about
half the prompt at every interval. §9 keeps only the three-name locator.
2026-10-20 22:34:13 -07:00
Frank 9f7457e39e docs(vault): answer "what are we doing wrong" on decode-vs-prefill
frank-archon Laws: add the measured interference census (decode-only step
15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median
decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token
per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test
that kills the residency theory, the busy_cap arithmetic, and the correction
that --prefill-interleave is the decoder dial and auto is not.

Facts that change how an operator reads this box: /diagnostics exists and
reports link-below-max at 100% busy; latency metrics live under vllm:, cadence
under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all
bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5
max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the
4600 corruption fix is recorded -- verify before blaming garbage tokens.

§9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters,
dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA
gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the
chunk), instruments that cannot be reached from Prometheus, and the knob list
priced from the pinned 1.8.1 source instead of the stale guide.
2026-10-20 22:08:00 -07:00
Frank 687dbfbabe docs(scripts): correct frank-archon's step-sharing and slot-count comments
The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1
source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding
sequence and 1/8 of the budget per waiting request, and refuses to shrink the
chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment
carried a 512 that is not this build's auto value (0 until the held-step
counter asks; 64 in help mode). The serve header still described three decode
slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.
2026-10-20 22:07:55 -07:00
dank 961713e92d Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56) from feat/e2e-audit-coverage into main 2026-10-20 21:21:44 -07:00
dank 01db29e754 Merge pull request 'feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring)' (#55) from feat/grafana-starter-dashboards into main 2026-10-20 21:20:53 -07:00
Frank ab06e4c221 test(e2e): cover the two guard-surface gaps found in the full-suite audit
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:

- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
  vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
  upstream would stay invisible - the exact 17-hermes class. Added to the 14-
  guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
  new 21- that proves authenticated reachability to open-notebook's own login
  through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
  wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
  there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
  answers {status:success}.

Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
2026-10-20 21:16:23 -07:00
Frank 1006e96312 feat(grafana): starter dashboards for host/system, Loki logs, and observability self-monitoring
llm-fleet.json already owns LLM capacity+spec-decode. These three cover the
domains that had NO dashboard:
- host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both
  Dankclaw and Dankserver (templated by job node-dankclaw|node-dankserver)
- logs-overview: Loki volume + per-container error rate + a sample log panel;
  container is parsed from the JSON line at query time (journal streams have
  no container_name stream label)
- observability: the stack's own health - Prometheus target up/down + scrape
  interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem.

Panel queries verified live against Prometheus/Loki; provisioned dashboards
confirmed via /api/search. Alertmanager is running but NOT scraped by
Prometheus - noted in the dashboard, no empty panels built on it.
2026-10-20 21:10:06 -07:00
dank 7d85fed1c9 Merge pull request 'Fix Grafana Authentik SSO (500 + missing email claim) + RP landing test' (#54) from fix/grafana-authentik-sso into main 2026-10-20 21:04:02 -07:00
Frank d8196ce499 docs: Grafana reads claims from userinfo, not USE_ID_TOKEN
The login lands once authentik emits the email claim (the IdP fix); Grafana needs
no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in
the literal-secret seam. Correct the two SSO notes accordingly.
2026-10-20 20:41:57 -07:00
Frank 6f442a14a3 fix(observability): repair Grafana Authentik OAuth2 relying-party login
The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.

Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
2026-10-20 20:41:36 -07:00
Frank afe72781bc docs: correct Grafana Authentik SSO facts
The vault claimed the fork property-mapping evaluator was broken and userinfo
returned only sub, so email could never reach Grafana (outcome unverified). That
was a misdiagnosis: the real causes were the NULL authorization_flow (the
NoneType.slug crash) and the email scope mapping returning a bare string rather
than a dict. Record the four verified causes and fixes, point to the repair
script and the new landing spec, and drop the unverified caveat.
2026-10-20 20:40:15 -07:00
Frank 200039d589 test(e2e): cover Grafana relying-party SSO landing (spec 20)
13-sso proves an authenticated session reaches Grafana login page but never
follows the OAuth button, so the relying-party login could be dead (500, or an
email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives
/login/generic_oauth through the real Anubis+authentik round-trip and asserts a
live session (/api/user 200 with a non-empty email). Wire 20- into the standard
project (qa-e2e session).
2026-10-20 20:40:15 -07:00
Frank 91d9864bb0 fix(observability): repair Grafana Authentik OAuth2 relying-party login
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.

Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.

scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
2026-10-20 20:40:15 -07:00
dank 980491668c Merge pull request 'Alloy ships logs: root host-network service so it can read the journal' (#53) from feat/alloy-root-journal into main 2026-10-20 20:05:00 -07:00
Frank e494617599 docs(alloy): document the root journal-read cutover
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
2026-10-20 19:50:15 -07:00
Frank 9809ed9c74 test(observability): pin the root alloy shape
The seams encoded alloy-as-rootless (default.target, dankclaw_default,
loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update
them to the root/system/host shape and add a test pinning the
load-bearing properties: Network=host, the /etc/machine-id mount, the
journal mounts, loopback UI, and the system-install + root-mirror
deploy path.
2026-10-20 19:50:15 -07:00
Frank 634fef242b feat(alloy): run it as a root host-network service so it reads the journal
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.

As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.

deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
2026-10-20 19:50:15 -07:00
dank 2fe68e69af Merge pull request 'Observability: vllm rule group for frank-archon + Loki datasource; correct stale log-pipeline facts' (#52) from fix/observability-vllm-loki into main 2026-10-20 19:39:38 -07:00
Frank 70dd7fbe6d docs(observability): correct current state and log-pipeline root cause
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a
Loki datasource is provisioned beside Prometheus.
Observability.md: current-state census refreshed (three LLM targets up,
two rule groups, Loki datasource); logs still carry no data.
Alloy.md: replace the 'wrong glob' theory with the proven cause - a
rootless container cannot read the 0640 systemd-journal files (a rootless
busybox read returns DENIED), and rootless quadlet logs land in the user
journal the system-only source never reads. Closing it is a
permission/architecture decision, not a config glob.
2026-10-20 19:28:20 -07:00
Frank 2626332828 feat(grafana): provision the Loki datasource
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
2026-10-20 19:28:20 -07:00
Frank eb4d9910cb feat(observability): add vllm-fleet rule group for frank-archon
radiance exports vllm:* metrics (verified on the live endpoint), not
llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a
vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache
collapse, preemption, p95 time-to-first-token) keyed to vllm:*.

Split the observability seam so llm-fleet rules bind llamacpp:* and
vllm-fleet rules bind vllm:*, and neither may evaluate the other's
namespace. Drops the retired no-metrics-for-archon doctrine the seam
still asserted and the vacuous namespace grep.
2026-10-20 19:28:20 -07:00
dank 5cc2b3c25a Merge pull request 'docs(openwebui): TTS reflects live frank-tts wiring (wilwheaton, LAN IP)' (#51) from docs/openwebui-tts-voice into main 2026-10-20 18:20:04 -07:00
Frank 18a29c40d2 docs(openwebui): TTS config reflects the live frank-tts wiring
The Text-to-speech section recorded audio.tts base_url host.containers.internal
and voice narrator1. Re-keyed the live config DB 2026-10-20: voice wilwheaton
(narrator1 retired), api_base_url the canonical host LAN IP 192.168.0.143:7772.
7772 is published so host.containers.internal also reaches it, but the LAN IP
matches this page's own on-host law. Browser-kokoro example voice updated to the
current pack; the dated 2026-10-18 verification is left as history.
2026-10-20 18:18:01 -07:00
dank 73a7ec3fc1 Merge pull request 'chore(frank-tts): repoint narrator slots to live voice packs' (#50) from chore/narrator-voice-repoint into main 2026-10-20 18:12:52 -07:00
Frank 130e9d739f chore(frank-tts): repoint narrator slots to the live voice packs
Frank replaced the narrator1/2/3 packs with cloned character voices
(wilwheaton, vaati, dr.kern, garrickhagon, jeffhays). The three narrator
slots are now carried by real packs: narrator1=wilwheaton, narrator2=vaati,
narrator3=dr.kern.

- frank-tts.env DEFAULT_VOICE narrator1 -> wilwheaton (a voice-less request
  400'd on the dead name; wilwheaton is narrator1's successor).
- Docs: correct the /health census and the podcast-speaker references in
  Frank TTS.md and Open Notebook.md to the live packs. Dated 2026-10-18/19
  measurement logs are left as history.

Open Notebook's live dank-duo speaker profile was repointed over REST
(Frank->wilwheaton, Comrade->vaati) - app-side, not in this commit.
Bats fixtures keep narrator1 as a self-contained sample pack; untouched.
2026-10-20 18:11:40 -07:00
dank 854890a155 Merge pull request 'chore(open-notebook): admin-side positioning + single-tenant note' (#49) from chore/open-notebook-admin-tenant into main 2026-10-20 17:44:45 -07:00
Frank d9f3d1be1e chore(open-notebook): position as admin-side; document single-tenant
open-notebook has no user accounts upstream (PasswordAuthMiddleware = one
shared password; SurrealDB has no user/account/owner table — verified in the
running image). One shared password = one shared workspace; the guard chain
gates access but cannot partition data, and Authentik identity is never
consumed. Multi-user is approved-but-unbuilt upstream (#712, post-v2); OIDC
declined (#607). So move the homepage card from the user grid to the admin
tier and add the single-tenant boundary to the vault page. Fix the card's
stale 'frank-zealot' -> frank-archon. Update the seam's incidental category
pin to the new intentional value. Password rotated on the host (not repo).
2026-10-20 17:38:22 -07:00
dank afa3a4521c Merge pull request 'docs(authentik): record 7-day login-session TTL + setter script' (#48) from chore/authentik-session-ttl into main 2026-10-20 16:57:29 -07:00
Frank 9cc8ca1c88 docs(authentik): record the 7-day login-session TTL and add a setter script
The login-session lifetime (raised to days=7 on 2026-10-20) is owned by the
default-authentication-login stage's session_duration, not a global setting and
not SESSION_COOKIE_AGE. Authentik exposes no SESSION_SAVE_EVERY_REQUEST, so the
window is absolute. session_duration is a STRING idiom; a timedelta stores a form
timedelta_from_string rejects, crashing login. Add scripts/authentik-set-login-session-duration.sh
that validates the form before writing (DB state, not repo-managed, re-run after
a blueprint re-import). Trim the page back under the OKF 500-line cap. tests/run.sh green.
2026-10-20 16:55:44 -07:00
dank 9c05c40aa0 Merge pull request 'docs(agent): scaffold agent-skills config in the vault' (#47) from chore/agent-skills-vault-setup into main 2026-10-20 16:11:47 -07:00
Frank 63219787af docs(agent): scaffold the agent-skills config as three vault pages
Point the mattpocock-lineage skills (triage, to-tickets, to-spec,
wayfinder) at repo facts instead of a parallel docs/ tree: Gitea is the
issue tracker (credential-safe curl shape, token-only writes), the five
canonical triage roles keep their default label strings (created on the
tracker today), and GLOSSARY.md/docs/adr/ stay lazy with this vault
owning operational facts. AGENTS.md gains the ## Agent skills pointer
block; indices regenerated.
2026-10-20 16:09:30 -07:00
Frank e914ecbca6 docs(pi-hole): the record-load trap rides a container restart, not a boot
The 2026-10-19 write-up inferred a host restart from FTL's 16:25 start line. It was wrong: uptime -s reads 2026-10-18 01:24 (two days up) and the user journal shows 'Stopping pihole.service' at 16:25:23 with StartedAt 16:25:27 — a plain container restart inside the open-notebook deploy window. That moves where the verification belongs: after every pihole restart, not just after a reboot.
2026-10-20 10:56:39 -07:00
Frank 1eebf3b129 docs(netns): the by-name path to frank-dragoon is verified from the consumer
open-notebook resolves frank-dragoon-halogen to 10.89.1.39, GET :8731/v1/models answers, and the engine logs the app's POST /v1/embeddings 200. That makes the 2026-10-19 dial census a pre-move record: pasta's forwarding rule is unchanged, the endpoint was moved out of pasta, so the 'menu is frank-tts + frank-archon' line is now past tense. Frontmatter date was still 2026-09-02 against October content; bumped.
2026-10-19 22:48:44 -07:00
Frank af5a1f69a2 docs(pi-hole): FTL can boot with the record file mounted and serve none of it
The 2026-10-19 reboot left every franksdank.cloud name NXDOMAIN for LAN clients while external names kept answering from cache; the file was mounted and readable and dns.hosts was intact, and 'pihole restartdns reload' restored every name in seconds. Mechanism not established. Instrument and remedy now on the owner page, plus a failure-mode row for the FTL rate-limit case seen beside it (10.0.2.x = a pasta container's forwarder).
2026-10-19 22:46:57 -07:00
Frank 0930510f9b docs(vault): the embedding lane has a consumer, and two shapes that bite
open-notebook's embedding default now points at qwen3-embedding-0.6b on frank-dragoon's XDNA2 lane, proven through the app's own paths (model test 1024 dims, worker 'Successfully embedded', vector search ranking two unrelated notes 0.659/0.292 and 0.906/0.318). Records the two app-side traps: defaults store the model ID and a name is accepted then fails in the worker, and POST /api/embed 500s on an upstream RecordID typing bug after the vector is already written. frank-dragoon's lane section moves from 'off until the mount fix' to armed-and-serving with the boot-ledger contract and the measured idle RSS; the page's stale 'not on dankclaw_default' reachability claims are retired.
2026-10-19 22:46:57 -07:00
dank af327bcc5e Merge pull request 'feat(frank-dragoon): open the NPU node the way rootless can actually reach it' (#46) from feat/frank-dragoon-npu-lane into main 2026-10-19 22:35:49 -07:00
Frank fcf94562d1 feat(frank-dragoon): open the NPU node the way rootless can actually reach it
The armed boot reached the engine and died in its own preflight:

  halogen npu: /dev/accel/accel0 is here, but this process cannot open it
    (crw-rw---- nobody:nogroup).

The node is root:render 0660 and a rootless container maps exactly one uid/gid,
so `render` is unmapped inside it and `--group-add keep-groups` — the vendor's
advice, and the ComfyUI pattern for /dev/kfd — cannot grant a group the
namespace does not map. Measured 2026-10-19 from a `dank` shell already in
`render`: without the flag denied, with the flag denied, with the udev mode
0666 the open succeeds (OPEN_OK). Same wall 99-comfyui-gpu-access.rules already
solved for the GPU compute nodes, so the accel node gets its counterpart:

  dankclaw/udev/98-halogen-npu-access.rules — KERNEL=="accel[0-9]*",
  SUBSYSTEM=="accel", MODE="0666"

The deployer installs the rule, reloads rules, re-triggers the node and only
then checks the mode; the serve script refuses any node that is not
other-writable, naming the rule, so this cannot return as a crash loop (189
restarts while chat was down is the failure mode being closed).

Both checks read the permission bit directly: `[ -o FILE ]` is not
"other-writable" — it means "exists" and `-o` doubles as the OR operator inside
test, which is why the first cut of the gate refused everything.

Dragoon seam: 33 -> 35 tests. The fixture node is 0666 like the installed one,
a 0660 node is refused with the rule named and nothing rendered, and the rule
file's shape plus the deployer's install/trigger are asserted.
2026-10-19 22:35:06 -07:00
dank 861920797d Merge pull request 'fix(deploy): the dragoon gate reads the deployer's own payload-dir twin' (#45) from feat/frank-dragoon-npu-lane into main 2026-10-19 21:49:16 -07:00
Frank 3ef711f146 fix(deploy): the dragoon gate reads the deployer's own payload-dir twin
The containment gate added in #44 parsed scripts/frank-dragoon.env through a
$CHECKOUT variable that does not exist in deploy-remote.sh, so the very next
deploy died at `set -u` before it touched the unit:

  ./scripts/deploy-remote.sh: line 1445: CHECKOUT: unbound variable

deploy-remote.sh runs from the checkout root and every other path in that block
is a parity-twin default (LLM_DRAGOON_CKPT/_VISION/_TOK_DIR), so the payload dir
gets the same treatment: LLM_DRAGOON_MODEL_DIR, defaulted to the deployed path,
compared directly. The seam now asserts that twin against the env file next to
the NPU twins it already checks, and the containment comparison itself is
verified both ways (inside accepted; the old outside path and a shared-prefix
sibling both refused).
2026-10-19 21:48:47 -07:00
dank 823fd52d0e Merge pull request 'fix(frank-dragoon): stage the NPU lane inside the payload mount' (#44) from feat/frank-dragoon-npu-lane into main 2026-10-19 21:46:25 -07:00
Frank bda011d26f docs(vault): the NPU lane's one door into the container, and the pin that landed
frank-dragoon's lane section: the staging dir must live inside the payload dir
(the image ships no /models, so a nested :ro bind has no mountpoint to land on),
the entrypoint's rejection of HALOGEN_NPU_DIR, and the fabric-clock paragraph
corrected — the vendor's root host unit is installed and reports `held (manual)`
as of today, so "nothing is held" is retired along with the hand-write it
replaced.

dankclaw-iommu-translation-off: the deferred `pp_dpm_fclk` pin is no longer
deferred; the fact is owned by [[frank-dragoon]] and this page links it.
2026-10-19 21:43:36 -07:00
Frank b19f1c49d6 fix(frank-dragoon): stage the NPU lane inside the payload mount
The lane's first boot died at container init, not at the engine:

  runc create failed: ... create mountpoint for /models/npu mount:
  mkdirat .../merged/models/npu: read-only file system

The image ships no /models, and the serve script binds the payload dir
there read-only, so a nested `-v $NPU_DIR:/models/npu:ro` had to create
its own mountpoint underneath a read-only parent — impossible. The
engine's own entrypoint confirms the container path is fixed: it rejects
HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/").

Fix is the mount shape, not the engine: FRANK_DRAGOON_NPU_DIR now lives
inside FRANK_DRAGOON_MODEL_DIR, so the payload's single read-only mount
puts the files at /models/npu/<id>/ by itself and the nested bind is
gone. Verified on the running container: after the move, `podman exec
frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b` answers with
the staged tree while the lane is still armed off.

Both gates refuse the old shape by name (serve + deployer parity) so a
future payload swap that moves the model dir cannot reintroduce a crash
that lands after the deployer has already stopped the running unit.
2026-10-19 21:42:31 -07:00
dank b1ca4e02fa Merge pull request 'feat(frank-dragoon): run the XDNA2 NPU lane beside the flash model' (#43) from feat/frank-dragoon-npu-lane into main 2026-10-19 21:28:41 -07:00
Frankandomp 7f65cf0252 docs(vault): the pasta endpoint has no address an app container can use
The 2026-10-19 legs table blamed the frank-dragoon refusal on a root/rootless
netavark rule-set split. It is not that: the container is rootless (the root
unit's User=dank runs podman run), it is on pasta, and pasta installs no DNAT
for rootless-bridge traffic while refusing a second network outright. The
gateway alias does reach it (200) and is link-local, which is the same dead end
the Open Notebook URL guard hit. Measured with the move onto the bridge now in
the deployer, including the /proc/net/tcp line that shows 8730 is loopback-bound
and so stays private after the move.

Co-Authored-By: omp <omp@dankclaw>
2026-10-19 21:27:07 -07:00
Frankandomp 346e1d22f4 feat(frank-dragoon): put the endpoint on the app bridge so app containers can name it
An endpoint on podman's rootless default (pasta) has no address another
container can use, measured on 2026-10-19 from inside open-notebook:

  - 192.168.0.143:7770            connection refused (pasta forwards the
                                  published port in the host netns; no netavark
                                  DNAT exists for traffic arriving from a
                                  rootless bridge)
  - host.containers.internal:7770 200, but the alias resolves to 169.254.1.2 —
                                  link-local, which open-notebook's URL guard
                                  refuses to store
  - frank-dragoon-halogen:8731    name unknown: different netns
  - --network pasta --network dankclaw_default
                                  "cannot set multiple networks without bridge
                                  network mode" — pasta cannot be added to

So reaching the NPU endpoints from an app container means moving the container
onto dankclaw_default (FRANK_DRAGOON_NETWORK; empty keeps today's shape). This
is not an exposure change: the engine's unauthenticated token port binds the
container's own loopback (/proc/net/tcp: 127.0.0.1:8730 against 0.0.0.0:8731),
so only the API port becomes name-resolvable — the same trust the published
7770 already carries on the LAN. The deployer refuses a network that does not
exist rather than starting a unit that restart-loops on it.

Co-Authored-By: omp <omp@dankclaw>
2026-10-19 21:27:07 -07:00
Frankandomp 53ef121180 feat(frank-dragoon): run the XDNA2 NPU lane beside the flash model
halogen-flash-server already ships an NPU path on the pin we run: for each
model named in HALOGEN_NPU_MODELS it loads the vendor's precompiled AIE
programs (XDNA2 executes only AIE-compiled graphs) and answers them on
/v1/embeddings inside the SAME container behind port 7770. That gives the
fleet an in-house embedding endpoint without touching the iGPU trunk.

The lane is opt-in and one line wide: HALOGEN_NPU_MODELS blank is the
pre-lane endpoint byte for byte (no device, no mounts, no engine env).

The three host preconditions the vendor names are gated in the serve script
and mirrored in the deployer (the 2026-10-01 twin law), each refused by name:
  - the staged model files. /models is read-only and HALOGEN_DOWNLOAD stays
    unset so the container opens no outbound connection at boot, so
    scripts/download-frank-dragoon-npu.sh stages them from the pinned
    revision with the catalog's own byte counts and sha256s (the engine
    re-checks both at boot);
  - the distro XRT's libxrt_coreutil/libxrt_core/libxrt_driver_xdna, each
    mounted twice — where the image looks for XRT and at its own path —
    because those libraries find each other where they were installed and
    an /opt/xilinx/xrt of symlinks dangles inside the container;
  - the GPU fabric clock HELD. On Strix Halo the iGPU and the NPU co-running
    hang the machine (and corrupt NPU results) while FCLK changes speed, and
    halogen refuses to start in that state; it cannot read host sysfs from
    inside, so the hold is proven host-side. dankclaw/host/ carries the
    vendor's halogen-fabric-clock{,.service} verbatim at tag v0.16.2 (sha256
    a3a4f7f4e0142e71… / 5f9f1cc4c85a1c02…); the deployer installs, enables
    and gates it only while the lane is named, and releases the clock plus
    drops the boot hook when it is not.

After health, the deployer's lane gate is a real embedding through
/v1/embeddings — a 200 on /health proves nothing about the NPU programs.

Planner: dankclaw/host/halogen-fabric-clock* re-dispatches frank-dragoon
(the deployer installs it); dankclaw/host/grub and the fetch script plan
nothing.

Seam: 11 new tests — the off arm really renders nothing lane-specific, the
on arm renders the device/mounts/env, every missing precondition refuses by
name, the deployer twins and the host unit shape are pinned.

Co-Authored-By: omp <omp@dankclaw>
2026-10-19 21:22:50 -07:00
dank fa093e617d Merge pull request 'docs(vault): halogen NPU lane on the current pin, and the greedy-schema law' (#42) from feat/open-notebook-portal into main 2026-10-19 21:14:52 -07:00
Frank 6215f1a6c8 docs(vault): halogen has an NPU lane on the pin we already run, and it is greedy-only
- frank-dragoon: measured the response_format contract. json_schema and json_object answer
  schema-valid JSON at temperature 0 and draw a 400 at any sampled temperature, and because
  the sampler default is server-side an absent temperature is sampled. The page claimed
  "response_format is unsupported", which sends the next agent past a working feature.
- frank-dragoon: new "The NPU lane" section. The pinned 0.16.2 image already ships the
  five-program AIE catalog including qwen3-embedding-0.6b, so /v1/embeddings is one env var
  plus the accel device, the distro-XRT mount and the fabric-clock hold; embeddinggemma-2
  needs 0.18.3+, the stock-kernel-driver memory-budget fix 0.18.8+. Host census recorded:
  no /opt/xilinx/xrt, distro XRT 1:2.21.75 with libxrt_driver_xdna, clocks unheld.
- Open Notebook: scriptwriter probe verdict — greedy-only, and open-notebook's model record
  is {credential, name, provider, type} with no per-row config, so it cannot ask for greedy;
  frank-archon stays the writer. The embedding gap now has a named door instead of "build a
  new llama.cpp slot".
- Frank TTS: "the NPU is not a door at all" was too broad. It is not a door for the TTS
  family; halogen's precompiled catalog is a door on the same device.
- System Specs: NPU runtime row corrected to the distro XRT actually installed, and the
  stale "frank-zealot serves 27B" line retired in favour of [[frank-zealot]] as sole owner.

Tests: ./tests/run.sh, 455 passed.
2026-10-19 16:40:48 -07:00
dank 8bac97ec87 Merge pull request 'feat(open-notebook): guarded vhost, single origin, live homepage card' (#41) from feat/open-notebook-portal into main 2026-10-19 16:24:11 -07:00
Frank 69739237c8 docs(open-notebook): record the deployed shape and what the first render proved
The page described an undeployed service whose scriptwriter was going to be
frank-zealot. Both halves are now false, and the reason is worth more than the
correction.

- Status, ports, coordinates: deployed 2026-10-19, guarded vhost, API published
  on loopback only. The first episode is the proof: 747 s of speech in 301 s of
  wall clock (2.5x real time, knobs at 1), download route 200 with the bearer
  and 401 without.
- New section for the guard chain: provider-before-route, one upstream because
  the image rewrites /api/* itself, and the API_URL law — autodetect appends
  :5055, which is seerr's port on this host, so an unset API_URL yields a UI
  that loads and an app that cannot call itself.
- Provisioned records table (credentials, model rows, dank-duo, dank-daily) and
  the two contract traps measured on the live API: generate takes profile
  NAMES, and the fleet has no /v1/embeddings, so every note save fails its
  embed_note command while renders from content work fine.
- Why frank-archon writes the scripts: frank-zealot is reachable from this
  netns only at 169.254.1.2, and open-notebook's URL guard refuses to store a
  link-local base URL with no override. Census lands on [[Container Network
  Namespaces]], which also gains the root-vs-rootless publish rule — a
  root-published port is invisible to a rootless container, so "publish the
  port" was the wrong remedy as written there.
- Frank TTS: the request's model field is ignored; only voice selects a pack.

Tests: ./tests/run.sh, 455 passed.
2026-10-19 16:22:38 -07:00
Frank e5580abb6f feat(open-notebook): put the podcast generator behind the guard chain
The service came up internal-only: UI on the LAN at 8502, API at 5056, both
published on every interface, with only its own shared password in front. That
put the episode library outside the fleet's auth chain and one port scan away
from anything on the network.

- Caddy: guarded vhost notebook.franksdank.cloud (Anubis -> Authentik; provider
  notebook-proxy, internal_host caddy:8017) plus the post-auth route to the
  published UI port. One upstream is enough, because the image's own Next.js
  layer rewrites /api/* to the FastAPI beside it — so the API needs no outward
  face at all, and 5056 moves to loopback, where only the deployer's health gate
  dials it.
- scripts/open-notebook.env: API_URL and CORS_ORIGINS become the single guarded
  origin. Left to itself the app auto-detects <host>:5055, the port seerr holds
  on this host; a LAN-origin API_URL would also take the browser around the
  guard chain, where its authenticated session does not ride.
- addn-hosts: the LAN A record. The homepage card goes live and links the vhost.
- Seam: the API origin is derived from whatever the Caddyfile routes to 8502, so
  renaming the vhost moves the test instead of leaving it blessing a stale name,
  and a new test proves the vhost hands the WHOLE request to Anubis.
2026-10-19 16:13:53 -07:00
dank ee6d8610b3 Merge pull request 'fix(open-notebook): the database probe has to be exec-form JSON' (#40) from feat/open-notebook-portal into main 2026-10-19 16:07:53 -07:00
Frank 933420fd29 fix(open-notebook): the database probe has to be exec-form JSON
podman wraps a bare HealthCmd as CMD-SHELL (measured on Dankclaw's podman 5.7:
--health-cmd "/surreal is-ready" stores {"Test":["CMD-SHELL","/surreal is-ready"]}),
and the pinned SurrealDB v2 image is distroless — no /bin/sh, not even `ls`.
So the probe could never run: every 30 s interval exited 1 in ~30 ms with empty
output while the database was listening and answering, and open-notebook's first
deploy died at the gate ("did not become healthy in 90s"). The hand-check
`podman exec … /surreal is-ready` printed OK, which is why the old comment
believed the probe worked.

The JSON array survives the Quadlet generator (--health-cmd "[\"/surreal\",\"is-ready\"]")
and stores ["CMD","/surreal","is-ready"], which execs directly: a throwaway
instance of the pinned digest went healthy on that form. The seam is re-pinned
to the shell-free contract; the old assertion forbade quotes in any HealthCmd,
which is what pinned the broken shape.
2026-10-19 16:07:28 -07:00
dank 9ffa99902d Merge pull request 'fix(frank-archon): the state-dir guard names what the directory is now' (#39) from fix/frank-archon-state-dir-comment into main 2026-10-19 15:02:01 -07:00
Frank b8f564f3bd fix(frank-archon): the state-dir guard names what the directory is now
The refusal message still named "the /data state dir"; /data left with the
prefix disk tier. The serve script mounts /models ro, the rad-tune cache and the
chat template only, and tests/frank_archon_unit.bats asserts no :/data mount
exists, so the line an operator reads at the moment a boot refuses pointed at a
mount that is gone.

Text only. The guard still refuses: the block was run standalone with the
variable unset and every neighbour set - it prints the new message and exits 1.
bash -n clean, ./tests/run.sh green (454 tests).
2026-10-19 15:02:01 -07:00
dank d42d053abe Merge pull request 'docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape' (#38) from docs/frank-archon-181-two-slot into main 2026-10-19 14:56:59 -07:00
Frank 7eed636bd1 docs(perf): destroy §9's vendor absolutes, keep the ratios
The t/s ladder and the TTFT delta triple measure a configuration neither
deployed nor compared; per the purpose test they are dead weight. What survives
is the knob pricing (halving the shared chunk costs roughly a fifth of prefill
throughput), the long-prompt-only shape of the span win, and the method. Run-on
now names the era the census was taken in.
2026-10-19 14:55:43 -07:00
Frank d9b82b372e docs(frank-archon): reconcile the runbook with the live 1.8.1 two-slot shape
The endpoint moved to radiance 1.8.1 with MAX_SEQS=2 and the
--vram-weights-mib/--vram-kv-mib pair removed (PR #30); env, serve script and
bats carried the new shape while the page and the AGENTS.md block still
described the 1.0.13 3-slot era, so every point-of-use number was wrong.

Re-derived from the live checkout and the 2026-10-19 11:10 boot ledger, not
from the stale pre-#31 draft: runtime block (digest, version oracle, the two
CLI breaks that are boot refusals), shape table (demand-signal rows, derived
VRAM split, restored spec/batch row, host-only prefix tier, step sharing),
container contract (no /data mount), the measured 2-slot ledger, a new Eras row
for the retired 1.0.13 era whose rollback is a digest plus a trio rewrite, and
the laws: pool sizing and the per-card budget re-scoped to the pair era, a new
resolver law (funds ~90% of the worst case, host RAM below a slab step is inert,
MAX_SEQS is a demand signal), the sludge law marked measured-at-3-slots with the
fix ordered-not-verified, the knob-availability law flipped, and the slots law
now pricing the third slot in KV (4.20 -> 2.76 GiB).
2026-10-19 14:55:43 -07:00
dank 3d10a4ca4f Merge pull request 'docs+fix(frank-zealot): the adapter's measured effect, and a unit description that stopped lying' (#37) from feat/zealot-uncensored-lora into main 2026-10-19 14:47:59 -07:00
Frank e5b731216c fix(frank-zealot): the unit description still described the 27B era
`systemctl status` said `2x262144 slots over one 524288 pool, unquantized f16
K+V, MTP off` — three payload eras and two reslices out of date, and still
wrong about the thing that decides whether the endpoint can be trusted (the KV
dtype and whether the drafter is armed). It now quotes the env: 3x200704 over
one 602112 pool, f16 K + q8_0 V, MTP armed, adapter, Q8_0 vision.

Touching the unit selects the service in the planner, so this deliberately does
NOT come with a deploy: the hub is serving live conversations. The revision
marker stays behind until the next real frank-zealot change.
2026-10-19 14:47:30 -07:00
Frank ce8d82d4a7 docs(frank-zealot): record what the adapter did, and where provenance is read
Before/after on the two probe prompts (both opened `I cannot provide…`; neither
now carries a marker — one ran to its 2048-token cap, the other finished at
1930), the residual `Important Disclaimer` heading named for what it is (style,
not the circuit), and the same-boot proof that chat/tools/vision still answer.
Provenance now points at the boot ledger: the fork prints every
`prism.abliterate.*` key at each start, so the PTQ1_0-vs-PQ2_0 mismatch is
visible to anyone who boots the service — the page links that instrument
instead of restating it.
2026-10-19 14:47:30 -07:00
dank 02d04bc1f6 Merge pull request 'feat(frank-zealot): apply the refusal-direction adapter (rank-1 LoRA) at load' (#36) from feat/zealot-uncensored-lora into main 2026-10-19 14:43:26 -07:00
Frank 428ac22736 Merge remote-tracking branch 'origin/main' into feat/zealot-uncensored-lora 2026-10-19 14:43:06 -07:00
Frank 5d5cac4da3 docs(frank-zealot): the adapter is a payload row, a shape line, and two laws
- Shape table: an Adapter row (bytes, sha, what it edits, and the PTQ1_0
  provenance asymmetry), the fetch line now names both fetchers, the flag list
  carries --lora-scaled.
- One paragraph: the adapter changes behaviour, not the budget equation, and the
  probe that proves it must budget >= 2048 tokens because the thinking channel
  eats a short budget whole and returns empty content with finish_reason=length.
- Eras: the new era's rollback is one env line on a file already loaded.
- Laws: provenance is metadata, not a gate (tensor names, not hashes; an
  embedding rank would crash, which is why the publisher left token_embd alone);
  the fork pin now has a third reason, since older builds refuse the adapter with
  `unsupported prism.hadamard.version: 2`. Base-alignment law updated: the 9B
  refusal escape hatch was a fetch, and it has now been taken.
- AGENTS.md: topology payload cell and the zealot operator block.
2026-10-19 14:42:56 -07:00
Frank 3eaba57925 feat(frank-zealot): apply the refusal-direction adapter at load
Frank ordered the adapter; this adds it to the shape as one knob and one flag.

- `LORA_FILE` / `LORA_SCALE` in the env, `--lora-scaled <file>:<scale>` as ONE
  argv token so a scale can never reach the engine without its adapter. The
  rollback follows the mmproj convention exactly: `LORA_FILE=` (explicitly
  empty) drops the flag entirely, no rebuild, and `LORA_SCALE=0` is the card's
  own "inert but present" value.
- New fetcher `scripts/download-frank-zealot-uncensored-lora.sh`: `-f`, byte
  count, sha256, `--verify-only`, idempotent. The file is NOT LFS (a plain git
  object), so the pinned sha is the content hash, not an LFS oid.
- The deployer gates on the adapter beside the pack and the projector, and its
  error names the ADAPTER's fetcher — sending an operator to the bonsai9 script
  for a 3 MB file is how a payload ships without it.
- Seam: path+scale pinned as one token, the scale knob including 0, absence on
  the empty opt-out, a refusal to boot on a missing adapter, and the env ↔
  deployer anti-drift pair extended to LORA_FILE/LLM_ZEALOT_LORA. 454/454.

Provenance, stated where it will be read: the adapter's own metadata names
`Ternary-Bonsai-2-9B-PTQ1_0.gguf` (3561df7a…) as the pack the direction was
fitted against. This endpoint serves the PQ2_0+MTP+Q8_0 packing (c79fcdfa…),
which is not that file and is not on this host. The loader matches tensor names,
not hashes, so it applies — and build 10777 is the build the card verified on —
but the strength is calibrated on a sibling quantization. The proof is therefore
the refusal probe against the live endpoint, captured before and after, and not
the card's table.
2026-10-19 14:42:56 -07:00
dank 9b9c32005f Merge pull request 'docs(frank-tts): the engine computes on the gfx1151 now; the NPU question is closed' (#35) from docs/frank-tts-igpu-followup into main 2026-10-19 14:38:33 -07:00
Frank ea2fc6440b docs(frank-tts): record the iGPU cutover, close the NPU question
The cutover PR merged the runtime; this carries the surface that has to be
true afterwards.

- Engine page: the shape is now `--device Vulkan0` on the gfx1151 as a
  co-tenant beside frank-dragoon; the registry/thread/streaming laws gain a
  fourth — the device law — whose numbers stay in the verdict report, and the
  deploy record gets the cutover steps including the `glslc` configure gate.
  The CPU-era arm/rollback prose keeps the qualitative argument and drops the
  dead numbers per the purpose test.
- Endpoint page: "Where the engine computes" now says iGPU + co-tenancy, and
  states the NPU verdict in one paragraph instead of a research tour.
- Quadlet comment: it claimed supplementary groups are what grant render
  access. On this host the udev rule world-writes the render nodes and the
  container's `id` shows no groups at all, so keep-groups is belt-and-braces;
  the RSS note now matches the device shape instead of the CPU trial.
- Verdict report: gate 2 (contention) recorded as measured after the cutover,
  and the build-stage prediction corrected — the package is `glslc`, not
  `libvulkan-dev glslang-dev`.
- index.md regenerated (one description changed).
2026-10-19 14:37:29 -07:00
dank 6248f02ca2 Merge pull request 'feat(frank-tts-engine): synthesize on the gfx1151 iGPU (Vulkan), CPU stays as the rollback' (#34) from feat/frank-tts-vulkan into main 2026-10-19 14:27:11 -07:00
Frank 25df9d05a2 feat(frank-tts-engine): synthesize on the gfx1151 iGPU via Vulkan
The NPU path is a dead end for this payload (verdict:
report/tts-npu-spike-verdict-2026-10-19.md); the iGPU is the accelerator that
exists on this box today, and it is already shared with frank-dragoon.

- Containerfile.engine: -DGGML_VULKAN=ON, plus the three packages the door
  needs to configure — libvulkan-dev, spirv-headers, and glslc, which Ubuntu's
  glslang-tools does NOT ship (find_package(Vulkan COMPONENTS glslc REQUIRED)
  fails without it) — and libvulkan1 + mesa-vulkan-drivers (the RADV ICD) at
  runtime. No NVIDIA ICD on purpose: one Vulkan device in the enumeration is
  what makes the NAME Vulkan0 safe to write down.
- tts-engine-start.sh: QWENTTS_DEVICE -> --device, default Vulkan0. A value
  that is set but EMPTY passes no flag at all, which is the CPU rollback; that
  is why the default is written as an -v test and not with :=, because := also
  fires on an empty value and would silently undo the rollback.
- quadlet: AddDevice=/dev/dri + GroupAdd=keep-groups (a rootless container that
  drops its supplementary groups enumerates no render device). The node is not
  named: this host is hybrid, renderD128 is the NVIDIA card and renderD129 the
  amdgpu iGPU.
- env: QWENTTS_DEVICE=Vulkan0 plus the device law in the header, including the
  rollback line.
- tests: the two CPU-only assertions now assert the Vulkan shape, and a new one
  pins the rollback (empty device => no --device in argv). 451/451 green.

Measured before writing any of it: the configure in the digest-pinned base
answers "Found Vulkan: ... found components: glslc" and "Including Vulkan
backend"; gate 1 (RADV GFX1151 visible to a rootless container off this base)
is recorded in the verdict report.
2026-10-19 14:26:58 -07:00
dank 7601edfe28 Merge pull request 'docs(LLM): the decode stall a prefill causes, and the knobs that turn it' (#27) from docs/radiance-prefill-decode-census into main 2026-10-19 14:21:45 -07:00
Frank e133d2412d Merge origin/main into docs/radiance-prefill-decode-census
Two conflicts, both from the 2026-10-19 vault compaction on main:

* frontmatter description — main shortened it to satisfy OKF DESC
  (max 120 chars); this branch's longer wording would have failed the
  seam. Kept main's shape and folded the new §9 in by naming
  step-shape attribution instead of profiler attribution.
* the Related bullet for [[vLLM Porting and Compile Triage]] — that
  page was destroyed on main, so the link is dropped rather than
  resurrected; this branch's [[frank-archon]] row is kept.

Index rows regenerated with scripts/gen-vault-index.py.
2026-10-19 14:20:05 -07:00
dank 9971cb559e Merge pull request 'docs(frank-tts): NPU spike verdict + iGPU co-tenancy plan' (#33) from docs/npu-tts-verdict into main 2026-10-19 14:14:46 -07:00
Frank 065d796d73 docs(frank-tts): renderD128 is the NVIDIA node — pass /dev/dri, not a guessed number
The trial's quadlet line said AddDevice=/dev/dri/renderD128. Resolved the nodes by
driver instead of assuming: on this hybrid host renderD128 -> nvidia and
renderD129 -> amdgpu, so the guess pointed the engine at the wrong card. Also
records the gate-1 command that works (vulkaninfo --text; --batch prints help and
exits 1) and the probe script path.
2026-10-19 14:12:05 -07:00
Frank 1e6b39c382 docs(frank-tts): gate 1 passes — RADV gives a container tenant the aperture, not the carve-out
vulkaninfo inside the engine's own base image (ubuntu:24.04 + mesa-vulkan-drivers
vulkan-tools, --device /dev/dri, rootless) reports Radeon 8060S Graphics (RADV
GFX1151) / driver radv / API 1.4.318 with two heaps: heap 0 host-visible 38.67 GiB
(budget 32.58) and heap 1 DEVICE_LOCAL 77.33 GiB (budget 65.16) with frank-dragoon
loaded. ggml's Vulkan backend asks for DEVICE_LOCAL, so the ~4 GiB the TTS engine
needs lands in the 65 GiB budget: memory is not the constraint, and the 2 GiB DRM
carve-out never enters the picture. Both heaps are GTT — the same host RAM, so the
budget is a ceiling on mappings, not free silicon memory.

Traps recorded: vulkaninfo --batch is not a flag in Ubuntu 24.04's vulkan-tools
(prints help, exits 1; use --text), and the container has no NVIDIA ICD, so Vulkan
device 0 happens to be the iGPU — select by name, not index.

Probe: /home/dank/spikes/npu-tts/vk_probe.sh on Dankclaw. Enumeration only; no
allocation, no compute, production untouched.
2026-10-19 14:10:12 -07:00
Frank 1e52e45fae docs(frank-tts): record the NPU spike verdict and the iGPU co-tenancy plan
The overnight spike settled whether frank-tts can run on the Strix Halo NPU.
It cannot, and the wall is the loader, not the silicon: AMD's own Linux
runtime opens /dev/accel/accel0, succeeds at the GEM ioctls, then declines
the NuTTS-Air graph without emitting a single log line — and the one run that
did load has never reproduced. The accelerator that does exist is the gfx1151
iGPU behind -DGGML_VULKAN=ON, a flag our own Containerfile.engine writes OFF,
and Frank set the target shape to co-tenancy: the engine resident beside
frank-dragoon, not a handover.

- report/tts-npu-spike-verdict-2026-10-19.md owns the measured result: the
  exoneration chain, the two arms' real-time bars (12.5 codes/s deployed vs
  50.00 for NuTTS-Air), the 65,536-entry speech-token census (ids
  151,671-217,206), the host gates (rootless cannot read accel0; dpkg-deb the
  vendor XRT, never dpkg -i), the iGPU census (18,523 of 116,733 MiB GTT in
  use, 17 GiB host RAM available, GTT pages are host pages), the two gates
  that decide co-tenancy (RADV heap for an APU tenant; dragoon's TTFT with a
  synthesis in flight), and the five-file change surface for the trial.
- report/npu-tts-runtime-landscape-2026-10-19.md and
  report/qwen3-tts-payload-portability-2026-10-19.md are the research slices;
  both now point at the verdict page instead of ending in loose ends.
- The vault page gains "Where the engine computes", so an operator doing a TTS
  task learns the NPU is a closed door without re-walking it.

No fleet change: frank-tts / frank-tts-engine still serve narrator1-3 on CPU.
tests: ./tests/run.sh 450/450 green.
2026-10-19 14:02:34 -07:00
dank 88342c144b Merge pull request 'fix(frank-archon): declare two slots, revert the inert host-pool raise' (#32) from fix/frank-archon-two-slots into main 2026-10-19 11:09:24 -07:00
Frank 4d41366f4c fix(frank-archon): declare two slots, revert the inert host-pool raise
FRANK_ARCHON_MAX_SEQS 3 -> 2. At 1.8.1 the budget resolver sizes the KV cache
from the cache bound (--max-model-len x --max-num-seqs), --gpu-headroom-mib and
--host-pool-mib, and the shipped CLI has no KV-size flag (the
--vram-weights-mib/--vram-kv-mib pair was removed at 1.3.0). The live boot
ledger reads that cache at 4.20 GiB = 556,348 tokens = 2.8 sessions at 200,000,
so a third full-height conversation is refused while /health answers green and
the harness advertises capacity the engine cannot honour. Two slots is what the
shape actually serves at the full window; a slot costs KV, never decode, and the
fused TP all-reduce drops from 9 rows to 2 x (1 + spec) = 6, still inside
wht6's 25-row bit-exact bound.

FRANK_ARCHON_HOST_POOL_MIB back to 44928 and FRANK_ARCHON_TTM_MIN_MIB back to
150000, reverting PR #30. That raise bought cache the resolver never hands out:
the two boot ledgers differ in exactly one line -- host_pool 44.47 -> 45.09 GiB
-- while vram_weights (25.81 GiB) and vram_kv (4.20 GiB / 556,348 tokens, both
ranks) are byte-identical, so the extra 640 MiB sat unclaimed in the host pool
while the TTM floor moved out from under the preflight's arithmetic. The
compute-bound cap on --host-pool-mib is unchanged either way (45,568 -> 22,222
MiB of experts either side of the 22,222 clamp).

The bats seam pins the new contract: MAX_SEQS=2, pool 44,928, and the ledger
floor that computes to 148,920 MiB at that pool against the 150,000 minimum.
2026-10-19 11:07:11 -07:00
dank d09d2745cf Merge pull request 'docs(vault): documentation exists for point of use — destroy the ledgers (−13.7k lines)' (#31) from docs/vault-destruction-20261019 into main 2026-10-19 10:59:51 -07:00
Frank a3bce7d123 docs(agents): documentation exists for point of use
*Documentation Policy* rewritten: a page earns its bytes only if it changes what
the reader does next. Measurements are disposable — a rate for anything not
running and not being compared is destroyed, not filed in an Eras row or a
technique page. "It is the only copy" is explicitly NOT a reason to keep a fact.
`log.md` retired from the policy; the Runbook Index no longer promises a page per
retired endpoint (rollback is a commit, not a document).

*Key Lessons Learned* purged of the dead: the dual-R9700 board-swap narrative and
the layer-split/x4-pair experiment laws (no live endpoint uses either), the QSA
patch note and the retired Kairic/FastMTP traps, and the throughput figures inside
the laws that survive — the law stays, the number goes. Two lessons added, both
from this campaign's own failures: a verdict is a lead, the scout's and the tool's
alike; and never let a census decide what to keep, because a tool can say where a
fact lives but not whether anything needs it.

997 → 820 lines.
2026-10-19 10:56:37 -07:00
Frank 327e78b950 docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
2026-10-19 10:56:37 -07:00
Frank 8775e64ed0 feat(scripts): tools to destroy documentation, not preserve it
Point-of-use enforcement needs hands, not advice:

- `destroy-vault.py` — moves the pages on a kill list to `.AGENTTRASH/` and then
  deletes every reference to them from the survivors, including the whole line
  when the dead link was the line's only content. Stems are seeded from the trash
  so the pass is re-runnable; `pathlib.Path.stem` is NOT used to compare link
  targets, because it strips the real suffix off names like `sd.cpp`.
- `axe-measurements.py` — deletes bench/throughput ledger sections and rate rows,
  blanks any table cell that reports a rate, and folds Eras older than the current
  one into a single row that says the numbers were destroyed and that rollback is
  a commit. Never touches a shape number or a Law/Wire/Rollback section.
- `repair-damage.py` — fixes the two artifacts those passes can leave (a list
  marker that lost its space, a table separator left without its header).
- `restore-punctuation.py` — restores a line whose only difference from HEAD is
  stripped trailing punctuation. Written after `repair-damage.py`'s first version
  matched every wrapped prose line ending in a comma and deleted 660 of them: the
  rule is now `… see .` only, and the incident is recorded in its docstring.
- `gen-vault-index.py` — never walks into `.AGENTTRASH/`, and sweeps rows whose
  page cell is empty (a description left nameless by a destruction).

Full suite: 451 tests pass.
2026-10-19 10:56:37 -07:00
dank 64765c6a47 Merge pull request 'fix(frank-archon): size the host pool so the KV cache clears 3 x 200,000' (#30) from fix/frank-archon-cache-ceiling into main 2026-10-19 10:30:13 -07:00
Frank d5934a6b47 fix(frank-archon): size the host pool so the KV cache clears 3 x 200,000
The 1.8.1 boot resolved the budget resolver to 4.20 GiB of cache a card —
556,348 tokens, 2.8 sessions at full height — so three concurrent
full-height requests are refused with a 400 at admission: the endpoint's
3-slot contract is broken by the cutover, silently, because /health and
every shorter conversation still answer.

The resolver gives the expert plane whatever the pinned host pool cannot
hold and the cache the rest, so the pool is the lever: +640 MiB a rank
moves that much expert demand into host RAM and the cache wins it back.
Tokens scale with the GiB won (8.11 KiB/token at fp8), so ~0.63 GiB buys
~44,000 tokens — 600,000 with margin — at ~2.5% off the expert slab.
TTM floor moves with the pool (2 x pool + 48,824 + prefix + MNBT =
150,200 MiB computed by the ledger test).

Verified by the boot ledger's `kv:` line and the probe's capacity leg,
not by this arithmetic.
2026-10-19 10:29:20 -07:00
dank 40ad1f4a60 Merge pull request 'feat(frank-archon): radiance 1.8.1 — the budget resolver, the sharing knobs, and the CLI breaks between them' (#29) from feat/frank-archon-radiance-181 into main 2026-10-19 10:21:27 -07:00
Frank 9de58ed6fd feat(frank-archon): move the endpoint to radiance 1.8.1 and its budget model
The decode stall is step contention (measured: 18.7 ms decode-only steps
against 317 ms steps that carry a prefill chunk, decode at 13 t/s median
while prefill runs), and the two engine knobs that fix it arrived after this
endpoint's pin. Moving to 1.8.1 (2026-10-18) brings both, and brings two CLI
breaks with them:

  - --vram-weights-mib / --vram-kv-mib were REMOVED at 1.3.0. The per-card
    split between the expert slab and the KV pool is now derived by the
    engine's budget resolver from --gpu-headroom-mib, the cache bound
    (--max-model-len, --max-num-seqs) and --host-pool-mib, and the cache may
    lend idle blocks to the expert slab. The slab size and the pool's token
    count are boot-ledger lines from here on, so the capacity contract
    (3 x 200,000 fp8) is verified live by the probe's capacity leg instead of
    computed from an env number.
  - The prefix cache has no disk tier any more: --prefix-cache-dir and
    --prefix-cache-disk-mib are refusals and --prefix-cache-host-mib is
    --prefix-cache-mib. 64 GiB of parked KV on NVMe becomes re-prefill, so
    prefix_hit_rate is the thing to watch after this boot.

env: image re-pinned to the 1.8.1 digest; the pair and the disk tier dropped;
FRANK_ARCHON_PREFIX_CACHE_MIB replaces the host/disk pair; new
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto (explicit, because auto vs off is the
difference between decode waiting one chunk and waiting a whole step) and
FRANK_ARCHON_PREFILL_INTERLEAVE=0 (opt-in through a guarded argv array).

serve: argv follows the new CLI, /data mount and the pre-created kvcache path
go with the disk tier, and the retired env keys are no longer required (they
would abort ExecStart). autoresearch-frank-archon.sh takes the same CLI, with
a cold prefill now coming from --prefix-cache-mib 0 instead of a per-run disk
directory. run-swift15-bf16.sh is untouched: it pins its own engines by design.

tests: the two budget-arithmetic seams asserted knobs that no longer exist and
are replaced by one seam that asserts the retired knobs are ABSENT from env and
argv - a stale line is a boot refusal, not a no-op. The container-contract seam
now asserts the tune mount and the absence of /data.
2026-10-19 10:18:24 -07:00
Frank e3a02e25c7 docs(LLM): census the decode stall a prefill causes, and the knobs that turn it
Live step-shape census on frank-archon (radiance 1.0.13, boot of 2026-10-15):
decode-only steps run 18.7 ms, prefill+mixed steps 317 ms — 60 % of engine
wall-clock inside 8 % of the steps — and a day of Prometheus shows decode at
13.3 t/s median while prefill runs >=1000 t/s against 106-240 t/s when it does
not. The lag test says the penalty is confined to the prefill window: no
lingering expert-residency tail, and preemptions are zero.

Adds the method as LLM Performance Diagnosis S9 (counter arithmetic, the lagged
bucket correlation that separates step contention from residency churn, and the
four radiance knobs with their published costs), and the endpoint's own numbers
plus the verdict as two frank-archon laws: --busy-prefill-chunk and
--prefill-interleave are absent from this pin (proven via the running
container's --help), so MNBT is the only crank here and an image bump is gated
by the 1.2.0 numerics dead end.

No behavior change; docs only.
2026-10-19 04:11:20 -07:00
202 changed files with 6467 additions and 14914 deletions
+101 -107
View File
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj | 7771 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj + rank-1 refusal adapter | 7771 |
Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
@@ -47,7 +47,7 @@ per attempt and no separate swap-history file.
The managed skill library is distilled there one page per skill-cluster:
`LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
`Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
pages. There is no `log.md`: the change diary was destroyed 2026-10-19
- `tests/` — bats suite + Playwright e2e
## Git Workflow — trunk-based (fleet law, aligned 2026-10-18)
@@ -166,9 +166,11 @@ bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet`
dashboard are provisioned from `shared/grafana/provisioning/`, and
`shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet`
(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert
+ recording rules, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
@@ -318,35 +320,36 @@ the next), so never report request #1.
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.
## Documentation Policy (fleet law, 2026-10-18)
## Documentation Policy (fleet law, 2026-10-19)
**One fact, one owner.** A page owns exactly one thing — an endpoint, a
service, or a technique — and every other file links to it with `[[Page]]`.
Restating a shape, a digest, a port list, or a measured number in a second
file is a defect: it is the next thing to go stale.
**Documentation exists for point of use.** A page earns its bytes only if it
helps whoever performs the next task — the shape to deploy, the flag that must
be passed, the failure signature to recognise, the rollback to run. It is not a
lab notebook and not a ledger. **Losing information is an acceptable, usually
correct outcome.** Never let an orphan census ("this is the only copy of that
fact") decide what to keep: sole custody of a useless fact is still useless.
- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
— current shape, an **Eras** table (`window | payload | runtime | shape |
replaced because | rollback`), the **Laws** the endpoint taught, the wire
contract, and the files that own it. A payload swap edits the shape block
and appends one Era row. It never creates a page, and the retired era's
page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
table.
- **`log.md` records incidents and laws** — anything with a causal story
(why the host went down, why a knob resets at reboot, why the RAM was the
fault). A model swap is not an incident: it is an Era row and nothing else.
- **`AGENTS.md` carries operator-critical numbers only** — port, context
ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
- **The purpose test — before you write, and again before you keep:** would the
reader act differently because of this line? No → delete it.
- **Measurements are disposable.** A rate or throughput figure (t/s, tok/s, ms,
acceptance %, a bench ladder, a measured MiB) for a payload, runtime, or
layout that is not running now and not being compared now is destroyed, not
filed — not in an Eras row, not on a technique page, not "preserved for
history". AI moves faster than this vault: the number is wrong before anyone
reads it, and a wrong number in a runbook gets believed. Keep the **law** the
measurement taught; drop the measurement.
- **One fact, one owner**, every other file links with `[[Page]]`. The env file
beside the serve script is the authority for every live number; the endpoint
page is the authority for the shape and the wire contract.
- **One page per endpoint:** current shape, **Eras** (`window | payload |
runtime | replaced because | rollback` — the rollback is a commit, not a
number), the **Laws**, the wire contract, the files that own it. A swap edits
the shape block and appends one row; the retired era's own page is
**destroyed** to `.AGENTTRASH/`, never merged into anything.
- **`log.md` is retired** (2026-10-19: 2,380 lines destroyed). Git holds the
chronology. A causal law belongs to the page that owns the thing it governs,
or to *Key Lessons Learned* below if it is cross-cutting — never to a diary.
- **Index pages are locators:** one row, one clause, no numbers.
- **A measured number is written once:** in the env-file header beside the
serve script that consumes it, and in the endpoint page. Every other page
cites it by link.
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
technique pages: they stop carrying a per-endpoint throughput table or a live
shape, but a campaign ledger that exists nowhere else is owned by the page that
measured it. The defect is two pages carrying one figure, not a number living on a
technique page — move a figure only when an endpoint page or an env header is its
home; otherwise leave it in place and name the era it measured.
## Model Config (harness)
@@ -439,7 +442,11 @@ disagrees with it is the bug.
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
loader ignoring that metadata dequantizes wrong); payload
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
tower can crash the server with concurrent image requests).
tower can crash the server with concurrent image requests) + the rank-1
refusal-direction adapter applied at load by `--lora-scaled …:1`. The
adapter's rollback is `LORA_FILE=` (empty) in the env — one line, no rebuild
— and build 10777 is also the gate that lets it load: older forks refuse it
with `unsupported prism.hadamard.version: 2`.
- Shape: `--ctx-size 602112 --parallel 3` = **three conversations at
200,704 each**, `CACHE_TYPE_K=f16` + `CACHE_TYPE_V=q8_0` (asymmetric: 25,088
B/token, 14,406 MiB of pool), batch 6144 / ubatch 1024,
@@ -457,12 +464,15 @@ disagrees with it is the bug.
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
`stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
`--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
+ 64 GiB disk, n-gram table pinned to RAM.
`stilldeadcode/radiance@sha256:d590e300…` (radiance 1.8.1); container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 2`, `--max-model-len 200000`,
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, `fp8` KV,
MTP depth 2, MNBT 2048, `--busy-prefill-chunk auto`, prefix cache 8 GiB host
tier only, n-gram table pinned to RAM. The per-card weights/KV split is derived
by the engine's budget resolver from `--gpu-headroom-mib`, the cache bound and
the host pool; the `--vram-weights-mib`/`--vram-kv-mib` pair and the prefix disk
tier are gone (removed at 1.3.0 / 1.8.1) and naming either is a boot refusal,
not a no-op.
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
is why this is a root unit, and `ttm-preflight.sh` raises
@@ -487,36 +497,10 @@ linked runbooks.
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board
whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH
endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB
per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack
gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce
here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
failing HIP call with its return code. [[PCIe Link and Atomics]]
- **Layer split is the no-collective path, on one vendor's cards and across two — and
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce),
and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But
`--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a REQUIRED term of the
scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a
pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with
the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is
last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix.
Pin the spec-decode drafter to one device; on a split its home is the card holding the target's
last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned —
the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in
the adapter or cable (card and port are both wider). For any engine that STREAMS weights across
it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per
token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM
(`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care.
[[x4-pair-experiment]]
**ROCm / GPU**
@@ -530,7 +514,7 @@ linked runbooks.
the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]]
and pinning the PLE tensor with `-ot` was slower than letting it page. [[Llama Cpp Serving Sizing]]
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
`1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
@@ -545,9 +529,6 @@ linked runbooks.
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
`blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the
kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]]
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
`/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
@@ -602,13 +583,11 @@ linked runbooks.
`LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
before adding flags. [[frank-dragoon]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a
loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103
(`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a
preference but the only door. Counting tensor types out of the GGUF KV section takes one python
read and settles a runtime argument in minutes. Build identity matters as much as the fork: the
2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a
file to a loader.** A non-standard tensor type in the KV section makes stock loaders refuse
the file at load, so a fork can be the only door rather than a preference. Counting tensor
types takes one python read and settles a runtime argument in minutes; build identity matters
as much as the fork, so pin the commit by SHA in any build script. [[GGUF Compatibility]]
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
(the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
@@ -617,12 +596,6 @@ linked runbooks.
exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
back before believing a number — and kill by pid, never `pkill -f llama-server`.
[[LLM Bench Windows and Measurement]]
- Retired-runtime traps that can come back: **Kairic Edge** needs
`third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean`
removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so
re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS
patch, since stock binaries fail the draft load with a tensor-shape mismatch.
**llama.cpp serving**
@@ -635,10 +608,10 @@ linked runbooks.
`--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
`ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
`draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash`
legal and it wins on prompt-echo traffic, but `draft-mtp,draft-dflash`
dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs
hybrid loses, because adding the ngram layer logs
`recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
[[frank-dragoon]]
@@ -647,7 +620,7 @@ linked runbooks.
declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
[[Llama Cpp Multi-Device Splits]]
[[Llama Cpp Serving Sizing]]
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
@@ -664,21 +637,20 @@ linked runbooks.
re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at
512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by
restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and
[[frank-zealot]].
control that proves the knob, not the traffic, moved the number. The optimum is a property of
the backend, so re-find it after any device or payload change. When a bench IS ordered, A/B it
by restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
port: one card holds one server.
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
[[Llama Cpp Multi-Device Splits]]
[[Llama Cpp Serving Sizing]]
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
too short and it proposes on ordinary prose fragments, costing verification without acceptance.
[[Llama Cpp Multi-Device Splits]]
[[Llama Cpp Serving Sizing]]
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
@@ -793,17 +765,24 @@ linked runbooks.
`curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three;
ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account
accepts the gate. [[Model Swap and Rollback]]
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six
read-only scouts worked the same pages: one classified era pages as stale *because* they
document retired payloads (that is an era page's content), one reported files that do not exist,
one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held
by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and
one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every
claimed duplicate, every claimed staleness and every quoted holder against the file and today's
fleet state before editing; a scout yield buys no evidence.
declared size, and check the LFS sha256 — the live `scripts/download-frank-*.sh` scripts do
all three; ad-hoc fetches tend not to. Gated repos 403 even with the host's HF token until
that account accepts the gate. [[Model Swap and Rollback]]
- **A verdict is a lead — the scout's and the tool's alike.** In the 2026-10-19
documentation campaign six scouts worked the same pages and produced invented
bullet IDs, files that do not exist, "duplications" grep disproved, line
numbers past the end of the file they judged, and Eras tables with invented
payloads. The mechanical tool was worse: `scripts/port-census.py` built its
corpus from the directory containing the page under test, so each page was held
by *itself* and ten era pages reported "0 sole-record lines" — which reads as
"pure duplication, trash it". Nothing was cut on a verdict; every holder was
re-grepped and every tool re-measured before it was believed.
- **Never let a census decide what to keep.** That same campaign then concluded
the era pages were irreplaceable because each held 14–63 sole-record lines —
wrong in the opposite direction: those were throughput figures for payloads that
no longer run. Point of use is the only ownership test, the operator applies it,
and 53 pages plus the 2,380-line change diary were destroyed as a result. A tool
can say where a fact lives; it cannot say whether anything needs it.
## Runbook Index
@@ -818,12 +797,27 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.
There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself
dies, its page is destroyed. Rollback is a commit, not a document.
`serverAppConfigs-docs/index.md` is the locator for every page that survives;
retired scripts live in `old/` and `.AGENTTRASH/`.
## Agent skills
### Issue tracker
Issues live in Gitea (`dank/serverAppConfigs` on Dankclaw), via its REST API with the stored credential. See `serverAppConfigs-docs/Agent/Issue Tracker.md`.
### Triage labels
The five canonical roles keep their default strings (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `serverAppConfigs-docs/Agent/Triage Labels.md`.
### Domain docs
Single-context: root `GLOSSARY.md` + `docs/adr/`, created lazily; this vault stays the owner of operational facts. See `serverAppConfigs-docs/Agent/Domain Docs.md`.
## Why
+29
View File
@@ -162,6 +162,25 @@ marreta.franksdank.cloud {
}
}
# --- Open Notebook (guarded: Anubis → Authentik; notebook-proxy provider, internal_host caddy:8017) ---
# The podcast generator. ONE upstream carries the whole app: the image's own
# Next.js layer rewrites /api/* to the FastAPI beside it (INTERNAL_API_URL,
# default http://localhost:5055 inside the container), so a browser on this
# vhost never needs the API port. That is why scripts/open-notebook.env sets
# API_URL to exactly this origin — left unset, the app auto-detects
# `<host-you-opened>:5055`, and 5055 on this host belongs to seerr — and why
# the published API port sits on loopback. App-level login
# (OPEN_NOTEBOOK_PASSWORD) stays on behind the chain: two owners, edge identity
# and the app's own session.
# Dialled by published LAN port, not container name: open-notebook is a
# rootless quadlet and this Caddy lives in root's storage, so root's DNS does
# not see the container (the same reason marreta and SearXNG are dialled by IP).
notebook.franksdank.cloud {
reverse_proxy anubis:8923 {
header_up X-Real-Ip {remote_host}
}
}
# --- Pi-hole (quadlet on slirp4netns; published 8083) ---
pihole.franksdank.cloud {
basicauth {
@@ -420,6 +439,16 @@ http://:8017 {
handle @marreta {
reverse_proxy 192.168.0.143:8082
}
@notebook host notebook.franksdank.cloud
handle @notebook {
# Bare proxy, like ai.franksdank.cloud: the app's SSE answers
# (/api/search/ask, chat sessions) stream for minutes through this
# same listener with no flush or timeout tuning, so nothing here
# needs it — a read/write timeout on the route would be the thing
# that cuts them.
reverse_proxy 192.168.0.143:8502
}
}
# --- Optional: Jackett ---
+65
View File
@@ -0,0 +1,65 @@
#!/bin/sh
# halogen-fabric-clock hold | release | status: the AMD GPU's fabric clock (FCLK), held at its top level.
#
# On Strix Halo the GPU and the NPU working at the same time can hang the machine, and corrupt the NPU's results,
# while the GPU's fabric clock changes speed; held at its top level, the two run together cleanly. halogen-flash-server
# with HALOGEN_NPU_MODELS refuses to start until the clock is held. This holds it: the performance level `manual` and
# only the top pp_dpm_fclk level, on every amdgpu device that has the control (the GPU and SoC clocks stay automatic).
# Cost: about 1 W at idle. Run as root; halogen-fabric-clock.service runs `hold` at every boot and after a resume.
#
# hold hold it (waits up to 60 s for the GPU driver to come up)
# release give it back to the driver (the performance level `auto`)
# status print each device's state; exit 0 when every device is held
set -u
SYS="${HALOGEN_SYSFS:-/sys}"
devices() {
for d in "$SYS"/class/drm/card*/device; do
[ -w "$d/pp_dpm_fclk" ] || [ -r "$d/pp_dpm_fclk" ] || continue
[ -e "$d/power_dpm_force_performance_level" ] || continue
echo "$d"
done | sort -u
}
held() { # held DEVICE_DIR: the performance level high, or manual with only the top fclk level selected
perf=$(cat "$1/power_dpm_force_performance_level" 2>/dev/null)
[ "$perf" = high ] && return 0
[ "$perf" = manual ] || return 1
top=$(grep . "$1/pp_dpm_fclk" | tail -n 1)
[ "$(grep -c '\*' "$1/pp_dpm_fclk")" = 1 ] && [ "${top%\*}" != "$top" ]
}
cmd="${1:-status}"
case "$cmd" in
hold)
n=0
while [ -z "$(devices)" ] && [ "$n" -lt 60 ]; do sleep 1; n=$((n + 1)); done
[ -n "$(devices)" ] || { echo "halogen-fabric-clock: no amdgpu device with a fabric clock control (pp_dpm_fclk) under $SYS/class/drm" >&2; exit 1; }
rc=0
for d in $(devices); do
lvl=$(grep . "$d/pp_dpm_fclk" | tail -n 1 | cut -d: -f1)
# the driver applies the level several seconds after the write, so the read-back waits up to 30 s
ok=0
if echo manual > "$d/power_dpm_force_performance_level" && echo "$lvl" > "$d/pp_dpm_fclk"; then
t=0; until held "$d" || [ "$t" -ge 300 ]; do sleep 0.1; t=$((t + 1)); done
held "$d" && ok=1
fi
if [ "$ok" = 1 ]; then
echo "halogen-fabric-clock: $d: held at level $lvl ($(grep . "$d/pp_dpm_fclk" | tail -n 1 | awk '{print $2}'))"
else
echo "halogen-fabric-clock: $d: could not hold it (level $lvl)" >&2; rc=1
fi
done
exit "$rc" ;;
release)
for d in $(devices); do echo auto > "$d/power_dpm_force_performance_level" && echo "halogen-fabric-clock: $d: released (auto)"; done ;;
status)
[ -n "$(devices)" ] || { echo "halogen-fabric-clock: no amdgpu device with a fabric clock control"; exit 1; }
rc=0
for d in $(devices); do
if held "$d"; then echo "$d: held ($(cat "$d/power_dpm_force_performance_level"))"
else echo "$d: NOT held ($(cat "$d/power_dpm_force_performance_level"))"; rc=1; fi
done
exit "$rc" ;;
*) echo "usage: halogen-fabric-clock hold | release | status" >&2; exit 2 ;;
esac
@@ -0,0 +1,19 @@
# Holds the AMD GPU's fabric clock at its top level at boot and after every resume, for halogen-flash-server with
# the NPU (HALOGEN_NPU_MODELS): on Strix Halo the GPU and the NPU at once are safe only with that clock held.
# Install (as root), from this directory:
# install -m 755 halogen-fabric-clock /usr/local/sbin/
# install -m 644 halogen-fabric-clock.service /etc/systemd/system/
# systemctl daemon-reload && systemctl enable --now halogen-fabric-clock.service
# Check: halogen-fabric-clock status. Undo: systemctl disable halogen-fabric-clock.service, then
# halogen-fabric-clock release (or reboot). The unit runs once at boot and once after each resume (a resume can hand
# the clock back to the driver); it holds nothing open, so it shows inactive between runs.
[Unit]
Description=Hold the AMD GPU fabric clock at its top level (GPU and NPU together)
After=systemd-modules-load.service suspend.target hibernate.target hybrid-sleep.target suspend-then-hibernate.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/halogen-fabric-clock hold
[Install]
WantedBy=multi-user.target suspend.target hibernate.target hybrid-sleep.target suspend-then-hibernate.target
+35 -39
View File
@@ -1,55 +1,51 @@
# Quadlet unit: alloy (rootless, boot-persistent).
# Quadlet unit: alloy (ROOT system service, boot-persistent).
#
# Install path on dankclaw: ~/.config/containers/systemd/alloy.container
# Unit name: alloy.service (container name: alloy)
# Install path on dankclaw: /etc/containers/systemd/alloy.container
# Unit name: alloy.service (container name: alloy) — SYSTEM manager, root.
#
# Log shipper. This replaces the compose service of the same name in
# dankclaw/compose.yaml ("Observability" block).
# WHY ROOT (not rootless, unlike its observability siblings). Alloy reads the
# host journal. The journal files are root:systemd-journal 0640 (other::---);
# a rootless container carries no supplementary groups and cannot open them —
# a rootless read of system.journal returns DENIED (see the Alloy page in the
# vault). Running as root, one loki.source.journal reads system.journal AND
# user-1000.journal, so it carries both host logs and the rootless Quadlet
# containers' logs. Validated 2026-10-20: a root probe read 31,825 journal
# lines and pushed 26,966 into Loki in 20 s, zero drops.
#
# IMAGE. franksdank/alloy is built locally by deploy-remote.sh
# (build_compose_image) from dankclaw/alloy/Containerfile: upstream
# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a
# boot from reaching for that local tag at a registry.
# NETWORK=host. Loki publishes 127.0.0.1:3100 (loopback only) on the host, and
# no bridge container — rootless or root — can reach the host's loopback, so
# this unit runs on the host network and pushes to 127.0.0.1:3100. The side
# effect: the UI must bind loopback too, so Exec sets the listen-addr to
# 127.0.0.1 (the old rootless unit used 0.0.0.0 only because container loopback
# was not host loopback; under Network=host they are the same address, so the
# unauthenticated UI stays loopback-only — nothing off-host reaches it).
#
# NETWORK. It joins dankclaw_default so its config's loki.write endpoint
# resolves the container name loki:3100.
# THE /etc/machine-id MOUNT IS LOAD-BEARING. sd_journal locates the journal
# directory by the local machine id; without it the container opens nothing and
# the journal source reads exactly 0 lines — silently, no error, health green.
# This mount is the entire reason the pipeline carries data. Never remove it.
#
# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own
# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x
# refuses the ui block), so the compose service had no reachable health
# endpoint at all: the homepage status card for alloy reported "down"
# continuously and there was nothing for a deploy to wait on. The command
# below therefore moves the UI to the container's wildcard address and the
# port is published on host loopback only, which puts alloy in the same class
# as the other three: reachable from the host, by Caddy over the private
# network, and by nothing off-host. Alloy has no authentication, which is
# exactly why it is not published wider.
#
# SOURCES. The host journald socket and the host log/container-storage trees
# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
# lives at /opt/alloy/config/config.alloy, installed from
# shared/alloy/config.alloy on every deploy.
#
# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
# client fails and its log pipeline backs up.
# IMAGE. Built rootless by build_compose_image alloy, then mirrored into the
# root image store by the deployer (a root build runs in the build container's
# own resolver and fails on DNS here). Pull=never: boot must not reach for the
# local tag at a registry.
[Unit]
Description=Alloy log shipper (Dankclaw)
After=network-online.target loki.service
Wants=network-online.target loki.service
After=network-online.target
[Container]
ContainerName=alloy
Image=localhost/franksdank/alloy:latest
Pull=never
Network=dankclaw_default
Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
PublishPort=127.0.0.1:12345:12345
Volume=/opt/alloy/config:/etc/alloy
Network=host
Exec=run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy
Volume=/opt/alloy/config:/etc/alloy:ro
Volume=/etc/machine-id:/etc/machine-id:ro
Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
Volume=/var/log:/var/log:ro
Volume=/var/lib/containers:/var/lib/containers:ro
HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy
Volume=/var/log/journal:/var/log/journal:ro
Volume=/run/log/journal:/run/log/journal:ro
HealthCmd=/usr/local/bin/busybox wget --spider -q http://127.0.0.1:12345/-/healthy
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
@@ -60,4 +56,4 @@ Restart=always
RestartSec=10
[Install]
WantedBy=default.target
WantedBy=multi-user.target
@@ -0,0 +1,56 @@
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
#
# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
# Unit name: cadvisor.service (container name: cadvisor)
#
# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
# now nothing reported per-container CPU/memory — Grafana could only see the
# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
# tree and exports container_cpu_* / container_memory_* per container.
#
# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
# subtree: cadvisor exported a single root series until this flag was added,
# then it walked the whole tree (996 per-container series measured 2026-10-20).
# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
# libpod/podman socket is needed to name containers.
#
# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
# The bridge PublishPort is what keeps the unauthenticated endpoint off the
# host's other interfaces: it publishes exactly loopback (host-local probes and
# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
# container, reaches a host-published port at its LAN address — the same netavark
# rule node-exporter relies on, only here the port really is published).
#
# It is rootless on purpose: dank owns the app containers' cgroups, so a
# rootless cadvisor reads every one of them; root would buy nothing here.
[Unit]
Description=cAdvisor container metrics (Dankclaw)
After=network-online.target
Wants=network-online.target
[Container]
ContainerName=cadvisor
Image=gcr.io/cadvisor/cadvisor:v0.49.1
PodmanArgs=--pid=host --cgroupns=host
PublishPort=127.0.0.1:8080:8080
PublishPort=192.168.0.143:8080:8080
Volume=/sys:/sys:ro
Volume=/:/rootfs:ro
Exec=--housekeeping_interval=30s
HealthCmd=wget --spider -q http://localhost:8080/healthz
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=40s
[Service]
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
# client_id query parameter — /application/o/<slug>/authorize/ does not exist.
Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
# ONLY when the token carries no `email` claim (authentik has no /emails path and
# 404s it, failing the login) — so the claim must actually be there: the fix is
# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
# <...>/userinfo/?api=1/emails, a 403.
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
# not a known scope here; requesting it makes the authorize view crash.
# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
# truncates SCOPES to its first token.
Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
# reaches this callback — the Anubis -> Authentik guard fronts every request.
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
@@ -48,13 +48,22 @@
# without `podman unshare` — accepted for a datastore nobody edits by hand;
# episode AUDIO is a host directory for the opposite reason (see the app unit).
#
# HEALTH. The base image is distroless: no shell, no curl, no wget, so no
# CMD-SHELL probe and no wget/curl probe is possible here (same class of unit
# as loki, whose probe had to move into the image). SurrealDB ships its own
# probe instead — `surreal is-ready` connects to ws://localhost:8000 by default
# and needs no credentials — verified against this digest: it prints `OK` and
# exits 0 while the server is up. Argument-only HealthCmd, like every unit on
# this host, because the quadlet generator splits it on spaces.
# HEALTH. The base image is distroless — measured against this digest, not even
# `ls` exists (`exec: "ls": executable file not found in $PATH`), so there is no
# `/bin/sh`, no curl and no wget. SurrealDB ships its own probe instead:
# `surreal is-ready` connects to ws://localhost:8000 by default, needs no
# credentials, prints `OK` and exits 0 while the server is up.
#
# It must be handed over as a QUADLET-SIDE JSON ARRAY, not a bare command.
# podman wraps a bare `HealthCmd` string as `CMD-SHELL` (measured on this host's
# podman 5.7: `--health-cmd "/surreal is-ready"` stores
# `{"Test":["CMD-SHELL","/surreal is-ready"]}`), and with no `/bin/sh` in the
# image that probe fails in ~30 ms with empty output on every interval while the
# database answers perfectly — which is exactly how this service's first deploy
# died on 2026-10-19 (`did not become healthy in 90s`). The quoted array survives
# the generator (`--health-cmd "[\"/surreal\",\"is-ready\"]"`) and stores
# `["CMD","/surreal","is-ready"]`, which execs directly and goes healthy. Do not
# "simplify" it back to a bare command.
[Unit]
Description=SurrealDB (open-notebook datastore)
@@ -70,7 +79,7 @@ EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env
EnvironmentFile=/home/dank/Containers/open-notebook/env
Exec=start --log info rocksdb:/mydata/mydatabase.db
Volume=open-notebook-db:/mydata
HealthCmd=/surreal is-ready
HealthCmd=["/surreal","is-ready"]
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
@@ -33,17 +33,21 @@
# and 5055 is exactly the port seerr already holds on this host
# (`PublishPort=127.0.0.1:5055:5055`; `ss -ltn` on Dankclaw 2026-10-18 shows
# 127.0.0.1:5055 taken). So the API leaves the container on host port 5056.
# That forces one setting: the frontend auto-detects the API as
# `<host-you-opened>:5055`, so API_URL in scripts/open-notebook.env must name
# the remapped port explicitly, or the UI loads and every call fails. Both
# ports are published on all interfaces because the consumer is a browser on
# the LAN — the fleet's browser-facing services (open-webui, comfyui, gitea)
# bind the same way, while loopback-only is reserved for services whose user
# path is the guard chain. This service has NO vhost and no Caddy route: it is
# internal-only until it is guarded, and its own password
# (OPEN_NOTEBOOK_PASSWORD) is what stands in front of it. That password is not
# decoration: with it unset the API answers everyone on the LAN, including the
# unauthenticated episode-audio download.
# Only 8502 is published on all interfaces, because it is the single upstream
# the guard chain dials: notebook.franksdank.cloud → anubis → authentik →
# http://:8017 → 192.168.0.143:8502 (dankclaw/Caddyfile), and this Caddy lives
# in root's storage, so a published port is the only way it can reach a rootless
# quadlet. The API needs no outward face at all — the image's own Next.js layer
# rewrites /api/* to the FastAPI beside it, so one origin serves UI and API —
# and the fleet's rule is that loopback-only is for services whose user path is
# the guard chain, which this became on 2026-10-19; 5056 stays published on
# loopback because the deployer's health gate dials it there. That is also why
# API_URL in scripts/open-notebook.env names the vhost rather than a port: left
# to itself the frontend auto-detects `<host-you-opened>:5055`, and the UI loads
# while every call fails. OPEN_NOTEBOOK_PASSWORD is not made redundant by the
# chain: it is what keeps the API honest for everything else on this host, and
# with it unset the API answers anyone, including the unauthenticated
# episode-audio download.
#
# VOLUMES. `/app/data` is a named volume: uploads, the LangGraph SQLite
# checkpoints and the extraction caches the image keeps under it
@@ -81,7 +85,7 @@ Pull=never
Network=dankclaw_default
Network=open-notebook.network
PublishPort=0.0.0.0:8502:8502
PublishPort=0.0.0.0:5056:5055
PublishPort=127.0.0.1:5056:5055
EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env
EnvironmentFile=/home/dank/Containers/open-notebook/env
Volume=open-notebook-data:/app/data
+1
View File
@@ -45,3 +45,4 @@
192.168.0.143 pihole.franksdank.cloud
192.168.0.143 seerr.franksdank.cloud
192.168.0.143 marreta.franksdank.cloud
192.168.0.143 notebook.franksdank.cloud
+26
View File
@@ -0,0 +1,26 @@
# /etc/udev/rules.d/98-halogen-npu-access.rules
#
# Why the NPU node is world-writable: `frank-dragoon` runs rootless podman under a
# unit with `User=dank`, and the container's user namespace maps exactly one uid and
# one gid (dank). `/dev/accel/accel0` is `root:render 0660`, so inside the container
# the node reads `nobody:nogroup` and the engine dies with its own diagnosis:
#
# halogen npu: /dev/accel/accel0 is here, but this process cannot open it
# (crw-rw---- nobody:nogroup).
#
# `--group-add keep-groups` is the vendor's advice and it does NOT work in this
# shape: measured 2026-10-19 from the host as a `dank` shell that was already in
# `render`, with and without the flag, the node still read `nobody:nogroup` and the
# open was refused — an unmapped gid cannot be granted to a process inside a
# namespace that does not map it. Same conclusion `99-comfyui-gpu-access.rules`
# reached for `/dev/kfd` from the systemd-user-spawn direction: the mode is the only
# spawn-context-independent door short of running the stack rootful.
#
# Blast radius: any process in this host's single uid domain (every rootless
# container here shares it) can submit to the XDNA2 NPU. The NPU holds no user data
# and no privileged context — the same trade already accepted for the GPU compute
# nodes beside it. Installed by `deploy-remote.sh` when the NPU lane is armed and left
# in place afterwards — the mode costs nothing while the lane is idle. `serve-frank-dragoon.sh`
# refuses to arm the lane on a node that is not other-accessible, so a host that lost this
# file fails loudly at boot instead of crash-looping.
KERNEL=="accel[0-9]*", SUBSYSTEM=="accel", MODE="0666"
+44 -6
View File
@@ -19,11 +19,14 @@
# is recorded in the parent tree, so cloning the parent commit recursively is
# reproducible without a second pin.
#
# THE BUILD LINE IS THE ONE PROVEN ON DANKCLAW, verbatim:
# THE BUILD LINE IS THE ONE PROVEN ON DANKCLAW, verbatim — with the Vulkan door
# open since 2026-10-19. That configure was run in this exact base image and
# answers `Found Vulkan: /usr/lib/x86_64-linux-gnu/libvulkan.so (found version
# "1.3.275") found components: glslc` then `Including Vulkan backend`:
#
# cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CCACHE=OFF \
# -DGGML_BACKEND_DL=OFF -DGGML_CPU_ALL_VARIANTS=OFF -DGGML_OPENMP=ON \
# -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=OFF \
# -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=ON \
# -DGGML_METAL=OFF -DGGML_SYCL=OFF -DGGML_RPC=OFF \
# && cmake --build build -j 8
#
@@ -42,8 +45,12 @@
# defaults it ON outside wasm/cross-compiles, i.e. -march=native for THIS
# host — correct while build host == run host, wrong if anyone ever builds
# this image elsewhere, which is why the fleet builds it on Dankclaw.
# CUDA/HIP/VULKAN/METAL/SYCL/RPC all OFF: this is a CPU endpoint (the trial
# ran on CPU; the box's GPUs belong to the text endpoints).
# CUDA/HIP/METAL/SYCL/RPC stay OFF, VULKAN is ON: the endpoint computes on the
# gfx1151 iGPU (decision and co-tenancy gates:
# report/tts-npu-spike-verdict-2026-10-19.md, shape in [[Frank TTS Engine]]).
# Nothing else is available to this image: no ROCm userland, no NVIDIA
# userland, no macOS, no oneAPI, and RPC would add a second process to
# supervise on a box that already has one engine per card.
#
# BUILD SHARED LIBS, RUNTIME SMALL. BUILD_SHARED_LIBS defaults ON on Linux, so
# tts-server links libggml*.so rather than swallowing them; the runtime stage
@@ -67,9 +74,28 @@ ARG BASE_IMAGE=docker.io/library/ubuntu:24.04@sha256:68ee22004a9109b44973313b02f
# ------------------------------------------------------------------ build
FROM ${BASE_IMAGE} AS build
# The three Vulkan build deps are not optional and not interchangeable:
# libvulkan-dev vulkan/vulkan.h + the VulkanHeaders/VulkanLoader CMake
# configs, i.e. what find_package(Vulkan) resolves to.
# glslc ggml compiles its shaders with glslc, not glslangValidator.
# Ubuntu's glslang-tools ships glslangValidator and NOT glslc,
# so without this package `find_package(Vulkan COMPONENTS
# glslc REQUIRED)` fails and there is no Vulkan backend.
# The archive's glslc is old (2023.8): it compiles
# GL_KHR_cooperative_matrix — the path ggml's quantized
# matmul needs — but reports cooperative_matrix2,
# integer_dot_product, bfloat16 and the micro-float
# extensions as unsupported, so those shader families are
# compiled out. If a Vulkan prefill ever reads as
# inexplicably slow, suspect this compiler before the device;
# the fix is a pinned glslang build passed as
# -DVulkan_GLSLC_EXECUTABLE=<path>, not a different runtime.
# spirv-headers find_package(SPIRV-Headers CONFIG REQUIRED); the distro
# package ships /usr/share/cmake/SPIRV-Headers/.
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \
ca-certificates cmake g++ git make \
libvulkan-dev glslc spirv-headers \
&& rm -rf /var/lib/apt/lists/*
# Checked out detached at a full SHA, then the recorded submodule. No branch
@@ -90,7 +116,7 @@ RUN cmake -B build -DCMAKE_BUILD_TYPE=Release \
-DGGML_BLAS=OFF \
-DGGML_CUDA=OFF \
-DGGML_HIP=OFF \
-DGGML_VULKAN=OFF \
-DGGML_VULKAN=ON \
-DGGML_METAL=OFF \
-DGGML_SYCL=OFF \
-DGGML_RPC=OFF \
@@ -102,9 +128,21 @@ FROM ${BASE_IMAGE}
# libgomp1: the OpenMP runtime the build above links against. curl: the unit's
# HealthCmd. ca-certificates: so that curl can speak https if anyone ever
# points a probe at a URL with a scheme this host terminates elsewhere.
#
# libvulkan1 + mesa-vulkan-drivers are what make `--device Vulkan0` mean
# anything at runtime: libvulkan1 is the loader libggml-vulkan.so dlopens, and
# mesa-vulkan-drivers is the RADV ICD that claims the gfx1151. Loader without an
# ICD enumerates ZERO devices, and tts-server then dies in device lookup AFTER
# both GGUFs have been read — the journal reads as a model fault, not a missing
# driver. Enumerated from this base on 2026-10-19 with `--device /dev/dri
# --group-add keep-groups` plus the host's ComfyUI udev rule: "Radeon 8060S
# Graphics (RADV GFX1151)", API 1.4.318 (the heap figure is owned by
# [[Frank TTS Engine]]; it is the GTT-backed aperture, not the ~2 GiB DRM
# carve-out). No NVIDIA ICD is installed on purpose: one Vulkan device in the
# enumeration is what makes the NAME Vulkan0 safe to write down.
RUN apt-get update -qq \
&& apt-get install -y -qq --no-install-recommends \
ca-certificates curl libgomp1 \
ca-certificates curl libgomp1 libvulkan1 mesa-vulkan-drivers \
&& rm -rf /var/lib/apt/lists/*
# Same unprivileged uid as the frank-tts façade (10000): the voice packs and the
@@ -40,12 +40,28 @@
# speaks to the server from inside its own namespace, which is where the address
# is valid.
#
# MEMORY / NO GPU. Peak RSS measured 5.7 GiB with both GGUFs resident
# (2026-10-18 trial); no memory limit is set because the box has the room and a
# limit would convert a busy moment into an OOM kill mid-utterance. No
# /dev/kfd, no /dev/dri, no keep-groups: every GPU backend is compiled out of
# the image, the endpoint is CPU-only by decision, and the GPUs on this host
# belong to the text endpoints.
# MEMORY / THE RENDER NODE. Host RSS is ~1.7 GiB, not the size of the GGUFs:
# the Vulkan backend copies both tensors into device memory at load, so they
# stop counting in the process's RSS (measured after the 2026-10-19 cutover —
# `podman stats` / `ps -o rss`). No memory limit is set: the box has the room
# and a limit would convert a busy moment into an OOM kill mid-utterance.
#
# `AddDevice=/dev/dri` is what makes the iGPU visible; without it the Vulkan
# ICD enumerates nothing and the server dies on device lookup after it has
# already read both GGUFs. `GroupAdd=keep-groups` is belt-and-braces, not the
# thing that works here: the units podman spawns carry no supplementary groups
# at all, and this host grants render access a different way — the udev rule
# that makes the render nodes world-writable (same rule ComfyUI rides, see
# dankclaw/comfyui). Measured: `id` inside this container lists only the
# container's own group, and `tts-server` still holds renderD129.
# The node is NOT named: this host is hybrid, renderD128 is the NVIDIA card and
# renderD129 the amdgpu iGPU, so a node number in the unit is a guess that a
# re-probe can invalidate, while the directory hands the container every render
# node and the driver picks by name.
#
# /dev/kfd stays absent on purpose: that is the ROCm compute interface, owned by
# the engines that use HIP. A Vulkan endpoint has no use for it, and handing a
# container a second way onto the GPU is surface for nothing.
#
# THE VOICE MOUNT IS READ-ONLY AND IS FOR THE WARMER, NOT THE ENGINE. A voice
# becomes known to the engine only through POST /v1/audio/voices, and the
@@ -98,6 +114,9 @@ Network=frank-tts-engine.network
EnvironmentFile=/home/dank/serverAppConfigs/scripts/frank-tts-engine.env
Volume=/home/dank/models/Qwen3-TTS-GGUF:/models:ro
Volume=/home/dank/voices:/voices:ro
# Render access for the iGPU (see MEMORY / THE RENDER NODE above).
AddDevice=/dev/dri
GroupAdd=keep-groups
HealthCmd=curl -fsS http://127.0.0.1:8880/health
HealthInterval=30s
HealthTimeout=10s
+54 -19
View File
@@ -7,7 +7,7 @@
# supervision, no retries, no logging of its own. PID 1 stays the server, so
# `podman stop` and the unit's Restart=always drive it directly.
#
# THE FLAGS THAT ARE NOT HERE, and must stay absent:
# THE FLAG THAT MUST STAY ABSENT:
#
# --threads / -t does not exist in this build. src/backend.h computes the
# CPU thread count as std::thread::hardware_concurrency()/2
@@ -18,13 +18,19 @@
# placement — pinning is deliberately not done anywhere in
# this stack. See the env file header for the same warning
# aimed at the operator.
# --device left at its default. Every GPU backend is compiled out of
# the image, so the default device pick can only land on
# CPU; naming a device would add a way to be wrong.
#
# Everything defaults to the shape the 2026-10-18 trial measured, so an empty
# environment still boots the decided configuration: 1.7 B talker Q4_K_M with
# the Q8_0 tokenizer, one request at a time, on the private bridge.
# --device IS here, but conditionally: it reaches the binary only when
# QWENTTS_DEVICE is non-empty, and empty is the CPU rollback — the binary then
# takes its own default device, which is CPU because the CPU backend is always
# built. One line in scripts/frank-tts-engine.env moves the endpoint between the
# two shapes, with no rebuild. What must never reach argv is a device NAME the
# loader does not enumerate: that is not refused by the argument parser, it dies
# during device lookup with the loader's own error, so the journal reads as a
# load that failed rather than a configuration refusal.
#
# Everything defaults to the shape this endpoint runs: 1.7 B talker Q4_K_M with
# the Q8_0 tokenizer, one request at a time, on the private bridge, on the
# gfx1151 iGPU. An empty environment still boots that shape.
set -euo pipefail
# The binary path. Overridable for one reason: the bats seam runs a stub
@@ -60,13 +66,33 @@ set -euo pipefail
: "${QWENTTS_LANG:=auto}"
# One request at a time. The server-side queue is a serialised FIFO at this
# setting, which is what keeps the measured per-voice timings true; batching
# on a CPU build splits the same cores between two syntheses and the warm RTF
# margin (0.55-0.67 against real time 1.0) disappears. The engine's own
# default is also 1; it is written here because it is a decision, not an
# accident.
# setting, which is what keeps the per-voice timings predictable. A second
# concurrent synthesis shares the device with the first — and on this box that
# device is also carrying frank-dragoon — so it buys no throughput and it
# doubles the wait for the caller behind it. The engine's own default is also 1;
# written here because it is a decision, not an accident.
: "${QWENTTS_MAX_BATCH:=1}"
# The compute device, NAMED and not numbered: `Vulkan0` is how ggml enumerates
# the gfx1151 iGPU on this host. An index would follow whatever order the
# loader's device list returns; a name says which device this endpoint means.
# The container sees exactly one Vulkan device because the image carries the
# Mesa ICD and no NVIDIA one, so the 3090 is not in the enumeration at all.
#
# EMPTY IS THE CPU ROLLBACK: a value that is set but EMPTY passes no `--device`,
# the binary picks its own default, and the CPU backend is always built — which
# is why moving this endpoint between the iGPU and the CPU is one line in
# scripts/frank-tts-engine.env and not a rebuild.
#
# That is also why the default below is NOT `${QWENTTS_DEVICE:=Vulkan0}`: `:=`
# fires on a set-but-empty variable as well as an unset one, so the rollback line
# `QWENTTS_DEVICE=` would be rewritten back to Vulkan0 and the endpoint would sit
# on the shared GPU with no way out short of a rebuild. Only UNSET takes the
# default.
if [[ ! -v QWENTTS_DEVICE ]]; then
QWENTTS_DEVICE=Vulkan0
fi
# Fail loudly BEFORE binding, so the journal says which artifact is missing
# instead of the unit reporting a container that starts and dies. This is the
# self-healing path: Restart=always re-runs it every RestartSec, and the same
@@ -79,11 +105,20 @@ for artifact in "$QWENTTS_MODEL" "$QWENTTS_CODEC"; do
fi
done
exec "$QWENTTS_BIN" \
--model "$QWENTTS_MODEL" \
--codec "$QWENTTS_CODEC" \
--alias "$QWENTTS_ALIAS" \
--host "$QWENTTS_HOST" \
--port "$QWENTTS_PORT" \
--lang "$QWENTTS_LANG" \
ARGS=(
--model "$QWENTTS_MODEL"
--codec "$QWENTTS_CODEC"
--alias "$QWENTTS_ALIAS"
--host "$QWENTTS_HOST"
--port "$QWENTTS_PORT"
--lang "$QWENTTS_LANG"
--max-batch "$QWENTTS_MAX_BATCH"
)
# Only when a device is named. An empty QWENTTS_DEVICE has to produce NO flag,
# because "the rollback shape" is argv identical to the pre-iGPU endpoint.
if [[ -n "$QWENTTS_DEVICE" ]]; then
ARGS+=(--device "$QWENTTS_DEVICE")
fi
exec "$QWENTTS_BIN" "${ARGS[@]}"
@@ -0,0 +1,39 @@
# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
#
# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
# Unit name: cadvisor.service (container name: cadvisor)
#
# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
# Same rootless shape, same --cgroupns=host requirement (see that file for the
# full story), only the LAN address differs: this host publishes on
# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
# network (cross-host, so the host.containers.internal trick does not apply).
# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
# tautulli) by their quadlet unit names via the cgroup path.
[Unit]
Description=cAdvisor container metrics (Dankserver)
After=network-online.target
Wants=network-online.target
[Container]
ContainerName=cadvisor
Image=gcr.io/cadvisor/cadvisor:v0.49.1
PodmanArgs=--pid=host --cgroupns=host
PublishPort=127.0.0.1:8080:8080
PublishPort=192.168.0.235:8080:8080
Volume=/sys:/sys:ro
Volume=/:/rootfs:ro
Exec=--housekeeping_interval=30s
HealthCmd=wget --spider -q http://localhost:8080/healthz
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=40s
[Service]
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
+1 -1
View File
@@ -4,7 +4,7 @@ After=network-online.target
Wants=network-online.target
[Container]
Image=lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325
Image=lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329
ContainerName=plex
Network=dankserver_appnet:alias=plex
Environment=PUID=0
@@ -0,0 +1,319 @@
# XDNA2 NPU inference runtimes on Linux (Strix Halo) — can any of them serve TTS?
Research slice, 2026-10-19. Worker: `NpuRuntimeLandscape`. Research only, no code.
Question: every way to execute model inference on an AMD XDNA2 NPU on Linux as of
2026-10, and whether any of them can serve a speech-synthesis model.
> **Status, same night — the spike that followed this slice settled it.** NO-GO on the
> NPU (the loader declines the graph, not the silicon); the actionable door is the gfx1151
> iGPU behind `-DGGML_VULKAN=ON`. Measured contract, exoneration chain and host gates:
> `tts-npu-spike-verdict-2026-10-19.md`. That page supersedes §8–§9 below.
Convention: vendor claims carry URL + the version/date they hold for.
**[host]** = Dankclaw state verified on the live machine 2026-10-19 (Main), not a
vendor claim. `[INFERENCE]` = my deduction from the cited artifacts.
## 0. Answers to the three questions Main ranked
**(1) A runtime that already runs TTS on XDNA2 today: it exists at AMD, not on our
host.** Ryzen AI Software **1.8.0** (docs `latest` == 1.8.0; released 2026-07-23)
lists `neutts-air` and `neutts-nano` as model type **TTS**, variant
*full-optimized* (NPU-only) — <https://ryzenai.docs.amd.com/en/latest/llm_list.html>;
release note "New TTS model support: neutts-air, neutts-nano" —
<https://ryzenai.docs.amd.com/en/latest/relnotes.html>. Pre-compiled artifacts are
published: `amd/neutts-air_rai_1.8.0_npu_4K` (arch `qwen2`, lastModified
2026-07-23, Apache-2.0, "AWQ / Group 128 / Asymmetric / BFP16 activations / UINT4
Weights") and `amd/neutts-nano_rai_1.8.0_npu_4K` (arch `llama`).
**Neither artifact is a TTS engine.** Both file lists are LM-only (`model.onnx`,
`genai_config.json`, BPE tokenizer, AIE-ML compiler metastate: `dd_metastate_*`
`.ctrlpkt/.super/.state`, `cache/txn_bins.zip`) with **no codec, no vocoder, no
reference-audio encoder**. [INFERENCE from HF file lists]
**(2) FastFlowLM: no TTS, and we cannot add one** — see §2. Halogen: no TTS entry
**[host]**. So "something installed already serves TTS on the NPU" is FALSE.
**(3) Our own ONNX graph compiled for the NPU on Linux:** possible for the plain
ONNX flows (they compile on Linux), **impossible for a new LLM/OGA model** — model
generation is Windows-only in 1.8.0 — and gated by an operator table where BF16
lacks Softmax/LayerNorm/ReLU/Gelu (see §4).
## 1. Landscape: every way to reach the XDNA2 array on Linux
| # | Path | License | Strix Halo + Linux, 2026-10 | Speech synthesis |
|---|------|---------|------------------------------|------------------|
| 1 | Ryzen AI SW → ONNX Runtime + Vitis AI / Ryzen AI EP; compile INT8 (`X2`) or BF16 (`VAIML`) ONNX at session creation | AMD proprietary (tarball) | supported: our device ID/rev is in AMD's STX table; compile+run works on Linux (§3, §4) | user-authored ONNX only; no TTS recipe/demo |
| 2 | Same EP driven by **OGA** (`onnxruntime-genai`), "full fusion" NPU-only LLM flow; runner = AMD `model_benchmark` / OGA C++ API | AMD + MS OGA | Linux: **vendor-precompiled models only**, no hybrid flow | **neutts-air / neutts-nano = TTS** (LM half) |
| 3 | **FastFlowLM** `flm` | MIT + free proprietary kernels | Strix Halo "Supported"; Ubuntu 24.04/25.10/**26.04**, Arch | none (ASR input only) |
| 4 | **IRON / mlir-aie** + Peano → hand-written AIE-ML design → XRT/`amdxdna` | Apache-2.0 + LLVM exception | explicitly documented "AMD Ryzen™ AI on Linux"; same driver/XRT stack | anything, hand-written; nothing ships TTS |
| 5 | Vendor-curated AIE-ML graph sets (halogen `HALOGEN_NPU_MODELS`, FLM packs) | vendor | present **[host]** | none **[host]** |
| 6 | Windows-only: DirectML, Ryzen AI Windows EP (`ryzenai-llm`), Studio Effects, `whispercpp` NPU | — | not usable | (per Lemonade matrix: Windows only) |
| 7 | ggml/llama.cpp/**qwentts.cpp**, whisper.cpp, sherpa-onnx, Piper | — | no XDNA backend in ggml/ORT-mainline | CPU / iGPU only |
| 8 | IREE, TVM, ONNX Runtime mainline EPs (OpenVINO = Intel NPU, MIGraphX/ROCm = AMD GPU) | — | no XDNA2 target; AMD's NPU EP is only "Xilinx Vitis-AI (preview)" in the ORT EP list and ships inside AMD's own build | n/a |
Sources: §2–§6 below.
## 2. FastFlowLM (`/usr/bin/flm` 1.0.1 **[host]**) — catalog + HTTP surface
* Repo moved into AMD's ROCm org **2026-08-11** (`github.com/ROCm/FastFlowLM`,
v1.0.0). "Supports all Ryzen™ AI Series chips with XDNA2 NPUs (Strix, Strix Halo,
Kraken, and Gorgon Point)."
<https://github.com/ROCm/FastFlowLM> (README fetched 2026-10-19).
* Linux support landed 2026-03; distros: Ubuntu 24.04, 25.10, **26.04**, Arch,
generic. Needs NPU firmware ≥ 1.1.0.0, kernel **7.0+** with `amdxdna` (or
`amdxdna-dkms`), XRT stack (`ppa:lemonade-team/stable` → `libxrt-npu2`,
`amdxdna-dkms`); `flm validate` reads the DRM path (`/dev/accel/accel0`), `flm
run` goes through **XRT** and fails with `No such device with index '0'` unless
`xrt-smi examine` lists the NPU; `memlock` must be raised.
<https://fastflowlm.com/docs/install_lin/> (page dated 2026-03-05, fetched
2026-10-19). Static build bundles **XRT v2.21.75** — the same XRT series as the
distro package on the host **[host]**.
* HTTP surface: `flm serve <model>` → OpenAI-compatible REST, **port 52625**, no
auth. Maintained endpoints: `v1/models`, `v1/chat/completions`,
`v1/audio/transcriptions`, `v1/embeddings`. **There is no
`/v1/audio/speech`.**
<https://fastflowlm.com/docs/instructions/server/openapi/>
* Catalog 2026-10-19 (<https://fastflowlm.com/docs/models/>): LLaMA, DeepSeek,
Qwen, Gemma, MedGemma, TranslateGemma, gpt-oss, LiquidAI/LFM, Phi, Nanbeige,
**Whisper — card states "Type: Speech-to-Text (ASR)"**
(<https://fastflowlm.com/docs/models/whisper/>), EmbeddingGemma, SmolVLA,
Hy-MT2, MiniCPM. No speech-synthesis entry.
* Engines are **prebuilt, closed, downloaded from Hugging Face per architecture**
("Internet access to HuggingFace is required to download the optimized model
kernels"). The only in-tree engine build is Gemma-4E
(`cmake --preset linux-default -DFLM_BUILD_GEMMA4E=ON`).
→ **A new TTS architecture cannot be added to `flm` by us.**
## 3. AMD Ryzen AI Software 1.8.0 on Linux — install source, device support
* **Install channel (important correction to the pypi.amd.com evidence).** The pip
index is Windows-only for the EP runtime (`ryzenai_dynamic_dispatch-1.7.0-…
win_amd64.whl`, `onnxruntime_vitisai-1.23.2-…win_amd64.whl`; `flexmlrt` has a
single **linux_aarch64** wheel) — but AMD's Linux stack is not on pip. It ships
as a tarball that is, as of 2026-10-19, **publicly reachable without an AMD
account**: `https://download.amd.com/opendownload/RyzenAI/1.8.0/ryzen_ai-1.8.0.tgz`
→ HTTP 206 on a range probe, `Content-Range: bytes 0-8/**7819506855**`
(**≈7.8 GiB**, magic `1f 8b 08` = gzip). (The docs link it behind the
EULA form `account.amd.com/...ryzenai-eula-public-xef.html`; the CDN path
serves it directly. Verified by me with curl 2026-10-19. I did not finish
downloading 7.8 GiB, so the tarball's *contents* list is unverified —
`[INFERENCE]` only where noted.)
* Driver/XRT package (public, 90,016,137 bytes, `application/zip`):
`https://download.amd.com/opendownload/RyzenAI/Driver/RAI_1.8_Linux_NPU_XRT.zip`
contains `xrt_202620.2.25.37_24.04-amd64-{base,base-dev,npu}.deb` +
`xrt_plugin.2.25.260102.56.release_24.04-amd64-amdxdna.deb` (verified by
download + unpack). **The plugin deb ships firmware for
`/usr/lib/firmware/amdnpu/17f0_11/npu.dev.sbin`** — exactly the directory our
NPU loads **[host]** — plus `1502_00`, `17f0_10`, `17f1_10`, `17f1_13`,
`libvxdna.so`, `libxrt_driver_xdna.so.2.25.260102.56`, and the DKMS driver
source tarball. **No xclbin/PDI and no compiler binary in this package**: it is
driver + XRT shim + firmware only.
* **Our device is officially STX.** AMD's APU table
(<https://ryzenai.docs.amd.com/en/latest/app_development.md#apu-types>):
`0x1022:0x17F0 rev 0x11 → STX` (0x00/0x10 also STX, 0x20 = KRK, `0x1502` rev 0
= PHX/HPT). Release notes: "Strix and **Strix Halo** as STX", and the STX/KRK
column enables CNN INT8, CNN BF16, NLP BF16, LLM (OGA).
* Vendor driver source (same deb, `share/amdxdna/amdxdna.tar.gz`, source dated
2026-06-29) maps `0x17f0 rev 0x11 → dev_npu5_info`, `.fw_path =
"amdnpu/17f0_11/"`, `default_vbnv = "RyzenAI-npu5"`, and the revision→name table
has `{ AIE2_DEV_REVISION_HALO, "NPU Strix Halo" }`; `xrt-smi examine` therefore
reports the device as **NPU Strix Halo**.
* Linux install recipe (1.8.0 docs
<https://ryzenai.docs.amd.com/en/latest/linux.html>): Ubuntu 24.04 packages,
**Python 3.12** (`apt install python3.12 python3.12-venv`),
`libboost-filesystem1.74.0`, dkms; then `install_ryzen_ai.sh -a yes -p <venv>`;
`LD_LIBRARY_PATH=…:${RYZEN_AI_INSTALLATION_PATH}/onnxruntime/lib/`;
`quicktest.py` "picks up a simple CNN model, **compiles it** and runs on AMD's
Neural Processing Unit" — i.e. **compilation runs on Linux** for the non-LLM
flows. "Linux currently supports NPU only flow" (no iGPU EP).
Host friction: our box is Ubuntu 26.04 + **Python 3.14** **[host]**, so the venv
must be built with a 3.12 interpreter and the packages are 24.04-built
`[INFERENCE]`.
* LLM flow on Linux
(<https://ryzenai.docs.amd.com/en/latest/llm_linux.html>): prequantized
ready-to-run models from HF, `deployment/lib`,
`RYZENAI_EP_PATH=$PWD/deployment/lib/libonnxruntime_providers_ryzenai.so`,
run via the `model_benchmark` binary; **"Linux does not support Hybrid flow"**;
**"Model generation is not supported in current release"** (repeated in
relnotes 1.8 Known Issues: "Models generated on Windows are compatible with
Linux"); needs `memlock unlimited`.
Sample throughput in that doc: **49.1 tokens/s decode**, 753 tokens/s prefill
(Phi-3.5-mini, 4K model).
* Third-party confirmation of the OS split: Lemonade's backend matrix
(<https://raw.githubusercontent.com/lemonade-sdk/lemonade/main/README.md>,
fetched 2026-10-19) — `flm`/npu/XDNA2 = **Windows, Linux**;
`ryzenai-llm`/npu = **Windows only**; `whispercpp`/npu (STT) = **Windows only**;
text-to-speech engines `kokoro` and `acestep` = metal / cuda / vulkan / rocm,
**no npu device at all**. Lemonade is the "Server Interface" layer of AMD's own
Ryzen AI stack (docs index), so this matrix is the vendor-sanctioned view.
## 4. Can we compile our own graph on Linux? (the no-recipe path)
Mechanics from <https://ryzenai.docs.amd.com/en/latest/modelrun.md> (1.8.0):
ONNX only, opset 17 recommended; the EP compiles on first `InferenceSession` and
caches (VitisAI EP cache, or ORT **EP context cache** — `ep.context_enable`, and
"precompile the EP context model using Python and deploy it using a C++ program"
is explicitly supported); EP auto-partitions unsupported nodes to CPU;
BF16 = `target VAIML` (`preferred_data_storage: unvectorized` for transformers),
BF16 **STX/KRK-or-newer only**; INT8 = `target=X2` with XINT8/A8W8/A16W8 and
"general asymmetric quantization … to enable third-party quantized models";
**deployment EP cannot compile BF16 on the fly** — ship a pre-compiled model
(<https://ryzenai.docs.amd.com/en/latest/app_development.md>);
cache dirs are not portable across EP or NPU-driver versions.
Operator coverage is the hard gate
(<https://ryzenai.docs.amd.com/en/latest/ops_support.md>, "operators supported in
Ryzen AI 1.8.0"; "Y" = broad coverage, some configurations may still fail):
* Good for TTS front-ends/vocoders: `ConvTranspose` **Y in BF16/A16W8/A8W8**
(absent only from XINT8), `Conv`, `Resize`, `Upsample`, `Pad`, `LSTM` (BF16),
`CumSum` (BF16/XINT8), `STFT` (XINT8 only).
* **Blockers for a decoder-style transformer in the plain flows:** in **BF16** the
table has **no Softmax, no LayerNormalization, no Relu/Gelu**; LayerNorm is
XINT8-only and Softmax is A16W8/A8W8/XINT8-only. So attention either lives in
the **LLM/OGA flow** (whose op set is exactly
`SimplifiedLayerNormalization`, `SkipSimplifiedLayerNormalization`, `MatMulNBits`
(W4ABF16/W4ABFP16), `RotaryEmbedding`, **`GroupQueryAttention`**, `Sigmoid`,
`Mul`, `Add`) or must be decomposed into a quantized INT8 flow.
* Missing entirely: `GRU`, `Trilu`, `OneHot`, `EyeLike`, `Multinomial`,
`NonMaxSuppression`, `CastLike`, `MelWeightMatrix`/`MelFilterGroup`,
`RMSNormalization` → graphs that emit them must be rewritten before they
partition onto the NPU. [INFERENCE from absence in the table.]
* Quantization is a separate tool, not in the driver package: **AMD Quark**
(`quark.torch` / `quarkonnx`), which documents `ConvTranspose2d`, `conv1d`,
softmax/LayerNorm/BFP quantization (relnotes 1.3/1.2/1.1/0.9) — Quark is *not*
on pypi.org under the obvious names **[host probe]** and ships from AMD's
gated/pip-private channel.
* **New LLM-shaped model (our Qwen-Talker analogue): Windows-only today.**
OGA model generation is not available on Linux in 1.8.0 (§3). So a *new*
speech-LM cannot be self-served onto the NPU on Linux without AMD (or a
Windows box) in the loop.
* Practical shape cost: the neutts artifacts carry **one compiled variant per
padded sequence length** (128/256/512/768/1024/1536/2048/2304/3072/4096 for
prefill; 256…4096 for token-phase attention mask) — i.e. per-shape compile
artifacts are the norm for the LLM flow. [INFERENCE from the HF file list.]
## 5. Does the AMD TTS artifact actually produce speech? (cloned-voice survival)
NeuTTS is **LM + separate neural codec** ("Simple LM + codec architecture"),
clone from ~3 s of reference audio, and the codec is **NeuCodec (50 Hz, single
codebook)**: `neuphonic/neucodec` (PyTorch, encode+decode),
`neucodec-onnx-decoder{,-int8}` (**decode only**), "Decoder-only (ONNX) codecs
require **pre-encoded references** — these ship with the repo for all bundled
voices." Phoneme input; 2048-token context ≈ 30 s of audio; NeuTTS-Air ~360 M
active / 552 M with embeddings; upstream benchmarks are **CPU-only llama.cpp**
(119 tok/s prefill/decode mix on a Ryzen 9 HX 370) and explicitly exclude the
codec. <https://raw.githubusercontent.com/neuphonic/neutts/main/README.md>
(fetched 2026-10-19); `neuphonic/neutts-air` HF metadata (lastModified
2026-08-26, `qwen2`, GGUF + safetensors).
Consequences for the fleet spike:
1. AMD's NPU artifact = LM only; **waveform decode and reference encoding are CPU
work either way**. Our `reference.wav` voices survive only if we run the
NeuCodec **encoder** (PyTorch, per the codec table there is no ONNX encoder)
once per voice and then feed pre-encoded codes. That is a one-time offline
step per voice; after that the compiled graph takes codes/prompt prefix as
ordinary inputs, so nothing about the compiled graph forbids a caller-supplied
speaker prefix. `[INFERENCE]`
2. Real-time math: 50 Hz codec means **≥50 tokens/s decode** for 1× RT. AMD's own
Linux sample output for a 4K Phi-3.5-mini is **49.1 tok/s** — so budget for
roughly RT-1× on one 4K "full fusion" graph, before the codec's CPU cost.
`[INFERENCE — different model; must be measured.]`
3. Host RAM: the same doc reports **peak working set 3.54 GB** for that 4K model.
NeuTTS-Air is ~552 M params at W4A-BFP16, so expect a smaller footprint than
3.5 GB `[INFERENCE]`; still non-trivial against 14 GiB available **[host]**.
4. AMD ships **no TTS demo, runner, or server** on any OS: the examples page has
only CNN/BF16/DistilBERT/super-resolution/OGA-LLM/VLM/RAG
(<https://ryzenai.docs.amd.com/en/latest/examples.md>); the RyzenAI-SW repo
tree (669 paths, `main`, fetched 2026-10-19) contains speech demos **only under
`Demos/ASR/{Whisper,Parakeet-TDT}` and `Transformer-examples/ASR/Whisper-AI`**
— no TTS directory. Serving TTS therefore means writing our own OGA host
(or driving `model_benchmark`-style C++ code) plus phonemizer + NeuCodec glue.
## 6. Two graphs on one NPU, and the iGPU hazard
* **Concurrency is a first-class driver feature.** The vendor `amdxdna` source
(in `xrt_plugin.2.25.260102.56`, 2026-06-29) sets `.hwctx_limit = 16` for
npu4/npu5 — i.e. **16 hardware contexts** on our device — and allocates each
context a set of AIE columns with QoS `opc = col_opc (4096) × num_col`
(`aie2_ctx.c`, `aie2_solver.h`), with `frame_boundary_preempt` /
`fill_preempt_slot` in `aie2_pci.h` (firmware preemption at frame boundaries).
So halogen's armed models and a TTS graph can hold separate contexts
simultaneously; they contend for columns and OPC share, they are not mutually
exclusive.
* AMD's user-visible history agrees: 1.4 "Support for up to **16 hardware
contexts** on Ryzen AI 300 series"; 1.5 "Application concurrency: improves the
resource distribution across applications"; 1.3 "Idle detection and suspension
of contexts", "Rebalance feature for AIE hardware resource optimization";
1.0 known issue "Support for multiple applications is limited to up to eight"
(all in <https://ryzenai.docs.amd.com/en/latest/relnotes.html>).
* **Package conflict, the real risk** [INFERENCE]: the vendor stack wants AMD's
XRT 2.25.37 + `xrt_plugin…amdxdna` deb **[host: distro `libxrt-npu2
1:2.21.75+dfsg-4` installed]**, and firmware must match the loaded `amdxdna.ko`
— the vendor README warns that a stale/mismatched `npu.sbin` causes
`ERT_CMD_ABORT`/mailbox timeouts, and that a distro **in-tree** `amdxdna` older
than the XRT SHIM makes `GET_ARRAY`/`QUERY_TELEMETRY` return
`EOPNOTSUPP`/`EINVAL` (<https://github.com/amd/xdna-driver> README,
fetched 2026-10-19). Installing AMD's debs over the distro XRT changes the ABI
under FastFlowLM/halogen, so the spike must decide the XRT version **once**, for
the whole host.
* **iGPU / fabric clock:** no vendor document supports a fabric-clock pin, and no
vendor document addresses NPU+iGPU concurrency hangs. AMD's Strix Halo
optimization doc (ROCm docs, dated 2026-02-20, and its successor "AMD RDNA3.5
system optimization", ROCm 10.1.0) covers GPUVM/GART/GTT, the TTM `pages_limit`
knob, `amd-ttm`, BIOS VRAM sizing, and minimum kernel versions
(<https://rocm.docs.amd.com/en/docs-7.2.0/how-to/system-optimization/strixhalo.html>,
<https://rocm.docs.amd.com/en/latest/reference/system-optimization/rdna3-5.html>).
The `pp_dpm_fclk` level-7 pin and the GPU+NPU hang observation stay
fleet-observed **[host]**, i.e. `[INFERENCE]` territory — no citation exists.
## 7. Paths I checked and ruled out (so nobody re-walks them)
* **Windows-only doors**: DirectML; Ryzen AI Windows EP; Lemonade's
`ryzenai-llm` and `whispercpp`-NPU engines (matrix above); AMD's OGA model
generation. Not usable on Dankclaw.
* **Mainline ONNX Runtime** has no AMD-NPU EP: its EP summary lists AMD ROCm and
AMD MIGraphX (GPU) and "Xilinx Vitis-AI (*preview*)" only
(<https://onnxruntime.ai/docs/execution-providers/>, 2026-10-19); the working
EP is AMD's own build (`libonnxruntime_providers_ryzenai.so` in the tarball's
deployment dir, per llm_linux).
* **ggml / llama.cpp / qwentts.cpp**: no XDNA backend → the current engine
(`tts-server` on GGUF, CPU-only build **[host]**) cannot be "turned on" for the
NPU; migration means a different runtime, not a flag.
* **`flm`**: catalog/engines are closed per-architecture (§2) — the cheapest door
by install cost, and it is not a door for TTS.
* **IRON / mlir-aie** is the only open toolchain that documents running on the
Ryzen AI NPU from Linux ("A close-to-metal Python API for programming AMD Ryzen™
AI NPUs, built on an open-source MLIR-based compiler toolchain"; "Getting
Started for AMD Ryzen™ AI on Linux", Ubuntu 24.04/24.10, XRT from
`ppa:amd-team/xrt`: `libxrt2 libxrt-npu2 … amdxdna-dkms`, `render` group
required) — <https://raw.githubusercontent.com/Xilinx/mlir-aie/main/README.md>,
fetched 2026-10-19. This is how FLM's kernels were made ("Low-level kernels
optimized using IRON + AIE-MLIR", FLM README) — i.e. real, but it is
hand-written per-op tile programming, not a model compiler: writing a TTS model
here is a multi-month engineering project, not a migration.
* **Embedded** Ryzen AI P100 support in the same Linux doc is irrelevant to STX.
## 8. Recommendation implied by the above (for Main's spike)
Spike the AMD path as decided, with these constraints planned in: Python 3.12 venv
+ 24.04-built wheels on Ubuntu 26.04; one host-wide XRT decision (AMD 2.25.37 vs
distro 2.21.75) with firmware/driver pairing checked via `xrt-smi examine` →
"NPU Strix Halo"; 7.8 GiB tarball staged off the public CDN URL (§3); pre-encode
each `reference.wav` with the **PyTorch** NeuCodec encoder because the ONNX codec
is decoder-only; keep phonemizer + codec on CPU; measure decode tokens/s against
the 50 Hz budget and peak RSS against the ~14 GiB headroom; expect to write the
serving shell yourself (AMD ships `model_benchmark`, no TTS demo).
## 9. Loose ends (not verified; I stopped on Main's stand-down)
* Contents of `ryzen_ai-1.8.0.tgz` (7.8 GiB) — whether the Linux tarball contains
the AIE-ML compiler (`vaiml`, `aiecompiler_client`) for the *BF16/INT8* flows
and which Python wheels it carries. My statement that non-LLM compilation works
on Linux rests on AMD's own Linux doc ("quicktest.py … compiles it and runs on
… NPU"), not on the package contents. Download was killed mid-flight.
* Whether AMD's `neutts` artifacts run correctly under the Linux `model_benchmark`
runner (the TTS rows are listed without OS qualifiers; no Linux TTS run is
documented anywhere in 1.8.0 docs).
* Exact AIE column count/total `ncols` exposed for Strix Halo (driver reads it
from AIE metadata: `ndev->total_col = min(XRS_MAX_COL, ndev->aie.metadata.cols)`).
@@ -0,0 +1,438 @@
# Qwen3-TTS payload + qwentts.cpp runtime: architecture and portability
Research deliverable (no code). Slice: *what the fleet's TTS payload actually is, and how
portable it is off the CPU ggml path.* Companion slice: `NpuRuntimeLandscape` (what NPU
runtimes exist on Dankclaw).
Evidence date: 2026-10-19. Everything below is grounded in:
- `github.com/ServeurpersoCom/qwentts.cpp` @ `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`
(2026-10-16, upstream HEAD on 2026-10-19) and its ggml submodule
`ServeurpersoCom/ggml` @ `cc5df49a6b878db2b68b9a7b45976e279594627d` (2026-10-16).
- The two GGUF files as they sit on Dankclaw (`/home/dank/models/Qwen3-TTS-GGUF/`), read
byte-for-byte from the live host.
- Upstream Qwen3-TTS model cards / release material.
Status: COMPLETE for the assigned slice. ANSWER C was confirmed on the live host the same
night — the running `frank-tts-engine` container carries `libggml-base.so.0.27.0` +
`libggml-cpu.so.0.27.0` and nothing else, and our own
`dankclaw/voiceservice/Containerfile.engine` writes `-DGGML_VULKAN=OFF` out explicitly, so
the iGPU change is a flag in a file the fleet owns. Verdict, measured contract and the NPU
exoneration chain: `tts-npu-spike-verdict-2026-10-19.md`.
---
## 1. What the payload is
Two files, both read byte-for-byte off the live host (`/home/dank/models/Qwen3-TTS-GGUF/`,
owned `dank:dank`, mtimes 2026-10-18 18:31), both from
`https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF` (community converter, not Qwen):
| file | bytes | GGUF | tensors | arch string | file_type |
|---|---|---|---|---|---|
| `qwen-talker-1.7b-base-Q4_K_M.gguf` | 1,219,245,248 | v3, kv=64 | 480 | `qwen3-tts` | `Q4_K_M` |
| `qwen-tokenizer-12hz-Q8_0.gguf` | 291,150,624 | v3, kv=51 | 398 | `qwen3-tts-tokenizer` | `Q8_0` |
`general.name` of the talker file is `Qwen3-TTS-12Hz-1.7B-base`; the header sizes add up
to the file sizes (sum of tensor bytes + header + alignment padding), so nothing in these
files is unaccounted for.
### 1.1 Talker file (480 tensors)
Three models in one container, per `docs/ARCHITECTURE.md` §GGUF layout
(qwentts.cpp @ pinned commit) and confirmed by the tensor census:
| prefix | tensors | bytes | what it is |
|---|---|---|---|
| `talker.*` | 316 | 1,061.9 MB | 28-layer Qwen3 decoder (hidden 2048, FFN 6144, 16 Q heads / 8 KV heads, head_dim 128, RoPE θ=1e6, RMS ε=1e-6, ctx 32768), text stream + codec stream summed, `codec_head` → 3072 codebook-0 logits |
| `code_pred.*` | 88 | 114.8 MB | 5-layer Qwen3 MTP head (hidden 1024, FFN 3072, 15 acoustic codebooks × 2048), own KV cache |
| `spk_enc.*` | 76 | 36.6 MB | ECAPA-TDNN speaker encoder (x-vector, base checkpoints only; mel → conv0 → 3 SE-Res2Net → MFA → ASP → 2048-d vector) |
Dtype split (exact tensor byte sums): `q4_K` 250 tensors / 864.1 MB, `q6_K` 17 / 312.0 MB,
`f32` 176 / 25.8 MB, `f16` 37 / 11.4 MB.
Which tensors carry which quant is the load-bearing portability fact:
- `q6_K` (17): `talker.text_embd.weight [2048, 151936]` (255.3 MB), `talker.codec_embd.weight
[2048, 3072]`, and the 15 `code_pred.codec_embd.<g>.weight [2048, 2048]` embedding tables.
These are the llama.cpp `Q4_K_M` "embedding/output rows go to Q6_K" rule, applied to
embedding tables rather than to an `lm_head`.
- `q4_K` (250): every attention/FFN projection of the talker and of the code predictor.
Distinct matmul shapes: `[2048,2048]`×73, `[2048,1024]`×62, `[2048,6144]`×56,
`[6144,2048]`×28, `[1024,2048]`×20, `[1024,3072]`×10, `[1024,1024]`×10, `[3072,1024]`×5,
`[2048,3072]`×2, `[2048,151936]`×1. Every one is 2-D and ≥ 1024 on both axes — good
GEMM shapes, nothing exotic.
- `f32` (176) / `f16` (37): all of `spk_enc` (conv weights F16 because K=1/3/5 rows are not
block-alignable, per the converter's `should_quantize` policy in `tools/quantize.cpp` +
`quantize.sh`; every bias/norm/LayerScale vector F32), plus the talker/predictor norms and
the `text_proj` biases.
Metadata the runtime actually reads (talker file): `qwen3-tts.num_code_groups=16`,
`talker.vocab_size=3072`, `text_vocab_size=151936`, `text_hidden_size=2048`,
`position_id_per_seconds=13`, `rope.mrope_section=[24,20,20]`, `mrope_interleaved=1`,
the 7 codec special ids (pad 2148 / bos 2149 / eos 2150 / think 2154 / nothink 2155 /
think_bos 2156 / think_eos 2157), `language_names` (10: chinese, english, german, italian,
portuguese, spanish, japanese, korean, french, russian) with their codec ids, the text-side
special ids (im_start 151644, im_end 151645, tts_pad 151671, tts_bos 151672, tts_eos 151673),
the whole `generation_config.json` (do_sample, top_k 50, top_p 1.0, temperature 0.9,
repetition_penalty 1.05, `subtalker_*` mirrors, max_new_tokens 8192), and an embedded
**Qwen2 BPE tokenizer** (`tokenizer.ggml.model=gpt2`, 151,676 tokens, 151,291 merges,
eos 151643) — that BPE block is 5.95 MB of the header, i.e. 49 % of the header is vocabulary.
Two live-file details that contradict the upstream doc text (worth knowing when reading
`docs/ARCHITECTURE.md`: the doc example block is stale against the shipped file):
`mrope_interleaved` is **1** (doc shows `false`), and `spk_enc.sample_rate` is **24000**
(doc says 16000). Doc also says "404 tensors on the 1.7B voice_design build" — the base
build shipped here has 480 because it carries `spk_enc.*`.
### 1.2 Tokenizer/codec file (398 tensors)
One neural audio codec, 12.5 frames/s over 24 kHz mono, hop 1920, 16 codebooks × 2048
entries (1 semantic + 15 acoustic). Dtype split: `f16` 50 / 143.6 MB, `q8_0` 110 / 76.9 MB,
`f32` 238 / 70.7 MB. `tok_dec.*` 253 tensors / 201.3 MB, `tok_enc.*` 145 / 89.9 MB.
Composition (metadata + tensor census, both files agree):
- **Encoder** (used only to make voice references, not on the synthesis hot path): SEANet
conv stack (init k=7, 4 stages with ratios 8/6/5/4 → cumulative 960×, 64→512 ch, final
k=3), an 8-layer transformer (hidden 512, 8 heads, head_dim 64, FFN 2048, RoPE 1e4,
LayerNorm+bias, GELU, LayerScale), a downsample conv k=4 s=2 (25 → 12.5 Hz), then the RVQ
argmin loop. Metadata records `encoder.num_quantizers=32` with
`encoder_valid_num_quantizers=16` — 16 codebooks are used, 16 are dead weight carried from
the upstream config.
- **Decoder** (the hot path: every generated frame): 16 codebook lookups + per-split
`output_proj` (256→512), `conv_pre`, an 8-layer transformer (hidden 512, 16/16 heads,
head_dim 64, FFN 1024, **sliding window 72**, RMSNorm, SwiGLU, LayerScale, input_proj
1024→512 / output_proj 512→1024), a ConvNeXt upampler (`upsampling_ratios=[2,2]`, causal
transposed convs + depthwise k=7), then a DAC-style decoder `dec.0..3`
(`upsample_rates=[8,5,4,3]`, channels 1536→768→384→192→96, `decoder_dim=1536`) whose blocks
are SnakeBeta → causal transposed conv (k = 2·stride) → 3 residual units (dilations 1/3/9)
→ snake_post → conv_post k=7 → 96→1.
- All 32 codebook tensors are **F32, 67.1 MB total** (16 encode + 16 decode, `[256, 2048]`),
deliberately unquantized: the converter's `should_quantize` keeps RVQ codebooks and their
wrapping projections at F32 because nearest-neighbour lookup is sensitive to per-row
quantisation noise (`docs/ARCHITECTURE.md` "Quantisation policy"; mirrored in
`quantize.sh`). 49 of the codec's 55 3-D conv tensors are **F16** with non-alignable K
(7, 3, 2, 1, 16, 10, …) — they cannot be block-quantized at all.
- **No flow matching, no diffusion, no vocoder-with-ODE.** It is a pure RVQ codec
(Mimi/SoundStream/Mimi-style: SEANet + transformer + DAC decoder). That matters for the
NPU question: every component is convolution, attention, or embedding lookup.
### 1.3 Where the payload came from, and what the official weights look like
Official upstream (`huggingface.co/api/models`, retrieved 2026-10-19):
| repo | last modified | license | weights |
|---|---|---|---|
| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 2026-01-23 | apache-2.0 | `model.safetensors` BF16, 1,928,677,440 params, plus an embedded `speech_tokenizer/model.safetensors` |
| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 2026-01-29 | apache-2.0 | same layout, named-speaker tables |
| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | 2026-01-29 | apache-2.0 | 1.7B only |
| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` / `-0.6B-CustomVoice` | 2026-01-29 | apache-2.0 | narrow variant (hidden 1024) |
| `Qwen/Qwen3-TTS-Tokenizer-12Hz` | 2026-01-29 | apache-2.0 | codec alone, F32, 170,557,441 params |
So the GGUF is **not** the only public form: the safetensors are the canonical form
(Apache-2.0, no separate community license risk), and the GGUF is a lossy re-quantization of
them produced by the same author who wrote the runtime (`Serveurperso`), via
`checkpoints.sh` → `convert.py` (F32 GGUF) → `quantize.sh` (BF16/Q8_0/Q4_K_M).
Sizes to remember for any re-export: the talker container is 1.93 B params in BF16
(that count includes the embedded codec), the codec alone is 170.6 M params in F32.
**Portability consequence #1 — the GGUF is a dead end off-ggml.** Q4_K/Q6_K are ggml
block formats (256-element super-blocks, F16 sub-block scales, 144 B / 210 B per block). No
ONNX graph, no Vitis-AI/AIE-ML compiler, and no vendor NPU runtime consumes them; AMD's own
quantizer works from ONNX (`ryzenai-quantize`, QDQ/QOP power-of-two — Ryzen AI Software
1.8.0 release notes, https://ryzenai.docs.amd.com/en/latest/relnotes.html). Moving the
payload to a non-ggml stack therefore means starting from the safetensors again, not
converting the GGUF.
**Portability consequence #2 — the quantization split is semantic, not cosmetic.** The
converter keeps RVQ codebooks, `input_proj`/`output_proj` around them, and every 1-D and
non-alignable-conv tensor out of the k-quants precisely because quantizing them breaks voice
fidelity. Any re-quantization path (GPTQ/AWQ/QDQ-INT8/INT4 for an NPU compiler) inherits that
constraint: the talker backbone is requantizable, the codec's codebook lookups and SnakeBeta
convs are not, at least not without re-measuring voice quality.
## 2. What the runtime is
`qwentts.cpp` @ `5916bf7c` (2026-10-16, MIT; upstream model Apache-2.0) is a C++17 ggml
application — 43 files under `src/` + `tools/`, three binaries (`qwen-tts`, `qwen-codec`,
`tts-server`) over one public plain-C99 ABI (`src/qwen.h`, `qt_*`, `QT_ABI_VERSION`
range-checked; `tests/abi-c.c` is compiled `-std=c99 -Wall -Werror -pedantic` on every
build). It is not a llama.cpp derivative: it links ggml directly and builds its own graphs.
Five graph families, and their shapes are the whole portability story:
| graph | batch/time axis | notes |
|---|---|---|
| talker prefill | T = prompt length (hundreds–thousands) | summed text+codec embedding streams, 28-layer GQA decoder, KV cache |
| talker decode | **T = 1 per audio frame** | 12.5 frames/s → 80 ms of audio per step; `codec_head` → 3072 logits |
| code predictor (MTP) | **T = 1, 16 steps per frame** (2 prefill + 14 decode, cache reset every frame) | 5-layer decoder, 15 heads × 2048 |
| speaker encoder | one pass per voice registration | ECAPA-TDNN with mel fused into the graph |
| codec decode | offline: T = 300-frame chunks; streaming: **T = 1 frame** with persistent conv/KV-ring state | the streaming graph is built once with fixed topology so the CUDA graph cache stays in pure replay |
Per generated frame the host therefore runs ~1 + 16 + 1 tiny graph computes for 80 ms of
audio, and `tts-server` serializes synthesis FIFO (`--max-batch` defaults to 1, and the
fleet runs it at 1). This is the archetypal latency-bound, bandwidth-bound,
batch-1 workload; the only big-M matmuls in the whole system are the prefill and the
chunked (buffered) codec decode.
Op contract actually issued to ggml (`src/`, counted): `mul_mat` 77, `cont` 70, `add` 57,
`mul` 40, `rms_norm` 22, `clamp` 15, `rope_ext` 12, `get_rows` 12, `set_rows` 8,
`scale_bias` 6, `swiglu_split` 5, `sqr` 5, `mean` 5, `elu` 4, `col2im_1d` 4, `soft_max` 3,
`conv_1d` 3, `im2col` 2, `cumsum` 2, `sin`, `sigmoid`, `tanh`, `div`, `argsort`, `transpose`,
`permute`, `reshape/view`, plus the CPU-side Philox multinomial sampler.
**No custom ggml op is required today.** The repo doc still says the fork "provides the two
custom ops the codec needs: `GGML_OP_SNAKE` and `GGML_OP_COL2IM_1D`"; that is stale at the
pinned commit. Verified 2026-10-19:
- the submodule `ServeurpersoCom/ggml` @ `cc5df49` is upstream `ggml-org/ggml` master
(`a05c992`, 2026-10-15) **+2 commits, 0 behind** (GitHub compare API), and
`include/ggml.h` is byte-identical to upstream master (`diff` empty);
- the 2 commits are `193e339` (2026-06-19) "cpu: add fused snake activation
(mul -> sin -> sqr -> mul -> add)" — a **CPU subgraph fusion**, not an op (the runtime
builds snake with `ggml_sin`/`ggml_sqr`/`ggml_mul`/`ggml_add` in `src/dac-decoder-v2.h:222`),
and `cc5df49` "sycl: fix get_rows on reordered quantized weights";
- `ggml_col2im_1d` is an **upstream ggml API** (same declaration, line 2097, in both trees);
the Vulkan backend fuses the snake chain through ggml's generic subgraph fuser
(`ggml-vulkan.cpp:14548 ggml_vk_can_fuse_snake`, `vulkan-shaders/snake.comp`).
Consequence: any stock ggml build (upstream, or llama.cpp's vendored ggml) can execute this
payload; only the CPU snake fusion is fork-specific, and it is a performance nicety.
Backend/device support: every tool takes `--device <name>` (`CUDA0`, `Vulkan0`, `MTL0`,
`CPU…`, default best available), resolved through the ggml backend registry; the deployed
container turns CUDA/HIP/Vulkan/Metal/SYCL/RPC/BLAS all OFF and compiles the native-CPU
backend with OpenMP, with CPU threads = `hardware_concurrency()/2` (`src/backend.h`).
The registry in this ggml knows CPU, CUDA, Metal, SYCL, Vulkan, WebGPU, ZDNN, OpenCL,
**Hexagon**, BLAS, RPC, VirtGPU, **CANN**, ZenDNN, **OpenVINO**, ET — and nothing AMD-NPU
(no `xdna`/`aie`/`vitis`/`ryzen` string anywhere in the ggml tree).
Voice handling (what the fleet depends on): the registry is **process RAM only**; a restart
empties it, which is why the unit has a boot warmer. `POST /v1/audio/voices` accepts either
`{name, ref_text, wav_b64}` (server extracts) or `{name, ref_text, spk_b64, rvq_b64}`
(pre-extracted latents verbatim); `ref_text` present ⇒ ICL clone mode, absent ⇒ x-vector-only
mode. `.spk` = the ECAPA x-vector; `.rvq` = 16 codes/frame packed 11 bits LSB-first
(`src/rvq-file.h`). `wav` and `pcm` `response_format` are different audio at the same seed
(buffered chunked decode vs stateful streaming decode) — the fleet already encodes that rule.
Parity expectations for any port: the upstream harness matches the PyTorch reference per
stage (embed cos 0.999988, all 28 hidden taps ≥ 0.999893, logits 0.999824) but explicitly
reports free-run divergence (`CodesFull exact 4.96 %`, `Audio cos 0.095`) and explains why —
one argmax tie at FP epsilon diverts the AR trajectory. So a migrated endpoint must be
accepted by listening + per-stage cosine, never by waveform diff against today's audio.
---
## 3. The three answers Main asked for
### ANSWER A — Qwen3-TTS GGUF → ONNX: no, and it does not matter
There is **no GGUF→ONNX path** and there will not be a useful one: GGUF is a weight
container for ggml, its Q4_K/Q6_K block layouts have no ONNX consumer, and nothing in the
GGUF carries the *graph* (the talker/predictor/codec structure lives in C++ code, only the
op-level structure exists in ggml at runtime). But the GGUF is not the payload's only form —
official Apache-2.0 safetensors are upstream, and **four independent ports of the full
pipeline already exist**:
| port | form | what it proves |
|---|---|---|
| `xkos/Qwen3-TTS-12Hz-1.7B-ONNX` (HF, updated 2026-03-03, apache-2.0) | FP16+FP32 ONNX, fp16/voice_clone + fp16/voice_design splits: `talker_decode.onnx`, `code_predictor.onnx`, `code_predictor_kv.onnx`, `code_predictor_embed_g0..15.onnx`, `text_embedding.onnx`, `codec_embedding.onnx`, shared `speaker_encoder.onnx`, `speech_tokenizer_encoder.onnx`, `speech_tokenizer_decoder.onnx`, `model_cache.npz` | every component of this architecture exports to ONNX, codec decoder included |
| `zhangluoyang/qwen3-tts-onnxruntime` | export tooling + Python inference for Base/CustomVoice/VoiceDesign | the export is reproducible from the official checkpoints |
| `Echo9Zulu/Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO`, `aurora2035/Qwen3-TTS-12Hz-0.6B-Base-OpenVINO-INT8` (2026-03) | OpenVINO IR, **INT8**: `talker`, `code_predictor`, `codec_embedding`, `text_model`, `openvino_speaker_encoder_model`, `speech_tokenizer/speech_decoder` | the talker and predictor INT8-quantize for an NPU-targeting stack |
| `litert-community/Qwen3-TTS-12Hz-0.6B-Base` (2026-10-17, apache-2.0) | `qwen3_tts_int8.litertlm` / `_int4.litertlm` (LiteRT-LM `OmniEngine`), legacy split graphs `talker_int4.tflite` (prefill_32/prefill_128/decode signatures, KV 1024, blockwise-32 OCTAV INT4), `mtp_folded_int8.tflite` (16 inner steps × 5 layers folded into one graph, in-graph argmax+gather, GPTQ int8, ~5× faster), `codec_partA` (transformer, fp32) + `codec_partB` (conv upsampler, XNNPACK FORCE_FP16, ~2.5×, "ASR-identical") | an NPU/mobile compiler accepts the whole thing — at the price of graph splitting, static prefill buckets, and folding the per-frame inner loop |
What the ports had to concede, and what any port inherits:
- the generation structure does not fit existing engine loops: LiteRT-LM's Engine "does not
support this generation structure yet", so their first shape was **three graphs + a
host-side loop**, and the MTP step was "invoked 17× per frame" until they folded it;
- per-step dispatch cost dominates: even on a Snapdragon 8 Elite Gen 5 they keep the MTP
loop, text projection, embedding gathers and the codec on CPU "avoiding 16 synchronous
CPU↔GPU round-trips per 80 ms frame", putting only the 28-layer talker on the GPU;
- RTF on that hardware is **2.57–2.97 (Galaxy S26) and 5.65–8.48 (Pixel 10 Pro)** for the
0.6 B model, i.e. 2.6–8× slower than real time. Dankclaw's current CPU engine measures
**warm RTF 0.55–0.67 with TTFA 94–112 ms** on the 1.7 B model (fleet trial, 2026-10-18).
An NPU port is not "faster by default" for this workload; it has to beat that.
- quantize the LM (int8/int4: proven), keep the codec's codebook lookups and SnakeBeta convs
in fp16/fp32 (the LiteRT split does exactly this).
So: the answer to "can the fleet's payload be exported" is *yes — from the safetensors, not
from the GGUF*, and the work is a pipeline rewrite (multi-graph host loop, per-frame fold,
static prefill buckets), not a conversion. That is the same conclusion NeuTTS-Air's
precompiled-graph route encodes: on AMD's Linux stack the graph is vendor-curated, so the
fleet chooses the model that AMD already compiled rather than compiling Qwen3-TTS.
### ANSWER B — is there any ggml backend for an AMD NPU? Not upstream; three out-of-tree, none for TTS
Checked against the trees today (2026-10-19), because this was the crux:
- **Nothing in-tree.** `ggml-org/llama.cpp` master (4,237 paths, GitHub tree API) contains no
`xdna`/`aie`/`vitis`/`ryzen` file or directory; the ggml backend set is CPU, BLAS, CUDA,
HIP, Metal, Vulkan, SYCL, OpenCL, WebGPU, MUSA, RPC, CANN (Ascend), **Hexagon** (Qualcomm),
OpenVINO (Intel CPU/GPU/NPU), ZenDNN, ZDNN, VirtGPU, ET. The deployed fork's registry
matches. The feature request "Feature Request: XDNA backend" (llama.cpp issue **#21725**,
opened 2026-04-10) is still **open**, 34 comments; the older #1499 was closed stale
2025-04-07. So Main's premise is confirmed: no ggml XDNA backend exists to flip a flag on.
- **`Cyronius/ggml-xdna`** — a real ggml backend (`XDNA0`) loaded into *stock* llama.cpp via
`GGML_BACKEND_PATH`, no fork. It does bfp16 (8-value blocks sharing an exponent) GEMMs from
GGUF weights, and is **prefill-only**: it takes only matmuls in prompt chunks ≥ 1024 tokens
(`GGML_XDNA_MIN_BATCH` default 1024, `GGML_XDNA_MIN_MFLOP` 256), attention and every decode
token stay on the Vulkan device; pinned to llama.cpp release b10944; tested on Strix Point
(HX 370) only — Strix Halo "should pass, but nobody has run it on one yet"; Linux support
new and untested on hardware by the author; needs `libxrt-npu2`, `/dev/accel/accel0`,
unlimited `memlock`. Best measured prefill gain 1.88× on a 27 B model; models under
2,048-wide **lose** (0.77× at width 1,536).
- **`BrandedTamarasu-glitch/OllamaAMDNPU`** — a llama.cpp fork with `-DGGML_XDNA=ON`,
explicitly targeting **Ryzen AI MAX / XDNA2 on Linux** (`/dev/accel/accel0`, XRT 2.21.75+,
amdxdna, unlimited memlock). It implements **only `GGML_OP_MUL_MAT`**, dispatched to
mlir-aie-compiled `.xclbin`s built per fixed tile shape (e.g. `M=2048 K=2048 N=64`,
int8 in / int32 out, per-row weight scales cached at first use), with a second slot for a
different K. Its own numbers on a Ryzen AI MAX 385 with Llama-3.1-8B Q4_K_M: NPU prefill
12–19 t/s vs **Vulkan iGPU 600–930 t/s**, NPU decode **0.65–1.40 t/s** ("Decode does not
use the NPU … M=1 per token, no xclbin covers it"), 0.6 % AIE utilization, and the stated
conclusion "**For LLaMA 8B on Strix Halo: use Vulkan exclusively**". Single-maintainer,
AI-assisted, self-measured — treat as a proof of concept, but it is the closest published
picture of what XDNA2 graph execution costs.
- Also referenced in the same thread: `Atomic-Germ/OpenFlowLM-Next` (whole-model NPU engine,
LLM-only — and slower than the iGPU in the same comparison), `GaTmaNnes/xdna2-`,
`hardWorker254/Npu-Embeddings-XDNA1` (adds Whisper ASR ops, NPU1/Linux). None is a TTS path.
Why none of them would carry this workload even if installed: the TTS hot loop is **batch-1
by construction** (one talker step + 16 predictor steps + one codec frame per 80 ms), which
sits under every dispatch gate those backends define (≥1024 tokens / ≥32768 M·N·K / tile M
covering only prefill), and the codec half of the pipeline is conv, col2im, gather and
elementwise ops — not `MUL_MAT` — so it would fall back to CPU op-by-op. The NPU case for
this host is therefore *not* "run Qwen3-TTS on the NPU"; it is "run a vendor-compiled AR TTS
graph on the NPU", which is exactly the NeuTTS-Air spike.
### ANSWER C — would `qwentts.cpp` build for the gfx1151 iGPU? Yes, via **Vulkan**, not HIP
- Upstream ships the build scripts and claims the backends: `buildvulkan.sh` ("AMD/Intel GPU
(Vulkan)"), `buildcuda.sh`, `buildsycl.sh`, `buildcpu.sh`, `buildall.sh` (all backends,
runtime dynamic loading), and README line 6 "runs on CPU, CUDA, Metal, Vulkan". CMake knobs
`GGML_VULKAN` / `GGML_HIP` / `GGML_SYCL` are the same ones the fleet's Containerfile turns
OFF, so the engine rebuild is a flag change: `-DGGML_VULKAN=ON` plus
`--device Vulkan0` on the same command line the unit runs today.
- The codec's exotic ops are covered by the Vulkan backend, which is the thing that decides
whether a GPU build is coherent: `vulkan-shaders/col2im_1d.comp`, `snake.comp`,
`swiglu*.comp`, `cumsum*.comp` exist and `supports_op` handles `GGML_OP_COL2IM_1D` /
`CONV_TRANSPOSE_1D` / `CUMSUM`. By contrast `COL2IM_1D` appears in **no** HIP, OpenCL,
WebGPU, Hexagon or CANN source — a **HIP** (ROCm) build would shard the ConvNeXt/DAC
transposed convs back to the CPU per op. On an AMD iGPU the portable path is RADV/Vulkan.
- `--device` also accepts a multi-device split (`-dev`-style naming is ggml's), and the
runtime's documented FP16 guard (`--clamp-fp16`) exists for sub-Ampere CUDA accumulation;
nothing in the tree is NVIDIA-specific.
- Unverified by me: any vendor statement pairing this fork with gfx1151. Supporting evidence
that the hardware/driver combination is viable: this very host already runs a Vulkan
endpoint on the gfx1151 iGPU (fleet ground truth, 2026-10-19), and Mesa's Vulkan
implementation is officially supported on Radeon Software for Linux
(AMD release notes RN-AMDGPU-UNIFIED-LINUX-25.20.3). A gfx1151 Vulkan build of this fork is
a build-and-measure spike, not an unsupported port.
- The real constraint is the one already in the fleet docs: the iGPU belongs to another
endpoint, and GPU+NPU concurrency has hung Strix Halo machines. An iGPU TTS build competes
with the text endpoints for the same silicon; the NPU route's only genuine advantage is
that it does not (the XDNA backend authors make the same argument: "Use NPU when the iGPU
is busy … no GPU contention").
---
## 4. What any replacement must reproduce, and what the voices cost
Endpoint contract in production today (from the live unit/env, not the doc):
`tts-server --model qwen-talker-1.7b-base-Q4_K_M.gguf --codec qwen-tokenizer-12hz-Q8_0.gguf
--alias frank-tts-engine --host 0.0.0.0 --port 8880 --lang auto --max-batch 1`, no published
port (the façade on :7772 dials the private bridge name), `GET /health`, `GET /v1/models`,
`GET|POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}`, `POST /v1/audio/speech` with
`response_format` wav|pcm (different audio at equal seed) and optional `seed`,
`max_new_tokens`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `subtalker_*`.
Voice registry is RAM-only, so boot warming is mandatory, and `--alias` keeps the quantization
choice out of every consumer's config.
Live voice census on Dankclaw (2026-10-19, `/home/dank/voices/`): three packs
(`narrator1..3`), each carrying `reference.wav` (24 kHz mono crop), `reference16k.wav`,
`manifest.json` (keys: name, kind, generated_utc, pipeline, source, forbidden_region_guard,
silence_scan, crop, loudnorm, audio, acceptance, acceptance_passed) and ffmpeg analysis
artifacts. **No `.spk` and no `.rvq` exist on the live host** — the repo's
`build-voice-pack.sh` treats them as required-but-they-are-not-there, so today the engine
re-extracts the x-vector (and the ICL codes when a transcript is supplied) from the WAV at
registration time.
That is the portable part, and it is the same shape AMD's NeuTTS flow wants
(`encode_reference(reference.wav)` + a reference transcript): the fleet's voice identity is
*the WAV plus its transcript*, processed at request/registration time by the model's own
speaker encoder + codec — **never a trained per-voice artifact**. Qwen3-TTS cloning is
inference-time conditioning (`generate_voice_clone(ref_audio=…, ref_text=…, x_vector_only_mode=…)`;
`create_voice_clone_prompt` exists purely to avoid recomputing prompt features). So any
replacement that consumes a WAV (+transcript) keeps all three narrators alive with no
re-derivation, and if it wants precomputed latents instead, the LiteRT precedent shows what
that looks like: a 4 KB x-vector `.npy` per voice, extracted offline
(`extract_speaker_embedding.py`) — for the 1.7 B base checkpoint that vector is 2048-dim
(`qwen3-tts.spk_enc.embedding_length = 2048` in the shipped GGUF; the LiteRT 0.6 B bundle
ships a 1024-dim one).
One trap for a mixed-runtime future: `.spk`/`.rvq` files, if they are ever written, are
engine-specific encodings (ECAPA x-vector layout; 11-bit-packed RVQ stream) and are **not**
interchangeable between runtimes. Keep `reference.wav` as the pack's source of truth and
regenerate anything derived per engine — which is what the NeuTTS re-clone plan does anyway.
## 5. Sources
Runtime/payload (all read locally at pinned versions, 2026-10-19):
- `github.com/ServeurpersoCom/qwentts.cpp` @ `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`
(commit date 2026-10-16; also upstream HEAD today): `README.md`, `docs/ARCHITECTURE.md`
(860 lines), `src/*`, `tools/*`, `docs/DOCKER.md`, `buildvulkan.sh`/`buildcpu.sh`/`buildall.sh`.
- submodule `ServeurpersoCom/ggml` @ `cc5df49a6b878db2b68b9a7b45976e279594627d` (2026-10-16);
delta vs `ggml-org/ggml` master `a05c992` (2026-10-15) = ahead 2 / behind 0, per
https://github.com/ServeurpersoCom/ggml/compare/ggml-org:master...HEAD ;
`include/ggml.h` identical to upstream
https://raw.githubusercontent.com/ggml-org/ggml/master/include/ggml.h .
- Live payload bytes: `/home/dank/models/Qwen3-TTS-GGUF/qwen-talker-1.7b-base-Q4_K_M.gguf`
(1,219,245,248 B) and `qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B), headers parsed from
the host; sizes/sha256 match `scripts/download-frank-tts-engine-qwentts.sh`.
- Live units/env/packs: `dankclaw/voiceservice/{Containerfile.engine,tts-engine-start.sh,tts-engine-warm.sh,quadlet/*}`,
`scripts/frank-tts-engine.env`, `scripts/frank-tts.env`, `/home/dank/voices/*`.
Model provenance:
- `https://huggingface.co/api/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base` (+ CustomVoice,
VoiceDesign, 0.6B variants, `Qwen3-TTS-Tokenizer-12Hz`), apache-2.0, last-modified
2026-01-23 / 2026-01-29; safetensors param counts 1,928,677,440 (BF16, incl. embedded
`speech_tokenizer/`) and 170,557,441 (F32 codec).
- `https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base/raw/main/README.md` (clone API:
`generate_voice_clone(ref_audio, ref_text, x_vector_only_mode)`,
`create_voice_clone_prompt`; reference stack = PyTorch + `device_map="cuda:0"` + bf16 +
flash-attn-2, Python 3.12 recommended).
- `https://github.com/QwenLM/Qwen3-TTS`.
Portability precedents:
- `https://huggingface.co/xkos/Qwen3-TTS-12Hz-1.7B-ONNX` (2026-03-03) ·
`https://github.com/zhangluoyang/qwen3-tts-onnxruntime` ·
`https://huggingface.co/Echo9Zulu/Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO` (2026-03-22) ·
`https://huggingface.co/aurora2035/Qwen3-TTS-12Hz-0.6B-Base-OpenVINO-INT8` (2026-03-18) ·
`https://huggingface.co/litert-community/Qwen3-TTS-12Hz-0.6B-Base` (2026-10-17) incl.
`legacy/README.md` (graph splits, MTP fold, codec partA/partB, RTF table).
AMD NPU / ggml backend status:
- `https://api.github.com/repos/ggml-org/llama.cpp/git/trees/master?recursive=1` (no
XDNA/AIE/Vitis backend, 2026-10-19) ·
`https://github.com/ggml-org/llama.cpp/issues/21725` (open since 2026-04-10) ·
`https://github.com/ggml-org/llama.cpp/issues/1499` (closed stale 2025-04-07).
- `https://github.com/Cyronius/ggml-xdna` (prefill-only, gates, Linux requirements, results
measured 2026-10-05/06 on HX 370) ·
`https://github.com/BrandedTamarasu-glitch/OllamaAMDNPU` (XDNA2 ggml backend on Strix Halo,
phases 1–12, "use Vulkan exclusively", measured on Ryzen AI MAX 385).
- `https://github.com/amd/RyzenAI-SW` (main, 669 paths, 2026-08-18): audio content is ASR
only (`Demos/ASR/Whisper`, `Demos/ASR/Parakeet-TDT`, `Transformer-examples/ASR/Whisper-AI`);
**no TTS example anywhere in the official Ryzen AI Software tree**.
- `https://ryzenai.docs.amd.com/en/latest/relnotes.html` (Ryzen AI SW 1.8.0: ONNX-format
models, ONNX Runtime as the loader, power-of-two QDQ/QOP quantization).
- Mesa/RADV on AMD iGPUs: `https://www.amd.com/en/resources/support-articles/release-notes/RN-AMDGPU-UNIFIED-LINUX-25-20.3.html`
(Mesa Vulkan driver officially supported).
Status: COMPLETE for the assigned slice (payload architecture, runtime architecture,
portability). `[INFERENCE]` marks: none above except the Mesa-version caveat in ANSWER C,
which is flagged there.
+191
View File
@@ -0,0 +1,191 @@
# TTS on the Strix Halo NPU — spike verdict (2026-10-19)
Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in
one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is
the gfx1151 iGPU behind a build flag we already own.
Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can
reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the
payload is and where it can be moved). This page owns only the **measured spike result** and
the **model-side contract** neither slice has. Throwaway work ran in containers under
`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and
`frank-tts-engine` still serve `narrator1|2|3` on CPU.
Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers ·
**[vendor]** primary vendor source, with the version it holds for.
## 1. Verdict
**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own
instruments; the runtime that owns this model family opens the device and then declines the
graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a
path that cannot be reproduced is not a foundation. **[spike]**
**Go on the iGPU, and the lever is in this repo.**
`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with
`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running
container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and
nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in
the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a
world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the
gfx1151 *beside* `frank-dragoon`, not a handover. §5 carries the census and both gates:
the **heap gate is PASSED** — RADV's `DEVICE_LOCAL` heap is 77.33 GiB with 65.16 GiB of
budget while dragoon is loaded, so memory is not the constraint — which leaves the
contention gate (dragoon's TTFT with a synthesis in flight) and one unproven piece:
whether qwentts.cpp's codec ops have Vulkan kernels for every op. **[host]** **[spike]**
Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs
only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a
dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan
exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way.
## 2. The two arms do not share a real-time bar
|Arm|Talker|Codec|Codes per second of audio|Bar|
|---|---|---|---|---|
|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**|
|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**|
Measured from the staged NPU artifact — the contract any future arm must satisfy:
|Fact|Value|
|---|---|
Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**|
Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**|
Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**|
Input ceiling|4096 tokens **total**, input + output **[vendor]**|
Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**|
Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target <dir>` so a second torch never shadows the venv's. **[spike]**|
Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**|
**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds
provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s
audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`,
`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]**
## 3. Exoneration chain — why the blame is not the hardware
Every row is an instrument, not an inference.
|Probe|Result|
|---|---|
`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**|
`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**|
`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**|
`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**|
`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**|
Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**|
Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**|
Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**|
No NPU token rate was ever measured, so none is quoted here. The single successful load
(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been
reproduced by the byte-identical command since, with the model files pristine by inode and
mtime. **[spike]**
## 4. Host gates the spike proved (so they are not re-litigated)
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host
gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`,
`--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail.
Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet
already does for `kfd`/`renderD*`. **[spike]**
- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro
driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin
deb is what carries `libxrt_driver_xdna.so`. **[spike]**
- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device
node is *not* the modern failure mode. **[host]**
- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` +
`libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`);
boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]**
- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census.
A multi-GB NPU partition plus an AIE compile workspace does not fit beside
`frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that
pressure. **[host]**
## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon`
The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19
(read-only, nothing loaded or unloaded):
|Measure|Value|Reading|
|---|---|---|
iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**|
DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**|
Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**|
TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**|
**Two gates, in this order.**
1. **Heap gate — PASSED.** RADV does **not** hand a container tenant the 2 GiB DRM
carve-out. `vulkaninfo` inside the engine's own base image (`ubuntu:24.04` plus
`mesa-vulkan-drivers vulkan-tools`, `--device /dev/dri`, rootless) sees
`Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, and reports:
|Heap|size|budget with dragoon loaded|flags|
|---|---|---|---|
|0|38.67 GiB|32.58 GiB|host-visible, not device-local|
|1|**77.33 GiB**|**65.16 GiB**|`DEVICE_LOCAL`|
The heaps sum to the ~116 GiB aperture in Mesa's ⅓/⅔ APU split — the same RAM twice:
a Vulkan allocation is a GTT mapping, so a budget is a ceiling on mappings, not a pile
of free silicon memory. ggml's Vulkan backend asks for `DEVICE_LOCAL`, which is heap 1,
and the TTS engine needs roughly 4 GiB of it. **Memory stops being the question.** **[spike]**
Probe traps: `vulkaninfo --batch` is not a flag in this `vulkan-tools` (it prints the
help page and exits 1 — use `vulkaninfo --text`), and the container has **no NVIDIA
ICD**, so Vulkan device 0 is the iGPU here. Do not trust that ordering — select by
device *name*, or an image that grows an `nvidia_icd.json` silently points TTS at the
3090 slot.
2. **Contention gate — PASSED, measured 2026-10-19 after the cutover.** With a 16.9 s
synthesis occupying the device, `frank-dragoon` answered a streamed 124-token request at
**17 ms to first token and ~35 tok/s decode**, against 17 ms and ~33 tok/s for the same
request with the card idle. Compute-unit sharing showed up as nothing. One request per
arm, no ladder: these are the gate's evidence, not a throughput record, and the engine's
own shape lives in [[Frank TTS Engine]].
**Change surface — as landed** (PR #34, `6248f02`; the CPU shape stays reachable as the
rollback):
1. `Containerfile.engine` build stage: add **`libvulkan-dev glslc spirv-headers`**. The
prediction above (`libvulkan-dev glslang-dev`) is wrong on two counts and the configure
says so: ggml looks for `glslc`, which Ubuntu's `glslang-tools` does **not** ship (it
ships `glslangValidator`), and `find_package(SPIRV-Headers CONFIG REQUIRED)` needs the
distro's `spirv-headers`, which does carry the CMake config. `glslc` is its own source
package on noble, at 2023.8 — old enough that only `GL_KHR_cooperative_matrix` compiles,
so the coopmat2 / integer-dot / bf16 / micro-float shader families are built out. Suspect
that compiler before the device if a Vulkan prefill ever reads as slow.
2. `-DGGML_VULKAN=OFF` → `ON`: one word in the written-out flag list.
3. Runtime stage: `mesa-vulkan-drivers` **and** `libvulkan1` — the ICD and its loader. An
image with a loader and no ICD enumerates zero devices.
4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — the
DIRECTORY, never the node: on this hybrid box `renderD128` is the NVIDIA card and
`renderD129` the amdgpu iGPU (`readlink -f /sys/class/drm/renderD129/device/driver` →
`amdgpu`). On this host the udev rule already makes the nodes world-writable, so the
group is belt-and-braces, not the gate.
5. Engine argv: `--device Vulkan0`, reached through `QWENTTS_DEVICE` in
`scripts/frank-tts-engine.env`. **An empty value passes no flag at all** — that is the
CPU rollback, and it is why the entrypoint's default is an `-v` test rather than `:=`,
which also fires on a set-but-empty value and would silently undo the rollback.
Gate 1 recipe, already run and **PASSED**: `mesa-vulkan-drivers` + `vulkan-tools` in a
throwaway container off the same base, `--device /dev/dri`, then `vulkaninfo --text` and read
the `VkPhysicalDeviceMemoryProperties` heap sizes — no model, no load, no compute. The probe
trap: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` — it prints the help
page and exits 1, which reads like a driver failure. Every package involved has a candidate on
the host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools`
16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are older
but present. Script: `/home/dank/spikes/npu-tts/vk_probe.sh` on Dankclaw.
Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI
pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because
systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img`
exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents;
`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins.
## 6. Re-visit condition
Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai`
build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that
model appearing in a runtime's own supported catalog with a Linux runner. Then the
prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices,
the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar.
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
#
# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
# Two DB-side facts (the provider was created by hand via `ak shell`, never
# repo-managed) silently broke that login, and BOTH had previously been
# mis-recorded here as a single "the property-mapping evaluator is broken, so
# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
#
# 1. provider.authorization_flow was NULL. The authorize view builds a flow
# planner from it and crashes with
# `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
# "Server Error — Powered by authentik", before any claim is touched.
# Fix: point it at default-provider-authorization-implicit-consent, the
# flow every other provider in this instance already uses.
# 2. The `grafana-email-scope` ScopeMapping's expression was the bare string
# `user.email`. authentik merges a scope mapping's result into the token as
# a MAPPING: a returned dict adds its keys as claims; a returned string is
# keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
# claim is ever emitted. Grafana, finding no email, falls back to
# GET <api_url>/emails, authentik 404s that path, and the login dies with
# "Login failed: InternalError". Fix: return
# {"email": user.email, "email_verified": True} — the shape the stock
# OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
# mapping has always worked, which is the proof the evaluator is fine.
#
# This is authentik DB state. It is NOT in any deployable file: a blueprint
# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
# facts and the symptom returns verbatim. Re-run this after any such rebuild.
# Every step is idempotent.
#
# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
# dankclaw/observability/quadlet/grafana.container and deploys normally.
#
# Usage: authentik-grafana-sso-fix.sh (override host: AUTHENTIK_HOST=…)
set -euo pipefail
HOST="${AUTHENTIK_HOST:-dankclaw}"
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
from authentik.flows.models import Flow
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.filter(name="grafana").first()
if not p:
raise SystemExit("no OAuth2Provider named 'grafana'")
# 1. The authorization flow every other provider uses.
want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
if not want:
raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
if p.authorization_flow_id != want.pk:
p.authorization_flow = want
p.save()
print(f"set grafana.authorization_flow -> {want.slug}")
else:
print(f"grafana.authorization_flow already {want.slug}")
# 2. The email scope mapping must RETURN A DICT, not a bare string.
m = p.property_mappings.filter(name="grafana-email-scope").first()
if not m:
raise SystemExit("grafana-email-scope mapping is missing")
fixed = 'return {"email": user.email, "email_verified": True}'
if m.expression.strip() != fixed:
m.expression = fixed
m.save()
print("repaired grafana-email-scope -> returns {email, email_verified}")
else:
print("grafana-email-scope expression already correct")
# 3. The id_token must carry the scope claims; Grafana reads identity from it.
if not p.include_claims_in_id_token:
p.include_claims_in_id_token = True
p.save()
print("enabled include_claims_in_id_token")
else:
print("include_claims_in_id_token already on")
PYEOF
echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
echo "the round trip must land on / and /api/user must answer 200 with an email."
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Set the authentik login-session TTL on Dankclaw.
#
# The login-session lifetime is governed by the `default-authentication-login`
# stage's `session_duration`, NOT a global setting and NOT SESSION_COOKIE_AGE
# (that is the pre-login session, sessions.unauthenticated_age). Authentik does
# not expose SESSION_SAVE_EVERY_REQUEST at all, so the session is an absolute
# window from login, not a sliding one.
#
# The field is a STRING in authentik's own idiom ("days=7", "seconds=30",
# "hours=12") — not a Python timedelta. Writing str(timedelta(days=7)) stores
# "7 days, 0:00:00", which authentik's timedelta_from_string parser REJECTS
# (ValueError), and login then crashes computing the session expiry. Always pass
# the "key=value" form; this script validates it before writing.
#
# Existing sessions keep their old expiry; the new TTL applies from the next
# login. This is DB state — it is NOT repo-managed and a blueprint re-import or
# rebuild-from-scratch reverts it. Run this again after such a rebuild.
#
# Usage: authentik-set-login-session-duration.sh ["days=7"] (default days=7)
set -euo pipefail
DUR="${1:-days=7}"
STAGE="default-authentication-login"
HOST="${AUTHENTIK_HOST:-dankclaw}"
# Refuse anything that is not the safe key=value duration idiom.
if ! printf '%s' "$DUR" | grep -Eq '^[A-Za-z]+=[0-9]+([ ]+[A-Za-z]+=[0-9]+)*$'; then
echo "invalid duration '$DUR' — use authentik form, e.g. 'days=7' or 'seconds=30'" >&2
exit 2
fi
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
from django.apps import apps
from authentik.lib.utils.time import timedelta_from_string
dur = "$DUR"
timedelta_from_string(dur) # validate the form before touching the row
ULS = apps.get_model("authentik_stages_user_login", "UserLoginStage")
s = ULS.objects.get(name="$STAGE")
s.session_duration = dur
s.save(update_fields=["session_duration"])
back = ULS.objects.get(name="$STAGE").session_duration
assert back == dur, "persisted value mismatch: %r" % back
print("session_duration set to", repr(back), "->", timedelta_from_string(back))
PYEOF
echo "Login-session TTL for '$STAGE' is now $DUR (applies on next login)."
+8 -13
View File
@@ -29,10 +29,10 @@
# The scratch mirrors the production container contract exactly (serve-frank-
# primary.sh): --network host, --init, seccomp=unconfined (io_uring for the
# n-gram table), memlock unbounded (the ~132 GiB pinned ledger), /dev/kfd +
# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
# differences: the container name (arbench, so prod's identity is untouched)
# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
# prefill is cold — the prod tier at kvcache/frank-archon is left intact.
# /dev/dri, the .rad ro, the tune cache mount. The ONLY differences: the
# container name (arbench, so prod's identity is untouched) and the prefix cache
# switched OFF (--prefix-cache-mib 0), so every prefill is cold — the tier is
# host RAM now, there is no per-run disk directory to wipe.
set -uo pipefail
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-archon.env}"
@@ -60,9 +60,8 @@ set -a; . "$ENV_FILE"; set +a
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
FRESH_KV="$FRANK_ARCHON_STATE_DIR/kvcache/$SCRATCH"
rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
# --- the prefix cache is switched off below, so there is no scratch KV tree ---
# to wipe; the cold prefill comes from --prefix-cache-mib 0.
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
# --- boot the scratch: the production run line, scratch name + fresh KV ------
@@ -73,7 +72,6 @@ podman run -d --name "$SCRATCH" \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
-v "$FRANK_ARCHON_STATE_DIR":/data \
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_ARCHON_IMAGE" \
@@ -87,12 +85,9 @@ podman run -d --name "$SCRATCH" \
--placement "$FRANK_ARCHON_PLACEMENT" \
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
--prefix-cache-dir "/data/kvcache/$SCRATCH" \
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
--prefix-cache-mib 0 \
--busy-prefill-chunk "${FRANK_ARCHON_BUSY_PREFILL_CHUNK:-auto}" \
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""Destroy measurement ledgers in the surviving pages.
Point of use (AGENTS.md → Documentation Policy): a rate/throughput figure for
anything not running and not being compared is destroyed, not filed. This pass
deletes
1. any section whose heading is a bench/throughput/rate/ladder/verdict ledger,
2. any table row or standalone bullet that REPORTS a rate (`123.4 t/s`,
`tok/s`, `ms/token`, `TTFT`, acceptance %, `llama-bench`, a bench arm),
and it never touches a shape number (ctx, slots, MiB budgets, ports) or a bullet
under a *Laws* / *Wire contract* / *Rollback* heading, where the sentence is the
payload and the number is decoration. Dry run by default; `--apply` writes.
Usage: axe-measurements.py [--apply] [--verbose]
"""
import pathlib
import re
import sys
VAULT = pathlib.Path("serverAppConfigs-docs")
RATE = re.compile(
r"\d[\d,.]*\s*(t/s|tok/s|tokens?/s|tok s|tps|ms/token|µs/token|GB/s|GiB/s)\b|"
r"\b(llama-bench|bench (arm|ladder|window)|decode ladder|prefill ladder)\b|"
r"\bacceptance (rate|of)\b|\bdraft_n\b|\bttft\b|\bServer-Timing\b",
re.I,
)
# A heading is a ledger only when the noun LEDS it and nothing instructional
# follows: `## Throughput` yes, `## 6. The context ladder (find the max bootable
# window by measurement)` no — that one is the method, which is point-of-use.
LEDGER_HEAD = re.compile(
r"^#{2,4}\s+(?:\d+[.) ]\s*)?(bench(es|mark|marks)?|throughput|results?|timings?|"
r"measured (figures|numbers|results|rates|rates)|rates|verdict|acceptance|ladder)"
r"\b[^|]{0,40}$",
re.I,
)
INSTRUCTIONAL = re.compile(r"how|why|law|policy|method|procedure|find|by measurement|execution|flip|checklist", re.I)
PROTECTED = re.compile(
r"^#{2,4}\s+.*(law|wire|rollback|contract|shape|eras|files|health|see also|related|"
r"boot|preflight|environment)",
re.I,
)
def blank_rate_cells(text):
"""Kill throughput COLUMNS wherever they appear: a cell that reports a rate
becomes an em-dash, so the row keeps whatever else it records (the payload
name, the reason, the rollback) and loses only the dead number."""
out = []
for line in text.split("\n"):
st = line.strip()
if st.startswith("|") and st.endswith("|") and RATE.search(line):
cells = line.split("|")
for i, c in enumerate(cells):
if RATE.search(c):
cells[i] = " — "
line = "|".join(cells)
out.append(line)
return "\n".join(out)
def collapse_eras(text):
"""One Eras row per era is fleet law; the NUMBERS in the old rows are not.
Keep the current era verbatim, fold everything older into one row that says
the numbers were destroyed and where the rollback actually lives (git)."""
lines = text.split("\n")
out = []
in_eras = False
ncol = 0
folded = 0
for line in lines:
if line.startswith("#"):
in_eras = bool(re.match(r"^#{2,4}\s+.*eras\b", line, re.I))
if folded:
out.append("| " + " | ".join(
[f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured "
"number destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1)))
folded = 0
out.append(line)
continue
st = line.strip()
if in_eras and st.startswith("|"):
cells = [c.strip() for c in st.strip("|").split("|")]
if set("".join(cells)) <= set(":- "):
ncol = len(cells)
out.append(line)
continue
if ncol == 0:
ncol = len(cells)
if re.search(r"2026-1[01]|current|active|running", st, re.I):
out.append(line)
else:
folded += 1
continue
if folded and not st.startswith("|"):
out.append("| " + " | ".join(
[f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured number "
"destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1)))
folded = 0
ncol = 0
out.append(line)
if folded:
out.append("| " + " | ".join(
[f"≤ 2026-09 — {folded} earlier eras destroyed 2026-10-19"] + ["—"] * max(0, ncol - 1)))
return "\n".join(out)
def main(argv):
apply = "--apply" in argv
verbose = "--verbose" in argv
total_lines = 0
pages = 0
for page in sorted(VAULT.rglob("*.md")):
s = str(page)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
text = page.read_text(errors="replace")
before = len(text.split("\n"))
if s.startswith("serverAppConfigs-docs/LLM/"):
text = collapse_eras(blank_rate_cells(text))
lines = text.split("\n")
out = []
skip = False
under_protected = False
cut = 0
for line in lines:
if line.startswith("#"):
under_protected = bool(PROTECTED.search(line))
skip = (bool(LEDGER_HEAD.match(line)) and not under_protected
and not INSTRUCTIONAL.search(line))
if skip:
cut += 1
continue
if skip:
cut += 1
continue
stripped = line.strip()
out.append(line)
if under_protected:
# Eras rows are the rollback record, Law bullets are the payload:
# the number is decoration inside them, and the row/bullet is not
# a ledger entry. Leave the whole section alone.
continue
if stripped.startswith(("|", "-", "*")) and RATE.search(line):
# Destroy only when the line EXISTS to report a number: a table
# row, or a bullet whose verb is measurement.
if stripped.startswith("|") or re.search(
r"\b(measured|measures|bench|benched|ran at|runs at|clocked|hit|"
r"landed at|came in at|peak|peaks at|sustained|wall.clock)\b", line, re.I):
out.pop()
cut += 1
final = re.sub(r"\n{3,}", "\n\n", "\n".join(out))
saved = before - len(final.split("\n"))
if saved or cut:
pages += 1
total_lines += max(saved, cut)
print(f" {s.replace('serverAppConfigs-docs/', '')}: -{max(saved, cut)}")
if apply:
page.write_text(final.rstrip("\n") + "\n")
print(f"{'APPLIED' if apply else 'dry run'}: {pages} pages, {total_lines} lines of ledger")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+26 -3
View File
@@ -39,6 +39,8 @@ SERVICE_ORDER = (
"loki",
"alloy",
"node-exporter",
"cadvisor",
"gpu-exporter",
"alertmanager",
"authentik-postgres",
"authentik-server",
@@ -92,6 +94,8 @@ DANKCLAW_SERVICES = {
"loki",
"alloy",
"node-exporter",
"cadvisor",
"gpu-exporter",
"alertmanager",
"authentik-postgres",
"authentik-server",
@@ -124,6 +128,8 @@ DANKSERVER_SERVICES = {
"readarr",
"frank-archon",
"node-exporter",
"cadvisor",
"gpu-exporter",
}
# FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -293,6 +299,17 @@ def services_for_path(path: str, target: str) -> set[str]:
if len(parts) >= 2 and parts[1] in {"caddy", "comfyui", "loki", "alloy", "hermes", "anubis", "seerr"}:
_add(services, parts[1])
return services
if len(parts) >= 3 and parts[1] == "host":
# Host state a service needs at boot lives under dankclaw/host/.
# The fabric-clock hold is frank-dragoon's NPU precondition (the
# engine refuses to start with HALOGEN_NPU_MODELS set while the
# clock moves) and the deployer installs and gates it, so changing
# it re-dispatches that endpoint. `grub` is the tracked boot line,
# which moves only through scripts/set-dankclaw-bootline.sh and is
# a reboot decision — it plans no service.
if parts[2].startswith("halogen-fabric-clock"):
_add(services, "frank-dragoon")
return services
if len(parts) >= 4 and parts[1] == "authentik" and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"authentik-postgres", "authentik-server", "authentik-worker", "authentik-outpost", "authentik-outpost-ldap"}:
@@ -363,7 +380,7 @@ def services_for_path(path: str, target: str) -> set[str]:
# consumer (prometheus before grafana, loki before alloy).
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
_add(services, name)
return services
if len(parts) >= 2 and parts[1] == "gitea":
@@ -406,8 +423,8 @@ def services_for_path(path: str, target: str) -> set[str]:
# node-dankserver job had nothing to scrape.
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name == "node-exporter":
_add(services, "node-exporter")
if name in {"node-exporter", "cadvisor"}:
_add(services, name)
return services
# frank-archon's quadlet-era directory left the tree on 2026-09-26
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -438,6 +455,12 @@ def services_for_path(path: str, target: str) -> set[str]:
# change to it redeploying alertmanager is the only path that
# reaches it on the host.
_add(services, "alertmanager")
if name in {"gpu-exporter.py", "gpu-exporter.service"}:
# The host-process GPU exporter is installed directly by the
# gpu-exporter deploy on BOTH hosts, so its script and unit select
# that service (target_services then keeps it only on the host being
# planned). Editing the collector restarts the running exporter.
_add(services, "gpu-exporter")
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-zealot only. The ik_llama.cpp
+234 -15
View File
@@ -86,7 +86,7 @@ service_is_allowed() {
dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
dankclaw:comfyui|dankclaw:hermes|\
dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:alertmanager|\
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
dankserver:frank-archon|dankserver:node-exporter|\
dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
@@ -275,6 +275,36 @@ install_signal_relay() {
return 1
}
# Host-process GPU metrics exporter (amdgpu sysfs + nvidia-smi). Not a quadlet:
# it runs python3 from the checkout on the host so it can read the DRM/cgroup
# sysfs and shell out to nvidia-smi. Same two-address bind posture as
# node-exporter/cadvisor (loopback + this host's LAN), so it is never a wildcard.
deploy_gpu_exporter() {
local unit_dir="$HOME/.config/systemd/user" lan dropin i
case "$TARGET" in
dankclaw) lan=192.168.0.143 ;;
dankserver) lan=192.168.0.235 ;;
*) echo "gpu-exporter: unknown target '$TARGET'" >&2; return 2 ;;
esac
mkdir -p "$unit_dir"
cp scripts/gpu-exporter.service "$unit_dir/gpu-exporter.service"
dropin="$unit_dir/gpu-exporter.service.d"
mkdir -p "$dropin"
printf '[Service]\nEnvironment=GPU_EXPORTER_BINDS=127.0.0.1,%s\n' "$lan" \
> "$dropin/bind.conf"
systemctl --user daemon-reload
systemctl --user enable gpu-exporter.service
systemctl --user restart gpu-exporter.service
# The interpreter needs a moment to bind after restart (the same race that
# once failed a healthy signal-relay deploy).
for i in $(seq 1 15); do
curl -fsS -m 5 http://127.0.0.1:9401/metrics >/dev/null && return 0
sleep 1
done
echo "gpu-exporter is not answering on 127.0.0.1:9401/metrics" >&2
return 1
}
copy_loki_config() {
run_privileged mkdir -p /opt/loki/config
run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
@@ -487,13 +517,15 @@ seed_grafana_env() {
|| { echo "grafana env missing GRAFANA_OAUTH_CLIENT_SECRET; set it in dankclaw/.env" >&2; return 1; }
}
# The five observability containers left compose on 2026-09-11 (they were
# started by hand with podman-compose and never came back after a reboot).
# They are ordinary rootless Quadlet units now: same images, same mounts, same
# arguments, same container names, so nothing that addresses them by name —
# Caddy, Prometheus' own scrape config, Alloy's loki.write endpoint — changed.
# Only the published ports did: the three HTTP UIs bind 127.0.0.1, because the
# compose services published on 0.0.0.0 and none of them authenticates.
# The observability containers left compose on 2026-09-11 (they were started by
# hand with podman-compose and never came back after a reboot). Five are ordinary
# rootless Quadlet units: same images, same mounts, same arguments, same
# container names, so nothing that addresses them by name — Caddy, Prometheus'
# own scrape config — changed. Only the published ports did: the three HTTP UIs
# bind 127.0.0.1, because the compose services published on 0.0.0.0 and none of
# them authenticates. Alloy is the exception: a ROOT system service on
# Network=host, because reading the host journal needs privileges a rootless
# container cannot get. It is handled by deploy_alloy, not the loop below.
deploy_observability() {
local service="$1"
# Observability units live under <target>/observability/quadlet: Dankclaw
@@ -512,16 +544,14 @@ deploy_observability() {
copy_loki_config
build_compose_image loki
;;
alloy)
copy_alloy_config
build_compose_image alloy
;;
grafana)
copy_grafana_provisioning
seed_grafana_env || return 1
;;
node-exporter)
;;
cadvisor)
;;
*)
echo "unknown observability service: $service" >&2
return 2
@@ -534,11 +564,72 @@ deploy_observability() {
alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;;
grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
alloy) wait_for_health alloy http://127.0.0.1:12345/-/healthy 60 ;;
node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
esac
}
# Mirror an image the deployer built rootless into the ROOT image store. A root
# quadlet reads root storage; a root build (build_compose_image is rootless) is
# not possible on this host — the build container's resolver fails DNS — so the
# only way to get a root-readable copy is save-and-load. Compares image ids so a
# rebuilt alloy actually re-lands in root storage instead of leaving a stale tag.
mirror_image_to_root() {
local ref="$1" rid
rid="$(podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" || {
echo "ERROR: rootless image $ref is not built; cannot mirror to root" >&2
return 1
}
if [[ "$(run_privileged podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" == "$rid" ]]; then
return 0
fi
local tar; tar="$(mktemp /var/tmp/alloy-mirror.XXXXXX.tar)"
podman save -o "$tar" "$ref"
run_privileged podman load -i "$tar"
rm -f "$tar"
}
# Install a quadlet into the SYSTEM generator dir (root manager) instead of the
# user dir. Boot persistence is the unit's own [Install] section (multi-user
# target); the v1 system generator creates the .wants on daemon-reload, so —
# like the user units — no `systemctl enable` (generated units cannot be enabled
# here).
install_system_quadlet() {
local source_dir="$1" unit="$2"
run_privileged mkdir -p /etc/containers/systemd
run_privileged cp "$source_dir/$unit.container" /etc/containers/systemd/
run_privileged systemctl daemon-reload
}
# Alloy as a ROOT system service on Network=host. See the unit file: a rootless
# container cannot read the 0640 journal files, and it cannot reach Loki's
# 127.0.0.1:3100 publish from a bridge netns. Root + host network fixes both;
# the price is that this one container runs as root and on the host network.
deploy_alloy() {
copy_alloy_config
build_compose_image alloy # rootless build (resolver works)
mirror_image_to_root localhost/franksdank/alloy:latest
# Retire the rootless unit BEFORE the system one starts: both managers
# generate alloy.service, and two containers named "alloy" both binding
# 127.0.0.1:12345 cannot coexist. Move the user quadlet out of the generator
# dir (it regenerates from there) and stop the user unit.
if [[ -f "$quadlet_unit_dir/alloy.container" ]] ||
systemctl --user is-active --quiet alloy.service 2>/dev/null; then
systemctl --user stop alloy.service 2>/dev/null || true
if [[ -f "$quadlet_unit_dir/alloy.container" ]]; then
local trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-alloy-root-cutover"
mkdir -p "$trash"
mv -- "$quadlet_unit_dir/alloy.container" "$trash/alloy.container"
echo "retired rootless alloy.container -> $trash"
fi
systemctl --user daemon-reload
fi
install_system_quadlet dankclaw/observability/quadlet alloy
run_privileged systemctl restart alloy.service
wait_for_health alloy http://127.0.0.1:12345/-/healthy 60
}
build_compose_image() {
local service="$1" image cfhash oldhash containerfile
# Every file the image COPIES IN, beside the Containerfile. The skip below
@@ -1329,6 +1420,9 @@ deploy_anubis() {
# port gate is the same as every cutover since 2026-09-16: nothing
# starts until 7770 is released.
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
# The payload dir is its own twin because the NPU lane's containment gate below
# reads it; every artifact path under it stays a full literal as usual.
LLM_DRAGOON_MODEL_DIR="${LLM_DRAGOON_MODEL_DIR:-/home/dank/models/halogen-v2-abliterated}"
LLM_DRAGOON_CKPT="${LLM_DRAGOON_CKPT:-/home/dank/models/halogen-v2-abliterated/qwen38-flash-next-v2-abliterated.hgn}"
LLM_DRAGOON_CKPT_BYTES="${LLM_DRAGOON_CKPT_BYTES:-66687678432}"
LLM_DRAGOON_NGRAM="${LLM_DRAGOON_NGRAM:-/home/dank/models/halogen-v2-abliterated/qwen38-flash-next-ngram.hgn}"
@@ -1337,6 +1431,18 @@ LLM_DRAGOON_VISION="${LLM_DRAGOON_VISION:-/home/dank/models/halogen-v2-abliterat
LLM_DRAGOON_VISION_BYTES="${LLM_DRAGOON_VISION_BYTES:-897916416}"
LLM_DRAGOON_TOK_DIR="${LLM_DRAGOON_TOK_DIR:-/home/dank/models/halogen-v2-abliterated/tokenizer}"
LLM_DRAGOON_TOK_FILES="${LLM_DRAGOON_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
# NPU lane (2026-10-19). HALOGEN_NPU_MODELS is the whole switch and the parity
# twin of the env file's line: empty here AND in the env = the pre-lane
# endpoint. The gates below mirror serve-frank-dragoon.sh, and the vendor's
# fabric-clock host unit (dankclaw/host/, verbatim at tag v0.16.2) is installed
# ONLY while the lane is named — on Strix Halo the iGPU and the NPU co-running
# hang the machine while the fabric clock moves, so halogen refuses to boot with
# the lane armed unless that clock is held.
LLM_DRAGOON_NPU_MODELS="${LLM_DRAGOON_NPU_MODELS:-qwen3-embedding-0.6b}"
LLM_DRAGOON_NPU_DIR="${LLM_DRAGOON_NPU_DIR:-/home/dank/models/halogen-v2-abliterated/npu}"
LLM_DRAGOON_DEV_ACCEL="${LLM_DRAGOON_DEV_ACCEL:-/dev/accel/accel0}"
LLM_DRAGOON_XRT_DIR="${LLM_DRAGOON_XRT_DIR:-/usr/lib/x86_64-linux-gnu}"
LLM_DRAGOON_NETWORK="${LLM_DRAGOON_NETWORK:-dankclaw_default}"
retire_halogen_dragoon_quadlet() {
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
@@ -1415,6 +1521,77 @@ deploy_llm_dragoon() {
fi
done
# The container network has to exist before the run line names it (a typo
# there is a restart-looping unit, not a config error anyone reads).
if [[ -n "$LLM_DRAGOON_NETWORK" ]] && ! podman network exists "$LLM_DRAGOON_NETWORK"; then
echo "ERROR: podman network $LLM_DRAGOON_NETWORK does not exist" >&2
echo "frank-dragoon: create it or blank FRANK_DRAGOON_NETWORK (pasta default)" >&2
return 1
fi
# --- NPU lane: the same predicates the serve script checks --------------
if [[ -n "$LLM_DRAGOON_NPU_MODELS" ]]; then
# One door into the container: the read-only /models mount of the
# payload dir. A staging dir outside it needs a nested bind whose
# mountpoint runc cannot create through a read-only parent, so the
# parity twin of the serve gate refuses the shape before the restart.
if [[ "$LLM_DRAGOON_NPU_DIR/" != "$LLM_DRAGOON_MODEL_DIR"/* ]]; then
echo "ERROR: NPU staging dir $LLM_DRAGOON_NPU_DIR is outside the payload dir $LLM_DRAGOON_MODEL_DIR — the lane's files reach the container only through the read-only /models mount" >&2
return 1
fi
if [[ ! -e "$LLM_DRAGOON_DEV_ACCEL" ]]; then
echo "ERROR: NPU node $LLM_DRAGOON_DEV_ACCEL absent — amdxdna is not bound (is iommu=pt still on the boot line?)" >&2
return 1
fi
# Access, not existence: a rootless container cannot open a root:render
# 0660 node (its namespace maps one uid/gid, and keep-groups cannot grant
# an unmapped gid — measured 2026-10-19). Install the mode rule, re-trigger
# the node, then PROVE the node is other-accessible before the restart.
run_privileged install -m 0644 dankclaw/udev/98-halogen-npu-access.rules \
/etc/udev/rules.d/98-halogen-npu-access.rules
run_privileged udevadm control --reload-rules
run_privileged udevadm trigger --sysname-match="$(basename "$LLM_DRAGOON_DEV_ACCEL")"
# (`-o FILE` is not "other-writable" — it means "exists", and `-o` is also
# the OR operator inside test), so the bit is read straight off the mode.
if [[ $(( 8#$(stat -c '%a' "$LLM_DRAGOON_DEV_ACCEL") & 8#2 )) -eq 0 ]]; then
echo "ERROR: $LLM_DRAGOON_DEV_ACCEL is still $(stat -c '%a %U:%G' "$LLM_DRAGOON_DEV_ACCEL") after installing the access rule — the lane cannot open it rootless" >&2
return 1
fi
for xl in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
if [[ ! -e "$LLM_DRAGOON_XRT_DIR/$xl" ]]; then
echo "ERROR: XRT NPU library $LLM_DRAGOON_XRT_DIR/$xl absent (Ubuntu libxrt2 + libxrt-npu2)" >&2
return 1
fi
done
for nm in ${LLM_DRAGOON_NPU_MODELS//,/ }; do
if [[ ! -d "$LLM_DRAGOON_NPU_DIR/$nm" ]]; then
echo "ERROR: NPU model $nm not staged in $LLM_DRAGOON_NPU_DIR" >&2
echo "frank-dragoon: stage it: bash scripts/download-frank-dragoon-npu.sh" >&2
return 1
fi
done
# The hold must exist BEFORE the engine starts and survive every boot
# (a resume hands the clock back too — the unit's WantedBy covers it).
run_privileged install -m 0755 dankclaw/host/halogen-fabric-clock \
/usr/local/sbin/halogen-fabric-clock
run_privileged install -m 0644 dankclaw/host/halogen-fabric-clock.service \
/etc/systemd/system/halogen-fabric-clock.service
run_privileged systemctl daemon-reload
run_privileged systemctl enable --now halogen-fabric-clock.service
if ! run_privileged /usr/local/sbin/halogen-fabric-clock status; then
echo "ERROR: fabric clock is not held — halogen refuses to start with HALOGEN_NPU_MODELS set" >&2
echo "frank-dragoon: journalctl -u halogen-fabric-clock.service -n 40" >&2
return 1
fi
elif [[ -x /usr/local/sbin/halogen-fabric-clock ]]; then
# Lane rolled back: hand the clock back to the driver and drop the boot
# hook. `disable` deletes the INSTALLED copy of this unit — unlike the
# tracked LLM unit symlinks the fleet's disable law protects, this file
# is repo-owned and the next lane-on deploy reinstalls it.
run_privileged systemctl disable halogen-fabric-clock.service 2>/dev/null || true
run_privileged /usr/local/sbin/halogen-fabric-clock release || true
fi
retire_halogen_dragoon_quadlet
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
# the container) before waiting on the port. The stray sweep is keyed
@@ -1434,6 +1611,33 @@ deploy_llm_dragoon() {
run_privileged systemctl enable frank-dragoon.service
run_privileged systemctl restart frank-dragoon.service
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
# The lane's own gate. /health proves the engine is up, nothing about the
# NPU programs; one real embedding is the observable output. The first
# request after a boot is a warmup artifact (fleet law), so retry before
# calling a booted lane dead.
if [[ -n "$LLM_DRAGOON_NPU_MODELS" ]]; then
for nm in ${LLM_DRAGOON_NPU_MODELS//,/ }; do
lane_ok=0
for _try in 1 2 3 4 5; do
if curl -fsS -m 60 -H 'content-type: application/json' \
-d "{\"model\":\"$nm\",\"input\":[\"frank-dragoon npu lane gate\"]}" \
http://127.0.0.1:7770/v1/embeddings 2>/dev/null \
| python3 -c 'import json,sys
v = json.load(sys.stdin)["data"][0]["embedding"]
raise SystemExit(0 if isinstance(v, list) and len(v) and all(isinstance(x, (int, float)) for x in v) else 1)'; then
lane_ok=1
break
fi
sleep 10
done
if [[ "$lane_ok" != 1 ]]; then
echo "ERROR: NPU model $nm answered no vector on /v1/embeddings" >&2
echo "frank-dragoon: read the boot ledger: tail -100 /home/dank/logs/frank_dragoon.log" >&2
return 1
fi
echo "frank-dragoon: NPU lane verified — $nm embedded"
done
fi
}
# deploy_llm_zealot — frank-zealot is llama.cpp (PrismML fork build) on the
@@ -1463,6 +1667,12 @@ LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism-b10
# 868.32 MiB worst-case VRAM (595.29 model + 248.10 compute in the boot ledger)
# lands AFTER the engine's fit check, so it is not inside the q8_0 budget.
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
# The refusal adapter is part of the payload identity now, so the deployer
# checks it beside the other two files — but it has its OWN fetcher, so a
# missing adapter must not send the operator to the bonsai9 script. Integrity
# (bytes + sha256) is the fetcher's job at download time; this gate only asks
# whether the file the env names exists, exactly like the two above.
LLM_ZEALOT_LORA="${LLM_ZEALOT_LORA:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf}"
deploy_llm_zealot() {
# Refuse rather than restart-loop: a unit whose runtime or payload is
@@ -1484,6 +1694,13 @@ deploy_llm_zealot() {
return 1
fi
done
if [[ ! -f "$LLM_ZEALOT_LORA" ]]; then
echo "ERROR: frank-zealot refusal adapter absent at $LLM_ZEALOT_LORA" >&2
echo "frank-zealot: run scripts/download-frank-zealot-uncensored-lora.sh first;" >&2
echo "frank-zealot: the env names the adapter, so the unit would refuse to boot" >&2
echo "frank-zealot: without it — starting the unit anyway is a crash loop." >&2
return 1
fi
relink_unit frank-zealot
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-zealot.service
@@ -1815,7 +2032,9 @@ deploy_service() {
copyparty) deploy_copyparty ;;
plex) deploy_plex ;;
tautulli) deploy_tautulli ;;
prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;;
alloy) deploy_alloy ;;
prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
gpu-exporter) deploy_gpu_exporter ;;
comfyui) deploy_comfyui ;;
*) deploy_compose_service "$service" ;;
esac
+5 -3
View File
@@ -133,13 +133,15 @@ fi
valid_service() {
local service="$1"
case "$TARGET:$service" in
# dankclaw:searxng-network is client-only: a network quadlet with no
# remote dispatch handler, so the remote gate does not (must not) list it.
dankclaw:caddy|dankclaw:gluetun|dankclaw:searxng-network|dankclaw:searxng-valkey|\
dankclaw:searxng|dankclaw:marreta|dankclaw:pihole|dankclaw:open-webui|dankclaw:gitea-postgres|\
dankclaw:gitea-server|dankclaw:homepage|\
dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:gpu-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:open-notebook|\
dankclaw:frank-tts-engine|dankclaw:frank-tts|\
@@ -147,8 +149,8 @@ valid_service() {
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
dankserver:frank-archon|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|dankserver:cadvisor|dankserver:gpu-exporter|\
dankserver:prowlarr|dankserver:frank-archon|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
+109
View File
@@ -0,0 +1,109 @@
#!/usr/bin/env python3
"""Destroy the vault's dead weight: era ledgers, retired runtimes, dead experiments.
`AGENTS.md` was treating a measurement as a record worth keeping forever. It is not: a
throughput figure for a payload that no longer runs, on a runtime that no longer exists,
is a liability — the next agent spends context on a number that AI progress has already
obsolete. This pass moves those pages to `.AGENTTRASH/` (repo law: never `rm`) and then
deletes every reference to them from the surviving pages, including the sentence that
carried the link when the link was the sentence's only content.
Usage: destroy-vault.py <list-file>
List file: one vault-relative path per line; a directory path destroys the whole tree.
"""
import pathlib
import re
import shutil
import sys
VAULT = pathlib.Path("serverAppConfigs-docs")
TRASH = VAULT / ".AGENTTRASH" / "20261019-destruction"
def targets(list_file):
out = []
for line in pathlib.Path(list_file).read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
p = VAULT / line
if p.is_dir():
out += [f for f in sorted(p.rglob("*.md"))]
elif p.is_file():
out.append(p)
else:
print(f" SKIP (missing): {line}")
return out
def main(argv):
if len(argv) < 2:
print(__doc__)
return 2
doomed = targets(argv[1])
# Stems come from the trash too, so re-running the pass (or running it after a
# partial destruction) still knows which names are dead and keeps scrubbing.
stems = {p.stem for p in doomed}
if TRASH.is_dir():
stems |= {p.stem for p in TRASH.rglob("*.md")}
moved = 0
for p in doomed:
dest = TRASH / p.relative_to(VAULT)
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(p), str(dest))
moved += 1
print(f"destroyed {moved} pages ({sum(len(p.read_text(errors='replace').splitlines()) for p in TRASH.rglob('*.md'))} lines staged in trash)")
# Repoint-or-delete: a link to a destroyed page is itself dead information.
link_wl = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|([^\]]*))?\]\]")
link_md = re.compile(r"\[[^\]]*\]\(([^)#]*\.md)(?:#[^)]*)?\)")
pages_cleaned = links_dropped = lines_dropped = 0
for page in sorted(VAULT.rglob("*.md")):
s = str(page)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
text = page.read_text(errors="replace")
lines = text.split("\n")
out = []
hits = 0
def dead(target):
# NOT pathlib.Path(...).stem: that strips a real suffix off names like
# `Lemonade and sd.cpp`, so a link to a destroyed page survives and the
# vault then fails its own link check.
t = target.strip()
if t.endswith(".md"):
t = t[:-3]
return t in stems or pathlib.PurePosixPath(t).name in stems
for line in lines:
found = [t for t in link_wl.findall(line) if dead(t[0])]
for m in link_md.finditer(line):
if dead(m.group(1)):
found.append(m.group(1))
if not found:
out.append(line)
continue
hits += len(found)
stripped = link_wl.sub(
lambda m: m.group(2) if (m.group(2) and not dead(m.group(1))) else "",
line)
stripped = link_md.sub(
lambda m: "" if dead(m.group(1)) else m.group(0), stripped)
# What is left of the line after the dead links are gone?
residue = re.sub(r"[\s\-*•·|:;,.\u2014\u2013]+", "", stripped)
if residue:
out.append(re.sub(r"\s+$", "", re.sub(r"^(\s*[-*]\s*)[\s:;,.\u2014-]+", r"\1", stripped)))
else:
lines_dropped += 1
if hits:
page.write_text("\n".join(out))
pages_cleaned += 1
links_dropped += hits
print(f"cleaned {pages_cleaned} surviving pages: {links_dropped} dead links dropped, "
f"{lines_dropped} lines left empty and deleted")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env bash
# download-frank-dragoon-npu.sh — stage the NPU half of frank-dragoon: the
# vendor-precompiled AIE programs halogen-flash-server runs on the XDNA2 NPU
# when HALOGEN_NPU_MODELS names them. These are NOT model weights you can
# quantize yourself: XDNA2 executes only programs compiled by AMD's AIE
# compiler, so the only source is peonist-ai's per-model NPU repo, and the
# engine checks every file's size and sha256 against its own catalog at boot
# (/opt/halogen/npu/models.txt inside the image) and refuses to load a model
# whose files do not match.
#
# The manifest below IS that catalog block (dumped from the pinned 0.16.2
# image on 2026-10-19), so the files land in the layout the entrypoint
# expects — <dir>/<model-id>/devices/*, <dir>/<model-id>/<model-id>.hnpw and
# <dir>/<model-id>/tokenizer/tokenizer.json — and the endpoint can keep
# /models read-only and open no outbound connection at boot (HALOGEN_DOWNLOAD
# stays unset; the vendor's own README: with it unset the container makes no
# outbound connections at all).
#
# The fetch law, same as every sibling script: curl -f (an access error must
# not become a "successful" 153-byte artifact), the exact byte count and the
# sha256 from the manifest, --verify-only to re-check a staged set.
#
# Adding a second NPU model: read its block out of the image's catalog and add
# a case below —
# podman exec frank-dragoon-halogen grep "^file <model-id> " /opt/halogen/npu/models.txt
# podman exec frank-dragoon-halogen grep "^model <model-id> " /opt/halogen/npu/models.txt
# and add the model id to HALOGEN_NPU_MODELS in frank-dragoon.env.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
# The env file owns the staging path (it must sit inside the payload dir, which
# is the container's only /models mount); an exported FRANK_DRAGOON_NPU_DIR wins
# so the bats seam can stage somewhere disposable.
DEST="${FRANK_DRAGOON_NPU_DIR:-}"
if [ -z "$DEST" ] && [ -f "$ENV_FILE" ]; then
DEST="$(sed -nE 's/^FRANK_DRAGOON_NPU_DIR=(.*)$/\1/p' "$ENV_FILE" | tail -1)"
fi
: "${DEST:?FRANK_DRAGOON_NPU_DIR is not set and $ENV_FILE names no FRANK_DRAGOON_NPU_DIR}"
HF="https://huggingface.co"
MODELS="${HALOGEN_NPU_MODELS:-}"
if [ -z "$MODELS" ] && [ -f "$ENV_FILE" ]; then
MODELS="$(sed -nE 's/^HALOGEN_NPU_MODELS=(.*)$/\1/p' "$ENV_FILE" | tail -1)"
fi
VERIFY_ONLY=0
case "${1:-}" in
--verify-only) VERIFY_ONLY=1 ;;
"") ;;
*) echo "usage: ${0##*/} [--verify-only] (models come from HALOGEN_NPU_MODELS)" >&2; exit 2 ;;
esac
[ -n "$MODELS" ] || { echo "no NPU model named — set HALOGEN_NPU_MODELS or run with it in the environment" >&2; exit 1; }
# manifest <model-id> -> repo revision + one "path bytes sha256" per file.
manifest() {
case "$1" in
qwen3-embedding-0.6b)
NPU_REPO="peonist-ai/halogen-npu-qwen3-embedding-0.6b"
NPU_REV="4960fd9d4423dd15373ad21c83db48dbdb3b2798"
FILES=(
"devices/NOTICE.md 3248 0f267b3d2523ddf14997ee58660a492a9eaf10ccf6b08bf19dfdc85b9a478b91"
"devices/consts.hnpb 5086496 ae430e41545a0b1012938cda373b0c450364d3be631111cca3a8c43679f93336"
"devices/devices.hnpm 24047 4e9bca9d95d084fb89ce5b99f579a00666c8ca1efae2613c53e3a4878cc9f174"
"devices/u0.elf 4931968 b38f165886cb037b7d3951d9cb878473eaba58a432acd45416ff85fd88aa4e46"
"devices/u1.elf 5456000 b1811feee851b8af833b37721da92099d89772110815bbbe2c3fa1c276f0ff42"
"devices/u2.elf 6505552 406ea9a95b470628a7382e376f4a72cdf4d7aa4c59b4ddf0a4bdd139d63d6ff2"
"devices/u3.elf 6817216 bc205d51b523d770151a0c61968f7c70bf624412827580bb6fe0aaf4650dc2ee"
"devices/u4.elf 7444032 a5a91338a2603acdcd5c11df532033fda2d3f5479ea44ae87f98fafbdff9a62d"
"devices/u5.elf 7432416 6c2bf41835569e0eba25ded7c05c7264ae69c6700bdeb4791057d884f2a4c126"
"devices/u6.elf 4941232 919d0038ca088eafddf195f3466ff38db27d3635be1cffa5d4d4b5140902a68e"
"devices/u7.elf 5488304 d904cf8afae9428dd167960549e18cce5441745f18c62f8c50975dffc077d3c2"
"devices/u8.elf 4941232 db3aef69b17825854ee0dd2df475125a3f5c872f95c686793d6af1ee83596fa7"
"devices/u9.elf 5488304 d182727e551084db8fb9600441d8aa373c7f09a3d267ae0a5e921ef92326db1c"
"qwen3-embedding-0.6b.hnpw 813032248 b9f9901429df7b46ec17153aa3899f738adbea8e4356080da0c6e6586fb750d1"
"tokenizer/tokenizer.json 11423705 def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"
) ;;
*)
echo "unknown NPU model '$1' — add its catalog block to manifest() in this script" >&2
return 1 ;;
esac
}
for MODEL in ${MODELS//,/ }; do
manifest "$MODEL" || exit 1
echo "== $MODEL ($NPU_REPO@$NPU_REV) =="
for entry in "${FILES[@]}"; do
read -r f want sha <<<"$entry"
path="$DEST/$MODEL/$f"
mkdir -p "$(dirname "$path")"
if [ "$VERIFY_ONLY" -eq 0 ]; then
if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
echo "HAVE $f ($want bytes) — skipping fetch"
else
echo "FETCH $f -> $path"
# HF answers /resolve/ with a 302 to the Xet CDN and curl
# DROPS the Range header across redirects, so resolve the
# effective URL first and resume against the CDN (the sibling
# 62 GiB fetch hit this on 2026-10-18).
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$HF/$NPU_REPO/resolve/$NPU_REV/$f") \
|| { echo "cannot resolve $NPU_REPO/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url"
fi
fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
if [ -L "$path" ]; then
echo "SYMLINK $path — the serve gate refuses symlinks; replace with a real file" >&2
exit 1
fi
got="$(stat -c %s "$path")"
[ "$got" = "$want" ] || { echo "SIZE $path: $got != $want" >&2; exit 1; }
got="$(sha256sum "$path" | cut -d' ' -f1)"
[ "$got" = "$sha" ] || { echo "SHA256 $path: $got != $sha" >&2; exit 1; }
echo "OK $f ($want bytes, sha256)"
done
done
echo "DOWNLOAD_DONE $DEST ($MODELS)"
+101
View File
@@ -0,0 +1,101 @@
#!/usr/bin/env bash
# Fetch the refusal-direction adapter that frank-zealot applies at load:
# AtomicChat/Ternary-Bonsai-2-9B-Uncensored-LoRA-GGUF — a **3.2 MB rank-1 LoRA**
# for prism-ml/Ternary-Bonsai-2-9B, not a checkpoint. Public repo: no auth
# header, and never a token on the command line (a Bearer token is
# world-readable in /proc/<pid>/cmdline).
#
# IT IS AN EDIT INSIDE THE COMPUTE GRAPH, NOT A SECOND MODEL. The card: B = -r
# (the refusal direction, row 23 of its own probe matrix), A = P^T(s *
# H(r^T W_fold)) built from the base pack's folded input basis, applied at every
# transformer layer (`target_layers = all`, `sites = 64`), with the embedding
# deliberately untouched because an embedding LoRA crashes this fork's graph.
# `--lora-scaled <file>:<scale>` is the flag pair: scale 1 is the card's
# "held-out refusals 0 %" point, scale 0 is the plain model again.
#
# THE RUNTIME IS A GATE, NOT A PREFERENCE. The card verified the adapter on
# `prism-b10777-e3de715` — the exact build frank-zealot runs — and says older
# forks (e.g. `prism-b10709`) refuse these packs with `unsupported
# prism.hadamard.version: 2`. The fork is already pinned for the payload's
# custom tensor type 142; this adapter adds a second reason it cannot move.
#
# PROVENANCE ASYMMETRY, STATED HONESTLY. The GGUF's own metadata names the pack
# the direction was fitted against:
# prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf
# prism.abliterate.base_sha256 = 3561df7ac7d26a71ffa991ac2db740b10841c3a350aa2676ccd3ff5d428b4e8b
# frank-zealot serves the PQ2_0 + MTP + Q8_0-embedding packing (its sha256 is
# c79fcdfa…, see download-frank-zealot-bonsai9.sh), which is NOT that file and is
# not on this host. The adapter still applies — same tensor names, and the
# metadata is provenance rather than a checksum gate — but its strength was
# calibrated on a sibling quantization. What proves the effect here is the
# behaviour of the endpoint (a refusal probe before and after the swap), not the
# metadata and not the card's table, which was measured on PTQ1_0.
#
# Fresh downloads, no resume; -f/--fail-with-body so an HTTP error is not a
# success, then byte count and sha256 (this repo is NOT LFS — the file is a
# plain git object, so the sha below is the content hash of the Hub's bytes as
# fetched, not an LFS oid). Idempotent: a complete file is skipped.
#
# Usage:
# scripts/download-frank-zealot-uncensored-lora.sh [DEST_DIR] [--verify-only]
set -euo pipefail
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/Bonsai2-9B-MTP}"
REPO="AtomicChat/Ternary-Bonsai-2-9B-Uncensored-LoRA-GGUF"
# name | bytes | sha256 of the Hub's bytes
FILES=(
"Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf|3155424|861b9c099fb5cd81438ba85bad30590847c777b1e18112586be7f4f9ae5fb07b"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
IFS='|' read -r name size sha <<<"$entry"
url="https://huggingface.co/${REPO}/resolve/main/${name}"
f="$DEST/$name"
have=0
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
have=1
fi
fi
if (( have )); then
echo "OK $name"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $name"
status=1
continue
fi
echo "FETCHING $name (from $REPO)"
tmp="$f.part"
curl -fsSL --fail-with-body -o "$tmp" "$url"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $name"
status=1
continue
fi
echo "VERIFIED $name"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+81 -31
View File
@@ -7,10 +7,15 @@
# only the reason a reader editing THIS line cannot recover from the page.
# Where the two disagree, this file wins on values; the boot ledger arbitrates
# the engine shape.
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
# 1.0.13 is pinned by digest on purpose: 1.2.0 is a DEAD END on this payload
# (acceptance 0.7123 vs 0.7549 — the page's Dead ends section names the suspect).
# The .rad file mounted under /models; /data holds the prefix-cache disk tier.
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed
# 1.8.1 (2026-10-18), pinned by digest. Two CLI breaks sit between it and the
# 1.0.13 line this endpoint ran for two eras, and both are boot refusals, not
# warnings: --vram-weights-mib/--vram-kv-mib were REMOVED at 1.3.0 (the card is
# now split by the engine's budget resolver — see below), and the prefix cache
# lost its disk tier. Draft acceptance has to be re-scored on any pin past 1.2.0:
# 1.2.0's MoE weight rotation measured 0.7123 against the 0.7549 control, and the
# suspect is untangled (the page's Dead ends section names it).
# The .rad file mounts under /models; the tune cache mounts separately below.
# The model dir holds ONLY .rad files — the HF download and staging stay beside it.
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
@@ -41,32 +46,47 @@ FRANK_ARCHON_TP=2
# stays EXACT for single-sequence decode and any step of <= 25 rows — see
# MAX_SEQS x (1 + SPEC_TOKENS) below.
FRANK_ARCHON_TP_WIRE=wht6
# Decode slots. A slot costs KV, never decode; the pool is sized to this
# contract and nothing more, so a FOURTH full-height conversation is REFUSED as
# too large to fit — never squeezed, paged or truncated. That is Frank's
# explicit trade: the freed MiB went to the expert slab.
FRANK_ARCHON_MAX_SEQS=3
# Decode slots. A slot costs KV and pinned RAM, never decode: the cache target is
# exactly MAX_SEQS x MAX_MODEL_LEN and the checkpoint store asks (1 + what the
# retention policy keeps) per slot. Ordered 2026-10-20: FOUR slots over a pool of at
# least 700,000 fp8 tokens, rounding UP if the resolver cannot land it exactly. The
# target is therefore 4 x 200,000 = 800,000 tokens and HOST_POOL_MIB decides whether
# the cache gets it (expert_vram_floor, core/place/planner.cpp:100-104: the card must
# keep expert-plane-minus-host-room). The 2-slot shape was the answer to a
# 365,204-token pool that backed 1.8 full-height sessions; at this pool the third and
# fourth conversations are backed by the cache instead of over-committing it.
FRANK_ARCHON_MAX_SEQS=4
# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
# limit, and models.yml contextWindow MUST quote this number (over-length is a
# refusal, not a truncation). The pool is sized for seqs x maxlen, not one
# sequence.
FRANK_ARCHON_MAX_MODEL_LEN=200000
FRANK_ARCHON_PLACEMENT=expert_tiered
# Pinned host pool for expert rows that do not fit VRAM. It moves WITH
# VRAM_WEIGHTS: the pair and this pool serve the per-rank resident set together,
# so cutting the slab spills here — under-sizing it is a boot failure (36,864
# refused a boot). The ledger's coverage line is the referee, not this file.
FRANK_ARCHON_HOST_POOL_MIB=44928
# Pinned host pool for expert rows that do not fit VRAM, PINNED ONCE PER RANK: the
# host must pin tp x this number, and the refusal that names it is
# core/engine_bringup.cpp:250-265. Under the budget resolver this is ALSO the KV
# lever -- the card keeps expert-plane-minus-this-room, so every MiB added moves one
# MiB of experts off the card and into the cache. Derived from the boot ledger (a
# 23.73 GiB floor at 44,928 room puts plane + 2 units at 67.60 GiB a rank), 49,152
# drops the floor to 19.6 GiB and leaves the cache ~6.9 GiB, which is enough to back
# the whole 4 x 200,000 target; the leftover returns to the experts. Cost is paid
# twice over: pinned RAM (tp x 48 GiB, against the TTM target below) and expert
# residency -- ~3.2 GiB a card of experts now lives in host RAM and is read over the
# x8 link when the router picks it. The ledger's coverage line is the referee, not
# this file: if vram_kv comes back under 700,000 tokens, raise this 1024 and redeploy.
FRANK_ARCHON_HOST_POOL_MIB=49152
# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
# floor is built from it.
FRANK_ARCHON_GPU_HEADROOM_MIB=96
# ONE budget per card: VRAM_WEIGHTS + VRAM_KV <= 30,969 MiB, the ceiling the
# engine's own budget table prints (derivation and the slab/pool price: the
# page's VRAM laws). Idle KV VRAM is ALSO the expert cache's spillover, so a
# bigger pool pays twice — which is why the pool sits at the contract and the
# slab takes the rest.
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
FRANK_ARCHON_VRAM_KV_MIB=5120
# There is no VRAM pair any more. --vram-weights-mib/--vram-kv-mib were removed
# at 1.3.0: the engine takes capacity - already-held - GPU_HEADROOM - arena,
# puts the static (non-expert) weights off the top, then divides the rest between
# the expert plane and the KV cache — experts first when the cache has no idle
# blocks to lend, and never less than ONE MAX_MODEL_LEN sequence for the cache.
# What moves the split from here is HOST_POOL_MIB (which experts stay resident),
# MAX_MODEL_LEN/MAX_SEQS (the cache bound) and GPU_HEADROOM_MIB. The slab size
# the prefill-slope law cares about is a boot-ledger line now, not a knob: read
# it after every resize instead of asserting it here.
# fp8 is the ONLY rung below bf16 on this engine: one kv_dtype for the whole
# cache, so llama.cpp's exact-K/quantized-V split (frank-zealot) is unservable
# here. bf16 costs 1.89x fp8, not 2x — the QSA block-key plane is dtype-fixed.
@@ -75,20 +95,50 @@ FRANK_ARCHON_KV_DTYPE=fp8
# 2, and 3 x (1+2) = 9 rows stays inside the wht6 25-row exactness bound.
FRANK_ARCHON_SPEC_TOKENS=2
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# Prefix cache tiers. The host tier is sized in ENTRIES and one entry tracks the
# cache dtype (58.50 KiB at fp8), so a dtype change re-sizes the parked set
# without touching these numbers.
FRANK_ARCHON_PREFIX_HOST_MIB=8192
FRANK_ARCHON_PREFIX_DISK_MIB=65536
# Step sharing. A prefill chunk that shares a step with decoding sequences holds
# the device for the chunk's whole time, and every decode row in that step waits
# that long for its next token: measured at this shape, decode-only steps 15.5 ms
# against chunk steps 166 ms, which is the harness's ~146 t/s falling to ~11 t/s
# while another session prefills (law and census: [[frank-archon]]).
# `auto` is a PREFILL-SPEED dial, not a decoder dial: it reserves one row plus the
# draft window per decoding sequence (3 tokens) and an eighth of the budget per
# waiting request or waiting prefill, and the source states it will not halve for
# decode rows alone. Live proof: prefill_chunks_capped 5,018 across 62,386 chunk
# steps. A number is the only value that applies whenever the chunk shares a step,
# and it must not go under the prefill stager's smallest pass -- 1025 rows here,
# named on the boot line "prefill staging: ... a step of more than 1024 tokens" --
# because below it every routed layer reads its pooled experts synchronously, so
# 512 is a trap and the vendor measures the whole job slower there. Interleave
# inserts N decode-only steps after a chunk step taken beside decoders: it protects
# their cadence and holds a waiting request out of those steps. That is the decoder
# dial. ARMED at 1 on 2026-10-20: at N=1 a stream takes its token from the chunk step
# AND one from the held step, ~2 tokens per 182 ms instead of one per 166 ms; the
# price falls on the prefilling prompt (its chunks lose every held step) and on any
# request still waiting to enter. `--prefill-span` re-funds a prefill only when it is
# ALONE, so it cannot buy this back while decoders run. Meter:
# radiance:engine_steps_held_total must leave zero, and the §9 buckets must move
# before N rises.
FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
FRANK_ARCHON_PREFILL_INTERLEAVE=1
# Prefix cache: pinned host memory only. The disk tier was removed outright
# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
# entry tracks the cache dtype (58.50 KiB at fp8). Watch prefix_hit_rate after
# this change: a drop is extra prefill, and prefill is exactly the traffic that
# costs decode its cadence.
FRANK_ARCHON_PREFIX_CACHE_MIB=8192
# Frank's pin law: the PLE n-gram table serves from pinned host RAM the cards
# read. `ram` REFUSES the start if memlock is too small — the unit's
# LimitMEMLOCK=infinity is the other half of this law.
FRANK_ARCHON_NGRAM_PLACEMENT=ram
# TTM preflight. The pinned ledger (~147 GiB) rides TTM's shared pages, and the
# kernel RESETS pages_limit to half of RAM on every boot; below the plan the
# load dies mid-pin with an anonymous allocator error. ttm-preflight.sh raises
# the knob itself before refusing by name, so recovery is a boot, not a visit.
FRANK_ARCHON_TTM_MIN_MIB=150000
# TTM preflight. The pinned ledger rides TTM's shared pages, and the kernel
# RESETS pages_limit to half of RAM on every boot; below the plan the load dies
# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
# itself before refusing by name, so recovery is a boot, not a visit.
# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT + the
# auto checkpoint store (MAX_SEQS x 10 slots of 111.98 MiB), per the bats ledger test;
# at the 49,152 pool with 4 slots that computes to 161,848 MiB.
FRANK_ARCHON_TTM_MIN_MIB=162000
FRANK_ARCHON_TTM_TARGET_MIB=204800
# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
# this unquoted on purpose, so keep multi-flag values as one string.
+45
View File
@@ -80,3 +80,48 @@ HALOGEN_TOP_K=20
# Weights stay pinned: the engine pins by default and only the retired w4b build
# honored HALOGEN_FLASH_PIN_TRUNK=0, which this payload rejects.
HALOGEN_WEIGHTS_LOCK=1
# --- NPU lane (XDNA2) -------------------------------------------------------
# HALOGEN_NPU_MODELS is the whole switch: EMPTY or unset = the endpoint exactly
# as it ran before this lane existed (no accel device, no XRT mount, no
# fabric-clock dependency, no /v1/embeddings). Rolling the lane back is blanking
# this one line and re-deploying — nothing else moves.
# Names are the vendor catalog's folder names (`/opt/halogen/npu/models.txt`
# inside the image) and the engine reads a model ONLY at /models/npu/<name>/ —
# HALOGEN_NPU_DIR is gone from the image and setting it is a boot refusal.
# The engine cannot fetch anything here: /models is read-only and
# HALOGEN_DOWNLOAD stays unset so the container opens no outbound connection at
# boot, so scripts/download-frank-dragoon-npu.sh must stage the files first
# (the serve gate and the deployer both refuse by name when a named model is
# missing). FRANK_DRAGOON_NPU_DIR therefore lives INSIDE the payload dir: the
# payload mount is the container's only handle on /models, and runc cannot
# mkdir a nested /models/npu mountpoint through a read-only bind — the lane's
# first boot died on precisely that (2026-10-19). Swap the payload dir and this
# path moves with it.
# Cost: ~1.5 GiB extra pinned host RAM per named model, and a hard dependency on
# the host holding the GPU fabric clock — on Strix Halo the iGPU and the NPU
# together hang the machine (and corrupt NPU results) while FCLK changes speed,
# so halogen refuses to start with this variable set unless the clock is held
# (dankclaw/host/halogen-fabric-clock; the deployer installs and gates it).
HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
FRANK_DRAGOON_NPU_DIR=/home/dank/models/halogen-v2-abliterated/npu
# Node paths and the XRT directory are variables so the bats seam can drive
# these gates on a box with no NPU. Ubuntu's distro XRT (libxrt2 + libxrt-npu2)
# is mounted FILE BY FILE and TWICE EACH — once where the image looks for XRT,
# once at its own path, because those libraries find each other at their install
# path; the vendor forbids mounting an /opt/xilinx/xrt of symlinks into the
# system dir, which dangle inside the container.
FRANK_DRAGOON_DEV_ACCEL=/dev/accel/accel0
FRANK_DRAGOON_XRT_DIR=/usr/lib/x86_64-linux-gnu
# sysfs root the fabric-clock preflight reads (same knob the vendor's
# halogen-fabric-clock status reads; the seam points it at a fixture).
FRANK_DRAGOON_SYSFS=/sys
# Container network. Empty = podman's rootless default (`pasta`): a private
# netns that nothing else can join — podman refuses a second network while the
# primary is pasta, the published port is refused from dankclaw_default
# containers, and host.containers.internal resolves link-local, which
# open-notebook's URL guard refuses to store. Named = the app bridge, so
# open-notebook can dial http://frank-dragoon-halogen:8731 for the NPU
# endpoints. The engine's unauthenticated 8730 stays private either way (it
# binds the container's own loopback); only the API port becomes name-resolvable.
FRANK_DRAGOON_NETWORK=dankclaw_default
+21
View File
@@ -27,6 +27,16 @@
# below stop describing the process you are running. Nothing in this fleet
# pins this container; that is a decision, not an oversight.
#
# THE DEVICE LAW (since the 2026-10-19 Vulkan cutover). Synthesis computes on
# the gfx1151 iGPU — the SAME card frank-dragoon serves. It is a shared device,
# never a dedicated one, so two numbers matter and neither is what `rocm-smi`
# prints first: the memory the container may use is the RADV heap inside the
# GTT-backed aperture (the ~2 GiB DRM carve-out is not the working set), and the
# timing that matters is dragoon's, measured WITH a synthesis in flight. The
# rollback is one line: set QWENTTS_DEVICE empty, and the entrypoint passes no
# --device at all — the pre-iGPU CPU shape, no rebuild, no restart of anything
# else on the box.
#
# THE REGISTRY LAW (why the unit has an ExecStartPost at all). Registrations live
# in memory only, so a restart loses them, and re-registration alone does NOT warm
# a voice — the warm pass has to synthesize something. Without it the façade
@@ -103,6 +113,17 @@ QWENTTS_LANG=auto
# round-trip outside it) is what absorbs the queue.
QWENTTS_MAX_BATCH=1
# The compute device, by ggml device NAME. `Vulkan0` is the gfx1151 iGPU: the
# image carries the Mesa (RADV) ICD and no NVIDIA ICD, so this container's
# Vulkan enumeration has exactly one entry and it is the iGPU. The unit hands
# the container /dev/dri and keeps the render group; on this hybrid host the
# iGPU's render node is renderD129 and renderD128 is the NVIDIA card, which is
# why the unit passes the directory and lets the driver resolve the node.
#
# ROLLBACK: an empty value means no --device flag at all — the CPU shape this
# endpoint shipped with until 2026-10-19. One line, no rebuild.
QWENTTS_DEVICE=Vulkan0
# --- the boot warmer (dankclaw/voiceservice/tts-engine-warm.sh) --------------
# These three are read by the warm hook, not by tts-server. They live here
# because the hook is exec-ed into this container and therefore inherits this
+2 -2
View File
@@ -7,7 +7,7 @@
# whichever TTS engine wins the engine decision (dankclaw/voiceservice/app.py
# is a single stdlib-only file in a locally built image). Consumers — omp's
# TTS device, OpenWebUI, open-notebook — address a VOICE BY NAME
# (voice=narrator1, or the harness' voice_id=narrator1) and never see a path:
# (voice=wilwheaton, or the harness' voice_id=wilwheaton) and never see a path:
# a name is real exactly when /voices/<name>/reference.wav exists, and
# anything else is a 400 that lists the names that are.
#
@@ -65,7 +65,7 @@ TTS_UPSTREAM_MODEL=voxcpm2
# name still 400s with the list that exists — this covers the absent case only —
# and the journal marks such a line `(default)` so an unrequested voice never
# reads as a requested one. Any pack directory in /home/dank/voices is legal.
DEFAULT_VOICE=narrator1
DEFAULT_VOICE=wilwheaton
# CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not
# configured": the endpoint answers a 500 that says so instead of running
+13 -1
View File
@@ -1,5 +1,6 @@
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file.
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
# the refusal-direction adapter applied at load.
#
# ONE OWNER: shape, payload identity, eras, laws and every measured number
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
@@ -13,6 +14,17 @@ MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.g
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
# under concurrent image requests, which is exactly --parallel 3.
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
# Refusal-direction adapter (a rank-1 LoRA, 3.2 MB) applied at load; it edits
# the compute graph, it is not a second model. Scale 1 is the card's "held-out
# refusals 0 %" point. LORA_FILE="" is the documented opt-out — the plain pack
# again, no rebuild — and LORA_SCALE=0 is the same result through the flag.
# PROVENANCE ASYMMETRY: the adapter's own prism.abliterate.base_gguf names
# Ternary-Bonsai-2-9B-PTQ1_0.gguf as the pack its direction was fitted against,
# which is NOT this packing and is not on this host. The tensor names match, so
# it applies; the strength was calibrated on a sibling quantization, so the
# effect is proven by the endpoint's behaviour, never by the metadata.
LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf
LORA_SCALE=1
# Served id — the harness/family name, not a fact about the payload.
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
ALIAS=frank-zealot
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 2x262144 slots over one 524288 pool, unquantized f16 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF + refusal adapter (qwen35 hybrid, 3x200704 slots over one 602112 pool, f16 K + q8_0 V, MTP armed, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
After=network-online.target
Wants=network-online.target
+6 -1
View File
@@ -107,6 +107,11 @@ def sync_subindex(index: pathlib.Path, vault: pathlib.Path):
lines = index.read_text().split("\n")
out, dropped, refreshed = [], 0, 0
for line in lines:
# A row whose first cell is empty is a description with no page: the link
# was deleted when the page was destroyed, so the row is a dead promise.
if re.match(r"^\|\s*\|", line.strip()):
dropped += 1
continue
for pat, wrap in ((ROW, "| [[{}]] |{} {} |"), (BULLET, "- [[{}]] — {}")):
m = pat.match(line)
if not m:
@@ -191,7 +196,7 @@ def main(argv):
index.write_text("\n".join(out) + "\n")
touched = 0
for sub in sorted(vault.rglob("index.md")):
if sub == index:
if sub == index or ".AGENTTRASH" in str(sub):
continue
dropped, refreshed, added = sync_subindex(sub, vault)
if dropped or refreshed or added:
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env python3
"""gpu-exporter: unified per-GPU metrics for a mixed AMD/NVIDIA host.
Emits the `gpu_*` Prometheus families the host-system dashboard graphs:
gpu_utilization_percent gpu_memory_used_bytes gpu_memory_total_bytes
gpu_temperature_celsius gpu_power_watts
each labelled {gpu, vendor, model}. The scrape job adds `host`.
Two sources, both host-native — no container, no GPU passthrough:
* AMD (R9700s, the 8060S iGPU, the Raphael display GPU): the amdgpu sysfs tree.
Utilization comes ONLY from gpu_busy_percent — amd-smi reports gfx_busy_inst
as N/A on gfx1201 (verified 2026-10-20). VRAM is already bytes, temperature
is milli-degrees, power is micro-watts. Crucially power1_average is EMPTY on
the R9700s, so those cards legitimately have no gpu_power_watts line; a
missing file becomes an ABSENT metric, never a fabricated 0 that a dashboard
would plot as a real reading.
* NVIDIA (the RTX 3090 on Dankclaw): one `nvidia-smi --query-gpu` CSV. On
Dankserver there is an nvidia-smi binary but no NVIDIA GPU, so the command
fails; that is tolerated and simply yields no NVIDIA series.
Default mode prints one scrape to stdout and exits (the test harness, and any
`service ... --oneshot` one-off). `--serve` binds the HTTP endpoint for
Prometheus; it binds EXPLICITLY (loopback + the host LAN address), never a
wildcard, matching node-exporter's posture on these boxes.
"""
import argparse
import glob
import os
import subprocess
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
FAMILIES = [
("gpu_utilization_percent", "gauge", "GPU busy time as a percentage of wall time."),
("gpu_memory_used_bytes", "gauge", "Device memory in use, in bytes."),
("gpu_memory_total_bytes", "gauge", "Device memory capacity, in bytes."),
("gpu_temperature_celsius", "gauge", "Package temperature in degrees Celsius."),
("gpu_power_watts", "gauge", "Board power draw in watts."),
]
DEFAULT_NVIDIA_CMD = (
"nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,"
"memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits"
)
def _read(path):
try:
with open(path) as fh:
return fh.read().strip()
except OSError:
return ""
def _pci_slot(device_dir):
uevent = _read(os.path.join(device_dir, "uevent"))
for line in uevent.splitlines():
if line.startswith("PCI_SLOT_NAME="):
return line.split("=", 1)[1]
return ""
def _pci_id(device_dir):
uevent = _read(os.path.join(device_dir, "uevent"))
for line in uevent.splitlines():
if line.startswith("PCI_ID="):
return "amd:" + line.split("=", 1)[1].lower()
return "amd"
def collect_amd(sys_dir):
for card in sorted(glob.glob(os.path.join(sys_dir, "card[0-9]*"))):
device = os.path.join(card, "device")
busy = _read(os.path.join(device, "gpu_busy_percent"))
if busy == "":
continue # not an amdgpu compute node (e.g. a render node w/o metrics)
labels = {"gpu": _pci_slot(device) or os.path.basename(card),
"vendor": "amd", "model": _pci_id(device)}
yield "gpu_utilization_percent", labels, busy
for name, leaf in (("gpu_memory_used_bytes", "mem_info_vram_used"),
("gpu_memory_total_bytes", "mem_info_vram_total")):
v = _read(os.path.join(device, leaf))
if v != "":
yield name, labels, v
temp = ""
for hwmon in sorted(glob.glob(os.path.join(device, "hwmon", "hwmon*", "temp1_input"))):
temp = _read(hwmon)
if temp != "":
break
if temp != "":
yield "gpu_temperature_celsius", labels, str(int(temp) / 1000.0)
power = _read(os.path.join(device, "power1_average"))
if power != "": # empty on the R9700s -> metric absent, not zero
yield "gpu_power_watts", labels, str(int(power) / 1e6)
def collect_nvidia(cmd):
if not cmd:
return
try:
out = subprocess.run(cmd, shell=True, capture_output=True, text=True,
timeout=10, check=True).stdout
except (subprocess.SubprocessError, OSError):
return # no NVIDIA GPU / no driver: tolerate, emit nothing
for line in out.splitlines():
parts = [p.strip() for p in line.split(",")]
if len(parts) < 7:
continue
idx, name, util, mused, mtotal, temp, power = parts[:7]
labels = {"gpu": idx, "vendor": "nvidia", "model": name}
yield "gpu_utilization_percent", labels, util
yield "gpu_memory_used_bytes", labels, str(int(float(mused)) * 1048576)
yield "gpu_memory_total_bytes", labels, str(int(float(mtotal)) * 1048576)
if temp not in ("", "[N/A]"):
yield "gpu_temperature_celsius", labels, temp
if power not in ("", "[N/A]"):
yield "gpu_power_watts", labels, power
def collect(sys_dir, nvidia_cmd):
samples = {}
for source in (collect_amd(sys_dir), collect_nvidia(nvidia_cmd)):
for name, labels, value in source:
try:
value = str(float(value)) # normalise; drop trailing junk
except ValueError:
continue
samples.setdefault(name, []).append((labels, value))
return samples
def _fmt(value):
f = float(value)
if f == int(f) and abs(f) < 1e15:
return str(int(f))
return repr(f)
def render(samples):
out = []
for name, kind, help_text in FAMILIES:
out.append("# HELP %s %s" % (name, help_text))
out.append("# TYPE %s %s" % (name, kind))
for labels, value in sorted(samples.get(name, []), key=lambda x: sorted(x[0].items())):
lset = ",".join('%s="%s"' % (k, v) for k, v in sorted(labels.items()))
out.append("%s{%s} %s" % (name, lset, _fmt(value)))
return "\n".join(out) + "\n"
class Handler(BaseHTTPRequestHandler):
sys_dir = "/sys/class/drm"
nvidia_cmd = DEFAULT_NVIDIA_CMD
def do_GET(self): # noqa: N802 (http.server API)
if self.path.split("?")[0] not in ("/metrics", "/"):
self.send_error(404)
return
body = render(collect(self.sys_dir, self.nvidia_cmd)).encode()
self.send_response(200)
self.send_header("Content-Type", "text/plain; version=0.0.4")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
def main(argv):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--amdgpu-sysfs", default="/sys/class/drm")
ap.add_argument("--nvidia-cmd", default=DEFAULT_NVIDIA_CMD)
ap.add_argument("--serve", action="store_true", help="serve /metrics instead of one-shot")
ap.add_argument("--port", type=int, default=9401)
ap.add_argument("--bind", action="append", default=None,
help="explicit bind address (repeatable); env GPU_EXPORTER_BINDS "
"comma-list if unset. NEVER a wildcard.")
args = ap.parse_args(argv)
if not args.serve:
sys.stdout.write(render(collect(args.amdgpu_sysfs, args.nvidia_cmd)))
return 0
binds = args.bind
if not binds:
env = os.environ.get("GPU_EXPORTER_BINDS", "")
binds = [b.strip() for b in env.split(",") if b.strip()] or ["127.0.0.1"]
Handler.sys_dir = args.amdgpu_sysfs
Handler.nvidia_cmd = args.nvidia_cmd
servers = []
for b in binds:
if b in ("0.0.0.0", "::", "*"):
sys.stderr.write("refusing wildcard bind %s\n" % b)
return 2
servers.append(ThreadingHTTPServer((b, args.port), Handler))
sys.stderr.write("gpu-exporter serving :%d on %s\n" % (args.port, ",".join(binds)))
servers[0].serve_forever()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+35
View File
@@ -0,0 +1,35 @@
# Systemd USER unit: gpu-exporter (both Dankclaw and Dankserver).
#
# Install path: ~/.config/systemd/user/gpu-exporter.service
# Installed and enabled by deploy_gpu_exporter in scripts/deploy-remote.sh.
#
# Per-GPU metrics (utilization / VRAM / temperature / power) for the mixed
# AMD + NVIDIA fleet, scraped by Prometheus. Like signal-relay it is a plain
# user unit, not a quadlet: it runs python3 straight from the checkout (a pull
# updates it with no copy step), a hand-written user unit can be enabled where
# the podman v1 generator's units cannot, and with Linger=yes for dank it starts
# at boot. It reads the host's own cgroup/drm sysfs and shells to nvidia-smi, so
# it must run on the host — a container would need GPU/DRM passthrough for
# nothing gained.
#
# BINDING. The base unit serves loopback only. deploy_gpu_exporter writes a
# drop-in adding THIS host's LAN address (the same two-address posture
# node-exporter and cadvisor use), so the endpoint is never a wildcard. It is
# unauthenticated metrics on the LAN, identical to node-exporter :9100 and
# cadvisor :8080 — Prometheus reaches same-host via host.containers.internal and
# the Dankserver instance over the LAN from Dankclaw.
[Unit]
Description=GPU metrics exporter (amdgpu sysfs + nvidia-smi)
After=network-online.target
Wants=network-online.target
[Service]
# Loopback by default; the deploy drop-in appends the host LAN address.
Environment=GPU_EXPORTER_BINDS=127.0.0.1
ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/gpu-exporter.py --serve
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
+27 -17
View File
@@ -1,5 +1,6 @@
# open-notebook.env — configuration for **open-notebook**, the fleet's podcast
# generator on Dankclaw (UI host port **8502**, API host port **5056**). Read by
# generator on Dankclaw (user path https://notebook.franksdank.cloud; UI host
# port **8502**, API host port **5056 loopback-only**). Read by
# BOTH Quadlet units (dankclaw/open-notebook/quadlet/) as the FIRST
# EnvironmentFile= line, which is why every path and host name below is the
# value the CONTAINER must see.
@@ -47,13 +48,17 @@
#
# THE SCRIPTWRITER. ESPERANTO_LLM_TIMEOUT is the app's own default (180 s), kept
# visible beside the TTS number because both belong to one podcast run: the
# outline call and the transcript call are the slow text work, and frank-zealot
# answers them well inside 180. Provider choice is made in the UI, not here —
# see [[Open Notebook]] for the one-time credential provisioning — and it is
# frank-zealot on purpose: script writing asks for a JSON-schema response, which
# the OpenAI-compatible llama.cpp endpoint is known to serve, while halogen's
# structured-output behaviour on frank-dragoon is UNVERIFIED. halogen remains
# the unproven alternative, not the fallback.
# outline call and the transcript call are the slow text work, and the endpoint
# provisioned today answers them well inside 180. The provider itself is chosen
# in the database, not here — [[Open Notebook]] owns that provisioning and the
# address census behind it. Short version: frank-zealot was the plan (script
# writing asks for a JSON-schema response, which llama.cpp serves and halogen's
# behaviour on frank-dragoon is UNVERIFIED), but no Dankclaw LLM is dialable from
# this container's netns, and the single address that reaches one —
# `host.containers.internal`, 169.254.1.2 — the app refuses to store at all,
# because link-local is a blocked address class. The writer is therefore
# frank-archon (`http://192.168.0.235:7770/v1`), the only fleet LLM that both
# answers from inside this netns and passes that check.
#
# THE DATABASE COORDINATES. SURREAL_URL addresses the sibling container BY NAME
# on systemd-open-notebook (container DNS is scoped to the owning network, so
@@ -73,15 +78,20 @@ SURREAL_PASS=
OPEN_NOTEBOOK_ENCRYPTION_KEY=
OPEN_NOTEBOOK_PASSWORD=
# WHERE THE BROWSER FINDS THE API. The UI auto-detects `<host-you-opened>:5055`,
# and 5055 on this host belongs to seerr, so the API leaves the container on 5056
# and this line has to say so. Both origins are listed in CORS_ORIGINS because
# the API is called cross-origin from the UI's own port: the LAN origin is the
# normal one, the localhost one covers a browser opened on Dankclaw itself. The
# app logs a warning at start while CORS_ORIGINS is unset and defaults to `*`,
# which is a wildcard in front of an endpoint that serves every episode's audio.
API_URL=http://192.168.0.143:5056
CORS_ORIGINS=http://192.168.0.143:8502,http://localhost:8502
# WHERE THE BROWSER FINDS THE API. The guarded vhost
# (notebook.franksdank.cloud → Caddy → 8502) is the only user path, and the
# image's own Next.js layer rewrites `/api/*` to the FastAPI running beside it,
# so ONE origin serves UI and API and API_URL has to name that vhost. This is
# not cosmetic: unset, the app auto-detects `<host-you-opened>:5055` — the port
# upstream ships, and the one seerr holds on this host — and hands the browser
# an address nothing answers; pointed at the LAN `:5056` instead, every call
# leaves the guarded origin and an authenticated browser session does not ride
# it. Same-origin traffic needs no CORS at all, but CORS_ORIGINS stays set to
# the one origin because the app warns at start while it is unset and defaults
# to `*` — a wildcard in front of an endpoint that serves every episode's
# audio. No trailing slash: the UI concatenates `${API_URL}/api/...`.
API_URL=https://notebook.franksdank.cloud
CORS_ORIGINS=https://notebook.franksdank.cloud
# Concurrency: see "THE CPU-SHUTDOWN GUARD" above. Defaults are 5 and 5.
TTS_BATCH_SIZE=1
+16 -5
View File
@@ -147,10 +147,21 @@ def main():
deep = corpus(160_000)
wide = corpus(1_100_000)
# 3.28 chars/token is the measured ratio of this corpus through the
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
# deploy); 98.5 % of the window leaves the response room under the ceiling.
chars = int(0.985 * maxlen * 3.28)
# The corpus's chars/token ratio is MEASURED here, not stored. A stored 3.28
# (646,000 chars -> 196,919 tokens, 2026-10-11) had drifted to 3.20 by
# 2026-10-20 (646,299 chars -> 201,876 tokens) as the repo's docs grew, and a
# stale constant sends leg B past --max-model-len, where the engine answers a
# perfectly legitimate 400 ("the request (201900 tokens) exceeds the available
# context size (200000)") and the leg reads as a capacity failure. One small
# request prices the ratio; 97 % of the window leaves the response room.
sample = prompt(wide, 4000, "CAL", "cal-marker")
cal, _, _ = post({"model": MODEL,
"messages": [{"role": "user", "content": sample}],
"max_tokens": 1, "temperature": 0.0,
"chat_template_kwargs": {"enable_thinking": False}})
ratio = len(sample) / cal["usage"]["prompt_tokens"]
print(f"[cal] corpus ratio {ratio:.2f} chars/token")
chars = int(0.97 * maxlen * ratio)
# Each marker's FIRST hyphen-separated word is unique, because show()
# scores the hit on that word — a shared prefix would score every slot HIT
# on any answer that says the word.
@@ -176,7 +187,7 @@ def main():
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
print(f"[B] total {time.time() - t0:.1f}s")
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
dchars = int(0.25 * maxlen * 3.28)
dchars = int(0.25 * maxlen * ratio)
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
for i in range(seqs)]
print(f"[D] {seqs} concurrent streams x 512 tokens, "
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env python3
"""Repair the two artifacts the destruction passes introduced.
1. The link stripper could eat the space after a `- ` marker (`-`X``) and leave a
dangling `see .` where a wikilink used to be.
2. The Eras collapse could swallow a table's HEADER row (it looked like just
another non-current row), leaving a separator with no header above it.
Usage: repair-damage.py [--apply]
"""
import pathlib
import re
import sys
VAULT = pathlib.Path("serverAppConfigs-docs")
ERAS_HEADER = "| Window | Payload | Runtime | Shape | Replaced because | Rollback |"
def main(argv):
apply = "--apply" in argv
marker = dangling = header = 0
for p in sorted(VAULT.rglob("*.md")):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
lines = p.read_text(errors="replace").split("\n")
out = []
# Only a pointer whose target was deleted reads as `… see .` — never touch
# a line merely because it wraps on a comma. (An earlier version of this
# rule stripped the trailing comma off 660 wrapped prose lines, which
# `scripts/restore-punctuation.py` then had to undo from HEAD.)
for line in lines:
new = line
if re.match(r"^(\s*)[-*](?![-* ])[$`[(*_]", new):
new = re.sub(r"^(\s*)[-*]", r"\1- ", new)
marker += 1
if re.search(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", new):
new = re.sub(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", "", new)
dangling += 1
out.append(new)
# A separator row whose line above is not a header gets its header back.
fixed = []
for i, line in enumerate(out):
if re.match(r"^\|-+", line.strip()) and (not fixed or not fixed[-1].strip().startswith("|")):
fixed.append(ERAS_HEADER)
header += 1
fixed.append(line)
if apply and "".join(fixed) != "".join(lines):
p.write_text("\n".join(fixed))
print(f"{'APPLIED' if apply else 'dry run'}: marker spacing {marker}, "
f"dangling pointers {dangling}, restored table headers {header}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Undo trailing-punctuation damage by restoring lines that match HEAD exactly
except for a stripped trailing `,` / `;` / removed trailing word.
The repair pass over-reached: its "dangling pointer" cleanup also matched every
wrapped prose line ending in a comma, and deleted the comma. A line whose text
is otherwise byte-identical to its HEAD version, differing only by trailing
punctuation, is not an edit — it is damage, so HEAD's line goes back.
Usage: restore-punctuation.py [--apply]
"""
import pathlib
import subprocess
import sys
REPO = pathlib.Path(".")
VAULT = REPO / "serverAppConfigs-docs"
def head_text(rel):
r = subprocess.run(["git", "show", f"HEAD:{rel}"], capture_output=True, text=True)
return r.stdout if r.returncode == 0 else None
def main(argv):
apply = "--apply" in argv
restored = 0
pages = 0
for p in sorted(VAULT.rglob("*.md")):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
rel = s
head = head_text(rel)
if head is None:
continue
cur = p.read_text(errors="replace").split("\n")
# Key on (length, prefix) of the HEAD line minus trailing punctuation so a
# damaged line can find its original.
index = {}
for hl in head.split("\n"):
h = hl.rstrip()
if h.endswith((",", ";")) or True:
key = (len(h.rstrip(",;. ")), h.rstrip(",;. ")[:60])
index.setdefault(key, h)
touched = 0
for i, cl in enumerate(cur):
c = cl.rstrip()
if not c or c.endswith((".", ":", "|", "`", ")", "," , ";")):
continue
key = (len(c), c[:60])
hl = index.get(key)
if hl and hl != c and hl.startswith(c):
cur[i] = hl
touched += 1
if touched:
pages += 1
restored += touched
if apply:
p.write_text("\n".join(cur))
print(f"{'APPLIED' if apply else 'dry run'}: {restored} lines restored across {pages} pages")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+32 -30
View File
@@ -15,17 +15,21 @@
# log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize.
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
# own budget table makes claimable, spent on the slab because prefill is
# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
# Deployed shape (2026-10-20, radiance 1.8.1): FOUR decode slots x 200,000 tokens over a
# >= 700,000-token fp8 KV pool, 48 GiB/rank pinned expert pool, MTP depth 2, prefix cache
# in pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab and the KV
# pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair and the budget
# resolver derives it from headroom, the cache bound and the host pool, so the slab
# size and the pool's token count are LEDGER lines. After every boot read them, plus
# the two lines that size a prefill step — "activation arena: ... lends ... while
# steps are at most N tokens" (the arena levels the expert slab borrows) and
# "prefill staging: ... a step of more than N tokens ..." (the chunk size below which
# pooled experts are read synchronously) — and the cache's session backing.
#
# The root unit (scripts/frank-archon.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
# pages, which is ALSO why this script refuses to start when ttm's
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 48 GiB expert pools + the 47.7 GiB PLE
# table + 8 GiB prefix host tier + the checkpoint store (slots x 112 MiB) — ~159 GiB of
# pinned pages, which is ALSO why this script refuses to start when ttm's
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
set -euo pipefail
@@ -44,7 +48,7 @@ set +a
: "${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
: "${FRANK_ARCHON_MODEL_DIR:?FRANK_ARCHON_MODEL_DIR must name the .rad dir}"
: "${FRANK_ARCHON_MODEL_FILE:?FRANK_ARCHON_MODEL_FILE must name the .rad file}"
: "${FRANK_ARCHON_STATE_DIR:?FRANK_ARCHON_STATE_DIR must name the /data state dir}"
: "${FRANK_ARCHON_STATE_DIR:?FRANK_ARCHON_STATE_DIR must name the host dir holding the rad-tune cache}"
: "${FRANK_ARCHON_CONTAINER:?FRANK_ARCHON_CONTAINER must name the container}"
: "${FRANK_ARCHON_SERVED_NAME:?FRANK_ARCHON_SERVED_NAME must name the alias}"
: "${FRANK_ARCHON_PORT:?FRANK_ARCHON_PORT must be the port}"
@@ -56,13 +60,11 @@ set +a
: "${FRANK_ARCHON_PLACEMENT:?FRANK_ARCHON_PLACEMENT must be set}"
: "${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
: "${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
: "${FRANK_ARCHON_VRAM_WEIGHTS_MIB:?FRANK_ARCHON_VRAM_WEIGHTS_MIB must be set}"
: "${FRANK_ARCHON_VRAM_KV_MIB:?FRANK_ARCHON_VRAM_KV_MIB must be set}"
: "${FRANK_ARCHON_BUSY_PREFILL_CHUNK:?FRANK_ARCHON_BUSY_PREFILL_CHUNK must be set (auto|off|N)}"
: "${FRANK_ARCHON_KV_DTYPE:?FRANK_ARCHON_KV_DTYPE must be set}"
: "${FRANK_ARCHON_SPEC_TOKENS:?FRANK_ARCHON_SPEC_TOKENS must be set}"
: "${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
: "${FRANK_ARCHON_PREFIX_HOST_MIB:?FRANK_ARCHON_PREFIX_HOST_MIB must be set}"
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
: "${FRANK_ARCHON_PREFIX_CACHE_MIB:?FRANK_ARCHON_PREFIX_CACHE_MIB must be set}"
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
@@ -103,19 +105,12 @@ if [ ! -f "$FRANK_ARCHON_TEMPLATE" ]; then
exit 1
fi
mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
# The engine's create is NOT recursive: with only the state dir mounted, its
# "cannot create /data/kvcache/frank-archon: No such file or directory"
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
# the full path on the host so the tier opens on first boot.
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
# the cache must ride a host mount or it dies with the --rm container
# that wrote it. The dir is pre-created for the same reason as kvcache
# (the engine's create is not recursive).
# that wrote it. The engine's create is not recursive, so the dir is pre-created.
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
@@ -148,6 +143,15 @@ done
LOG=/home/dank/logs/frank_archon.log
mkdir -p "$(dirname "$LOG")"
# Interleave is opt-in: 0 is the engine's default and passing it anyway would put
# a flag in argv that changes nothing but the record. The sharing cap is always
# explicit because `auto` vs `off` is the difference between decode waiting one
# chunk and waiting the whole step.
PREFILL_ARGS=()
if [ "${FRANK_ARCHON_PREFILL_INTERLEAVE:-0}" -gt 0 ]; then
PREFILL_ARGS=(--prefill-interleave "$FRANK_ARCHON_PREFILL_INTERLEAVE")
fi
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
# --network host (the server binds the host's addresses directly, no NAT),
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
@@ -156,8 +160,9 @@ mkdir -p "$(dirname "$LOG")"
# makes for host memory), memlock unbounded (the pinned ledger — legal
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
# The template rides a single-file mount; the .rad dir is ro; /data holds
# the prefix-cache disk tier.
# The template rides a single-file mount and the .rad dir is ro. The prefix cache
# has no disk tier any more, so the only host state this container writes is the
# rad-tune cache under /opt/radiance/share/radiance/tune.
podman run -d --name "$FRANK_ARCHON_CONTAINER" \
--network host \
--init \
@@ -165,7 +170,6 @@ podman run -d --name "$FRANK_ARCHON_CONTAINER" \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
-v "$FRANK_ARCHON_STATE_DIR":/data \
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_ARCHON_IMAGE" \
@@ -179,15 +183,13 @@ podman run -d --name "$FRANK_ARCHON_CONTAINER" \
--placement "$FRANK_ARCHON_PLACEMENT" \
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
--prefix-cache-dir /data/kvcache/frank-archon \
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
--prefix-cache-mib "$FRANK_ARCHON_PREFIX_CACHE_MIB" \
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
--busy-prefill-chunk "$FRANK_ARCHON_BUSY_PREFILL_CHUNK" \
"${PREFILL_ARGS[@]}" \
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
> "$LOG" 2>&1
+120
View File
@@ -27,6 +27,11 @@
# the flash-server line reads it; it ships as a container; the root unit
# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
# exactly like frank-archon's.
# Since 2026-10-19 the container also carries the XDNA2 NPU lane
# (HALOGEN_NPU_MODELS): the vendor's precompiled AIE programs for the models
# named there run on the NPU inside THIS container and answer /v1/embeddings,
# beside the Flash model, on the same port. Opt-in — the variable empty is the
# pre-lane endpoint, byte for byte.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -115,6 +120,119 @@ if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
echo "frank-dragoon: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
exit 1
fi
# --- NPU lane (XDNA2), opt-in via HALOGEN_NPU_MODELS ------------------------
# Empty = the endpoint exactly as it ran before the lane existed: no device, no
# extra mounts, no gate below. Set = three host preconditions, each refused BY
# NAME (the 2026-10-01 law: the serve gate and the deployer gate must agree):
# 1. the staged model files under $FRANK_DRAGOON_NPU_DIR — a path that MUST
# live INSIDE $FRANK_DRAGOON_MODEL_DIR, because the container's only
# handle on them is the read-only /models mount. A nested bind at
# /models/npu cannot work: runc has to mkdir its mountpoint and the parent
# bind is ro (`mkdirat .../merged/models/npu: read-only file system`,
# measured 2026-10-19), and the image ships no /models at all for it to
# land on. `$MODEL_DIR/npu/<id>` is visible at /models/npu/<id> through
# the existing mount, which is also where the vendor's catalog expects it.
# The engine re-checks size+sha256 against that catalog at boot; it cannot
# fetch anything here because /models is read-only and HALOGEN_DOWNLOAD
# stays unset — no outbound connection at boot),
# 2. the distro XRT's three libraries, each mounted twice (where the image
# looks for XRT, and at its own path — the libraries find each other there;
# an /opt/xilinx/xrt of symlinks into the system dir dangles inside the
# container, which the vendor calls out explicitly),
# 3. the GPU fabric clock HELD: on Strix Halo the iGPU and the NPU co-running
# hang the machine while FCLK changes speed, and the engine refuses to
# start when it moves — it cannot read host sysfs from inside the
# container, so the proof lives here, checked exactly the way the vendor's
# `halogen-fabric-clock status` checks it.
NPU_ARGS=()
NPU_MODELS="${HALOGEN_NPU_MODELS:-}"
if [ -n "$NPU_MODELS" ]; then
DEV_ACCEL="${FRANK_DRAGOON_DEV_ACCEL:-/dev/accel/accel0}"
NPU_DIR="${FRANK_DRAGOON_NPU_DIR:?HALOGEN_NPU_MODELS is set, so FRANK_DRAGOON_NPU_DIR must name the staged model dir}"
XRT_DIR="${FRANK_DRAGOON_XRT_DIR:-/usr/lib/x86_64-linux-gnu}"
SYSFS="${FRANK_DRAGOON_SYSFS:-/sys}"
# The lane's files have exactly one door into the container: the read-only
# payload mount. A staging dir OUTSIDE it would need a nested bind whose
# mountpoint runc cannot create through that read-only parent, so refuse the
# shape by name here rather than let runc die at container init (2026-10-19).
case "$NPU_DIR/" in
"$FRANK_DRAGOON_MODEL_DIR"/*) ;;
*) echo "frank-dragoon: FRANK_DRAGOON_NPU_DIR=$NPU_DIR is outside the payload dir $FRANK_DRAGOON_MODEL_DIR — the lane's files reach the container only through the read-only /models mount; move the staging dir inside it" >&2
exit 1 ;;
esac
if [ ! -e "$DEV_ACCEL" ]; then
echo "frank-dragoon: $DEV_ACCEL missing — the NPU lane needs the amdxdna node (is iommu=pt still on the boot line?)" >&2
exit 1
fi
# Existence is not access. This endpoint runs rootless, so the container's
# namespace maps exactly one uid/gid (dank) and a root:render 0660 node reads
# `nobody:nogroup` inside it — the engine then dies naming the node itself
# (measured 2026-10-19). --group-add keep-groups cannot help: an unmapped gid
# cannot be granted into a namespace that does not map it, verified with and
# without the flag from a shell already in `render`. The door is the udev mode.
# NB: `[ -o FILE ]` is NOT "other-writable" — it means "exists", and `-o` is
# also the OR operator inside test — so the permission bit is read directly.
if [ $(( 8#$(stat -c '%a' "$DEV_ACCEL") & 8#2 )) -eq 0 ]; then
echo "frank-dragoon: $DEV_ACCEL is $(stat -c '%a %U:%G' "$DEV_ACCEL") — a rootless container cannot open it; install dankclaw/udev/98-halogen-npu-access.rules and re-trigger udev (sudo udevadm control --reload-rules && sudo udevadm trigger --sysname-match=accel0)" >&2
exit 1
fi
for M in ${NPU_MODELS//,/ }; do
if [ ! -d "$NPU_DIR/$M" ]; then
echo "frank-dragoon: NPU model $M not staged in $NPU_DIR;" >&2
echo " fetch it: bash scripts/download-frank-dragoon-npu.sh" >&2
exit 1
fi
done
for L in libxrt_coreutil.so.2 libxrt_core.so.2 libxrt_driver_xdna.so.2; do
if [ ! -e "$XRT_DIR/$L" ]; then
echo "frank-dragoon: XRT NPU library $XRT_DIR/$L missing — the NPU lane cannot load (Ubuntu libxrt2 + libxrt-npu2)" >&2
exit 1
fi
REAL="$(readlink -f "$XRT_DIR/$L")"
NPU_ARGS+=(-v "$REAL:/opt/xilinx/xrt/lib/$L:ro" -v "$REAL:$XRT_DIR/$L:ro")
done
HELD=0
for D in "$SYSFS"/class/drm/card*/device; do
[ -e "$D/pp_dpm_fclk" ] || continue
[ -e "$D/power_dpm_force_performance_level" ] || continue
PERF="$(cat "$D/power_dpm_force_performance_level" 2>/dev/null || true)"
TOP="$(grep . "$D/pp_dpm_fclk" | tail -n 1)"
if [ "$PERF" = high ] || { [ "$PERF" = manual ] \
&& [ "$(grep -c '\*' "$D/pp_dpm_fclk")" = 1 ] && [ "${TOP%\*}" != "$TOP" ]; }; then
HELD=1
else
echo "frank-dragoon: fabric clock at $D is '$PERF', not held — the NPU lane refuses to start (GPU+NPU together need FCLK pinned)" >&2
echo " hold it: sudo halogen-fabric-clock hold" >&2
exit 1
fi
done
if [ "$HELD" = 0 ]; then
echo "frank-dragoon: no amdgpu device with a fabric-clock control under $SYSFS/class/drm — cannot prove FCLK is held" >&2
exit 1
fi
NPU_ARGS+=(--device "$DEV_ACCEL" \
-e HALOGEN_NPU_MODELS="$NPU_MODELS")
echo "frank-dragoon: NPU lane armed: $NPU_MODELS on $DEV_ACCEL (fabric clock held)" >&2
fi
# Container network. Empty = podman's rootless default (`pasta`), which is a
# private netns and CANNOT be joined by anything else: podman refuses a second
# network while the primary is pasta ("cannot set multiple networks without
# bridge network mode"), the endpoint's published port answers 200 on the host
# and on the LAN but is REFUSED from a dankclaw_default container (measured
# 2026-10-19), and `host.containers.internal:7770` answers 200 yet resolves to
# 169.254.1.2 — link-local, which open-notebook's URL guard refuses to store.
# So an endpoint on pasta has no address an app container can use; naming the
# app bridge here is the only shape that lets open-notebook dial
# http://frank-dragoon-halogen:8731 for the NPU endpoints. It does not widen
# the surface: the engine's unauthenticated 8730 is bound to the container's
# OWN loopback (/proc/net/tcp: 127.0.0.1:8730 vs 0.0.0.0:8731, checked the same
# day), so only the API port is reachable by name.
NET_ARGS=()
if [ -n "${FRANK_DRAGOON_NETWORK:-}" ]; then
NET_ARGS=(--network "$FRANK_DRAGOON_NETWORK")
fi
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
# The engine refuses to pin when MemAvailable is below the checkpoint plus
# a 16 GiB floor: pinned pages cannot be swapped, and under the floor the
@@ -189,6 +307,8 @@ podman run --name "$FRANK_DRAGOON_CONTAINER" \
${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
-v "$FRANK_DRAGOON_MODEL_DIR:/models:ro" \
-v "$FRANK_DRAGOON_MODEL_DIR/tokenizer:/tokenizer:ro" \
"${NET_ARGS[@]}" \
"${NPU_ARGS[@]}" \
"$FRANK_DRAGOON_IMAGE" \
> "$LOG" 2>&1
+31
View File
@@ -33,6 +33,19 @@
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
# opt-out.
#
# A refusal-direction adapter rides on top of the pack (Frank, 2026-10-19):
# AtomicChat's rank-1 LoRA, applied at load by --lora-scaled. It is an edit
# inside the compute graph, not a second model — B = -r (the refusal direction),
# A built from the base pack's folded input basis, every transformer layer, the
# embedding deliberately untouched because an embedding LoRA crashes this fork's
# graph. The card verified it on prism-b10777-e3de715 — the build pinned here —
# and says older forks refuse the pack outright (`unsupported
# prism.hadamard.version: 2`), which makes the runtime lock doubly load-bearing.
# Its own metadata names the PTQ1_0 packing as the pack the direction was fitted
# against, so what proves the effect is a refusal probe against the live
# endpoint, not the metadata and not the card's table. LORA_FILE="" drops the
# flag and the endpoint is the plain pack again.
#
# Speculative decoding ships ON (Frank, 2026-10-19). The head is IN the file and
# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
# what made it lose that evening was the bill, not the drafts — the draft context
@@ -136,6 +149,13 @@ fi
# Vision projector. No colon on purpose: an explicitly empty value means
# "text-only, on purpose" and must NOT be replaced by the default path.
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
# Refusal adapter. No colon on purpose, the same convention as MMPROJ_FILE: an
# explicitly empty LORA_FILE is the documented opt-out (the plain pack again)
# and must NOT be replaced by the default path. The scale travels INSIDE the
# same argv token as the file, so LORA_SCALE only matters while LORA_FILE names
# something; 0 is the card's "switch it off without touching the file" value.
: "${LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf}"
: "${LORA_SCALE:=1}"
# Fail loudly rather than half-loading: a missing file is a download that
# has not finished, and Restart=always would turn it into a crash loop that
@@ -149,6 +169,17 @@ if [[ -n "$MMPROJ_FILE" ]]; then
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
fi
# The adapter reaches the engine as ONE token — --lora-scaled <file>:<scale> —
# so a scale can never arrive without its file. It is applied once, at load;
# nothing here hot-swaps it (--lora-init-without-apply plus POST /lora-adapters
# is that shape, and a process holding three conversations is not the place to
# re-parameterise weights mid-flight). A missing file is a fetch that never
# finished, so refuse rather than let Restart=always loop on it.
if [[ -n "$LORA_FILE" ]]; then
[[ -e "$LORA_FILE" ]] || { echo "frank-zealot: missing $LORA_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--lora-scaled "${LORA_FILE}:${LORA_SCALE}")
fi
# Prometheus scraping is opt-IN in llama.cpp: without --metrics the server
# answers 501 on /metrics, and the fleet scrape then reports the endpoint DOWN.
if [[ "$METRICS" = "1" ]]; then
+8
View File
@@ -293,6 +293,14 @@ for path in "${CHANGED[@]}"; do
add_test tests/caddyfile.bats
add_test tests/deployment_scoping.bats
;;
# The GPU exporter is a host process; its seam pins the collector, the
# user unit, the deployer function, the planner route, the Prometheus
# scrape jobs and the host-system GPU panels — a cross-file contract, so
# the whole seam rides on any of its own files.
scripts/gpu-exporter.py|scripts/gpu-exporter.service)
add_test tests/gpu_exporter.bats
add_test tests/deployment_scoping.bats
;;
dankclaw/udev/*)
# Host-level udev rules are applied by manual sudo install + udevadm
# trigger, not dispatched as a service; nothing here is bats-testable.
@@ -21,7 +21,7 @@ Damage catalogue:
no longer frontmatter (title/tags/description become body text).
2. A rewrite **appended** instead of merged → duplicate headings and a stray
second `---` mid-file.
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`
`+- `, `- - `.
4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
5. Line-range replacement that dropped a sentence tail → dangling fragment,
@@ -46,7 +46,7 @@ Repair order (matters):
2. If it merely duplicated: **salvage unique content first** — cut the
one-of-a-kind section, paste it into the copy you keep, THEN delete the
duplicate ranges. Never delete first.
3. Re-add missing sentence tails (insert before the orphaned fragment),
3. Re-add missing sentence tails (insert before the orphaned fragment)
keeping list continuation lines indented.
4. Re-run all checks above.
@@ -57,7 +57,7 @@ safe, but its snapshot tag changes on every edit — re-read before each
subsequent edit. `.AGENTTRASH/` is gitignored (`.gitignore:25`), so NEW trash
stays local — but do not read that as "the repo has none of it": on 2026-10-18
`git ls-files | grep -c AGENTTRASH` returns **37** tracked paths
(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`,
(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`
`tests/.AGENTTRASH/`), because gitignore never untracks what was committed
before the rule. Trash is a temporary hold, not an archive.
@@ -0,0 +1,55 @@
---
type: reference
title: Domain Docs
description: "Consumer rules for GLOSSARY.md and docs/adr/, and where this vault out-ranks them on operational facts."
tags: [agent, glossary, adr, documentation]
timestamp: 2026-10-20T00:00:00Z
---
# Domain Docs
How the engineering skills (`/grill-with-docs`, `/domain-modeling`,
`/improve-codebase-architecture`, `/to-spec`) should consume this repo's design
vocabulary when exploring the codebase.
## Before exploring, read these
- **`GLOSSARY.md`** at the repo root
- **`docs/adr/`**: read ADRs that touch the area you're about to work in
If either doesn't exist, **proceed silently**. Don't flag their absence; don't
suggest creating them upfront. `/domain-modeling` (reached via
`/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily
when terms or decisions actually get resolved. As of 2026-10-20 neither exists;
this page is the pointer, not a promise.
## Ownership against this vault
This repo already carries `serverAppConfigs-docs/` — the operational knowledge
base and, per repo law, the owner of operational facts (endpoint shapes, Eras,
Laws, runbooks). `GLOSSARY.md` / `docs/adr/` hold **design vocabulary and
decisions**. Where a term is defined in both, the vault page that owns the fact
wins — one fact, one owner. A glossary entry restating an endpoint number is
the bug, not the second source.
## Use the glossary's vocabulary
When output names a domain concept (an issue title, a refactor proposal, a
hypothesis, a test name), use the term as defined in `GLOSSARY.md`. Don't drift
to synonyms the glossary explicitly avoids.
If the concept you need isn't in the glossary yet, that's a signal: either
you're inventing language the project doesn't use (reconsider) or there's a
real gap (note it for `/domain-modeling`).
## Flag ADR conflicts
If output contradicts an existing ADR, surface it explicitly rather than
silently overriding:
> _Contradicts ADR-0007 (…), but worth reopening because…_
## Related
- [[Issue Tracker]] — where specs and tickets built on this vocabulary are published.
- [[Agent Docs and Edits]] — the edit-safety rules that apply while touching any doc in this checkout.
@@ -0,0 +1,85 @@
---
type: reference
title: Issue Tracker
description: "How agent skills read and write Gitea issues here: credential-safe curl, label IDs, and wayfinding without sub-issues."
tags: [agent, gitea, issues, triage, api]
timestamp: 2026-10-20T00:00:00Z
---
# Issue Tracker
Issues and specs for this repo live as Gitea issues in `dank/serverAppConfigs`
on `http://192.168.0.143:3000` (Dankclaw). There is no `gh`/`glab` CLI here —
the door is the GitHub-compatible REST API driven with `curl`. The
mattpocock-lineage skills (`/triage`, `/to-tickets`, `/to-spec`, `/wayfinder`)
read this page as their tracker contract. The git workflow around issues and
PRs (branches, PR-only merges, attribution footer) is owned by
[[Deployment Flow]]; this page owns only the API shape.
## Auth
The token lives in the git credential file `~/.config/secrets/gitea`: one
line in git-credential URL form, `http://user:TOKEN@192.168.0.143`. Extract it
without echoing:
```bash
tok=$(sed -nE 's#^http://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)
```
**A token on a command line is world-readable in `/proc/<pid>/cmdline`** (the
same leak class AGENTS.md records for HF tokens). Never pass it with `-H` —
put it in a mode-600 curl config and pass `-K`:
```bash
cfg=$(mktemp); chmod 600 "$cfg"
printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
# ... every curl below uses -K "$cfg"; rm -f "$cfg" when done
```
**GETs also accept HTTP Basic; writes do not** — a POST/PATCH with Basic auth
is refused with `422 {"message":"token is required"}` (measured 2026-10-20).
Use the `Authorization: token` header for everything.
## Conventions
`$API=http://192.168.0.143:3000/api/v1/repos/dank/serverAppConfigs`
- **Create an issue**: `curl -s -K "$cfg" -X POST -H 'Content-Type: application/json' -d "$(jq -n --arg t "$TITLE" --rawfile b "$BODY" '{title:$t, body:$b}')" $API/issues`
- **Read an issue**: `curl -s -K "$cfg" $API/issues/<number>` (fields: `number`, `title`, `body`, `labels[].name`, `state`)
- **List issues**: `curl -s -K "$cfg" "$API/issues?state=open&limit=50"` — add `&labels=<id>` to filter
- **Comment**: `curl -s -K "$cfg" -X POST -d "$(jq -n --rawfile b "$BODY" '{body:$b}')" $API/issues/<number>/comments`
- **Apply labels**: `POST $API/issues/<number>/labels` with `{"labels":[<label-ID numbers>]}` — Gitea wants numeric label **IDs**, not names; resolve them from `GET $API/labels` (see [[Triage Labels]] for which labels exist)
- **Remove a label**: `DELETE $API/issues/<number>/labels/<label-id>`
- **Close**: `PATCH $API/issues/<number>` with `{"state":"closed"}`
- **Blocking relationships**: this instance has no native sub-issues or dependencies — put `Part of #<parent>` / `Blocked by: #<n>, #<n>` lines at the top of the child body. A ticket is unblocked when every listed blocker is closed.
- **Attribution footer (repo law)**: every issue body you open ends with `cwd: <working directory> · session: <omp session id>` — full rule in [[Deployment Flow]].
**PRs as a request surface: no.** Issues and PRs have separate number spaces on
Gitea; a bare `#42` is an issue, a PR is spelled "PR #42".
## When a skill says "publish to the issue tracker"
Create a Gitea issue as above.
## When a skill says "fetch the relevant ticket"
`GET $API/issues/<number>` and read `title` + `body` + `labels[].name` +
comments.
## Wayfinding operations
Used by `/wayfinder`. The **map** is a single issue labelled `wayfinder:map`,
holding the Notes / Decisions-so-far / Fog body.
- **Map**: create an issue, apply the `wayfinder:map` label (create it via `POST $API/labels` `{"name":"wayfinder:map","color":"#6f42c1"}` if missing).
- **Child ticket**: an issue carrying `Part of #<map>` at the top of its body, plus a `wayfinder:<type>` label (`research`/`prototype`/`grilling`/`task`).
- **Blocking**: the `Blocked by:` body line (see Conventions).
- **Frontier**: list open issues (`?state=open`), drop any with an open `Blocked by` target or an assignee; lowest `number` wins.
- **Claim**: `POST $API/issues/<n>/assignees` with `{"assignees":["dank"]}` — actually assign it or it stays in the frontier; the session's first write.
- **Resolve**: post the answer as a comment, `PATCH` to `closed`, then update the map issue's body (GET → edit Decisions-so-far → PATCH) with a context pointer.
## Related
- [[Triage Labels]] — the role strings these operations apply.
- [[Domain Docs]] — what a skill must read before touching the codebase.
- [[SSH and Remote Access]] — reaching the host when an issue needs live evidence.
+11 -11
View File
@@ -16,7 +16,7 @@ this file is repo-managed; edits are harness-side only.
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and
`frank-zealot` (Dankclaw) — engine, port and card behind each id live on
[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`,
[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
files, runbook pages); pages written in the old era name the service,
the harness names the model. The wire now matches: each endpoint's
@@ -26,7 +26,7 @@ picker/grants, the homepage docs and the raw API all use one id per model
note on the endpoint echoing the requested name).
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
192.168.0.143:7771 each answer with the unit name (`frank-archon`,
192.168.0.143:7771 each answer with the unit name (`frank-archon`
`frank-dragoon`, `frank-zealot`).
## Compaction chain arithmetic (verify before ANY window change)
@@ -40,11 +40,11 @@ breaks compaction.
Where the contract lives:
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL:
`http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`),
`methodOrder` (`shake → snapcompact → soft → handoff → remote`,
`http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`)
`methodOrder` (`shake → snapcompact → soft → handoff → remote`
`config.yml:77-82`), `thresholdPercent` (**95**, `config.yml:84`), idle
settings (`idleEnabled`, `idleThresholdTokens: 100000`)
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`
`compactionModel` references
- Implementation: omp plugin
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
@@ -104,16 +104,16 @@ input. Boot-shape OOM laddering is a different problem
## Per-model sampler config (`compat.extraBody`)
`models.yml` has **no per-model sampler keys**. The global knobs are
`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`
`repetitionPenalty`, and a knob reaches the wire only when it is set — the
bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`),
bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`)
and this workstation's `config.yml` sets **none** of them. The 2026-10-05 note
that the unset default is `-1`/"not sent" is retired: there is no `-1`
sentinel, an unset key is simply absent. The per-model hook is
`compat.extraBody`:
**Why it works:** `applyOpenAIExtraBody` runs **last** in the
openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
openai-completions builder — after `applyChatCompletionsCompatPolicy` —
`Object.assign(params, extraBody)`. So extraBody overrides global settings
and compat-policy output.
@@ -191,7 +191,7 @@ assistant turn carrying a long `reasoning_content`, then compare
`usage.prompt_tokens` with the flag true vs false. A large delta means the
field changed template rendering; zero delta means it was ignored.
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`)
never `rm`.
## Static-preamble audit ("the context feels bloated")
@@ -205,7 +205,7 @@ What loads statically every session (measure each, don't guess):
```bash
python3 - <<'EOF'
import glob, re, os
for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
for root in [os.path.expanduser("~/.omp/agent/managed-skills")
os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
n=tot=0
for sk in glob.glob(root+"/*/SKILL.md"):
@@ -305,4 +305,4 @@ standalone; fold go/no-go arithmetic skills into their probe skill
- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
- measuring the endpoints this config routes
@@ -0,0 +1,39 @@
---
type: reference
title: Triage Labels
description: "The five canonical triage roles mapped to this repo's Gitea label strings, plus the numeric-ID quirk."
tags: [agent, gitea, triage, labels]
timestamp: 2026-10-20T00:00:00Z
---
# Triage Labels
The mattpocock-lineage skills speak in terms of five canonical triage roles.
This page maps those roles to the actual label strings used in this repo's
tracker ([[Issue Tracker]] owns the API shape).
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the
corresponding label string from this table. Edit the "Label in our tracker"
column to match whatever vocabulary you actually use.
## Instance quirks (Dankclaw Gitea)
- Labels are applied by **numeric ID**, never by name: `POST /issues/<n>/labels`
with `{"labels":[<id>]}`; resolve names via `GET /labels`.
- The repo also carries the pre-existing `human-only` label from the manual era
(e.g. the podcast-destination issues). `ready-for-human` maps to itself; the
`(HUMAN ONLY)` title prefixes on existing issues stay as they are — the label
is what the skills read.
- An issue with **no** triage label is simply untriaged; absence is not a state.
## Related
- [[Issue Tracker]] — the auth-safe curl shape that applies these labels.
+4 -1
View File
@@ -6,7 +6,10 @@ docs get edited safely, and how the agent's home is deployed.
- [[OMP Harness]] — The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits.
- [[Agent Docs and Edits]] — Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each.
- [[Command Center Dotfiles]] — How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas.
- [[Issue Tracker]] — How agent skills read and write Gitea issues here: credential-safe curl, label IDs, and wayfinding without sub-issues.
- [[Triage Labels]] — The five canonical triage roles mapped to this repo's Gitea label strings, plus the numeric-ID quirk.
- [[Domain Docs]] — Consumer rules for GLOSSARY.md and docs/adr/, and where this vault out-ranks them on operational facts.
## Related
- [LLM](../LLM/index.md) — the fleet these agents operate
- the fleet these agents operate
@@ -8,7 +8,7 @@ tags:
- netavark
- rootless
- dankclaw
timestamp: 2026-09-02
timestamp: 2026-10-19
---
# Container Network Namespaces
@@ -47,7 +47,7 @@ network namespace**:
`Network=` values are read from the live Quadlet sources in
`/home/dank/.config/containers/systemd/*.container`; runtime network names on the
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`,
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`
`systemd-searxng`, `systemd-open-webui`.
Two consequences that are easy to miss:
@@ -83,13 +83,58 @@ podman exec <name> cat /proc/net/route # default route → gateway IP
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?'
```
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`,
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`
`10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns,
published) and `:9119` (its own published port) connect; `:8080` and `:8787`
are **refused instantly** even though both listeners are bound to `0.0.0.0` in
the host netns. Probes to `host.containers.internal` (`169.254.1.2`) from that
network time out rather than refuse — the alias is an address on the *default*
`podman` network, and this container is not on it.
network time out rather than refuse — but see the 2026-10-19 re-measure below,
where the same alias answers from another `dankclaw_default` container.
### 2026-10-19: which published ports a rootless container can actually dial
Probed from inside `open-notebook` (rootless, on `dankclaw_default` **and** the
private `systemd-open-notebook`), each address aimed at a live listener:
| Dial | Result | Mechanism |
|---|---|---|
| `192.168.0.143:3000` (Gitea) | **200** | published by a **rootless** container — rootless egress crosses the host netfilter where netavark installed that DNAT |
| `192.168.0.143:7770` (`frank-dragoon`) | refused | the container is **rootless on podman's default `pasta` netns** (the root unit's `User=dank` runs `podman run`, so it lands in dank's storage). Pasta forwards the published port in the host netns — the host and the LAN both answer 200 — but installs no netavark DNAT for traffic arriving from a rootless bridge, so the same address is refused from a container. This is NOT a root/rootless rule-set split: `podman ps` as `dank` lists it |
| `host.containers.internal:7770` (`169.254.1.2`) | **200** | the pasta forwarder is bound `0.0.0.0` on the host, so the gateway alias reaches it — and the alias is link-local, which is the dead end below |
| `192.168.0.143:7771` (`frank-zealot`) | refused | host-netns listener, unpublished — the rule above, reconfirmed |
| `10.89.1.1:7771` | refused | the bridge gateway address belongs to the rootlesskit netns, not to the host |
| `host.containers.internal:7771` (`169.254.1.2`) | **200** | the gateway alias is the one address that carries a rootless container to a host-netns listener. Why `hermes-dashboard` timed out on this alias in 2026-09-02 while `open-notebook` answers today is **not established**; the difference to check first is the second network and where the default route lands |
| `192.168.0.235:7770` (`frank-archon`) | **200** | a different machine: plain routing, no DNAT in the path |
The consequence that changed designs: **a rootless container's menu of local model
endpoints was only `frank-tts` (container DNS, same network) and `frank-archon` (LAN
address on another host).** A `pasta` endpoint has no address an app container can
use: the LAN address is refused, and the one alias that answers is link-local,
which some applications refuse to store at all ([[Open Notebook]] meets exactly
that refusal). Pasta cannot be *added to* either —
```
podman run --network pasta --network dankclaw_default …
Error: cannot set multiple networks without bridge network mode, selected mode pasta
```
so the only shape that makes such an endpoint nameable is **moving it onto
`dankclaw_default`** (`FRANK_DRAGOON_NETWORK` in `scripts/frank-dragoon.env`,
2026-10-19). For frank-dragoon that is not an exposure change: the engine's
unauthenticated token port binds the container's **own loopback**
(`/proc/net/tcp` on 2026-10-19: `127.0.0.1:8730`, `0.0.0.0:8731`), so only the
API port becomes name-resolvable, which is the same trust the published 7770
already carries on the LAN. `frank-zealot` (7771) stays a host-netns binary and
remains unreachable except through the link-local alias.
The move is verified from the consumer side, so the table above reads as the
**pre-move** census and not as today's menu: inside `open-notebook`,
`getent hosts frank-dragoon-halogen` → `10.89.1.39`, `GET :8731/v1/models`
lists both served models, and the engine's log carries the app's
`POST /v1/embeddings 200` from that address (2026-10-19) — see
[[Open Notebook]] for what runs over it. What the table still owns is the
pasta rule itself: nothing about pasta's forwarding changed, the endpoint was
moved out of it.
## Operational rules
@@ -98,9 +143,13 @@ network time out rather than refuse — the alias is an address on the *default*
host — and never the reverse either: the deployer's health probe runs on the
host, so it sees host-netns listeners that the container itself cannot dial.
- To give a bridge-net container access to a host-netns listener, the options
are (a) move it to `Network=host`, or (b) publish the needed port. Option (b)
makes that port reachable from every netns and from the LAN — an exposure
change that needs operator approval before deployment, not a silent fix.
are (a) move it to `Network=host`, or (b) publish the needed port — but
publishing is advertised **only inside the Podman that owns the container**:
a root container's published port stays invisible to a rootless consumer
(2026-10-19 table above), so the fix must be made on the same root/rootless
side as the consumer. Either way it is an exposure change — a published port
also answers on the LAN — and needs operator approval before deployment, not
a silent fix.
- `chrome-devtools-protocol` is intentionally `Network=host` with a loopback-only
bind so the browser tool stays unreachable from the LAN; do not "tidy" it onto
a bridge network — its two consumers (gateway, webui) are host-netns and reach
@@ -36,7 +36,7 @@ possibly zero behavioral change.
2. Probe the target (ssh as `dank`, passwordless sudo present):
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
- `git -C ~/serverAppConfigs rev-parse HEAD`
- `systemctl is-active/is-enabled <svc>`;
- `systemctl is-active/is-enabled <svc>`
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
- the on-host env file already carries the new keys (it landed via the
ff-pull).
@@ -119,8 +119,8 @@ each, kill stale ones by explicit PID.
A killed deploy leaves behind:
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
explicit stop — start it yourself);
- the state marker **pending** (pre-deploy or in-flight revision);
explicit stop — start it yourself)
- the state marker **pending** (pre-deploy or in-flight revision)
- the e2e gate un-run (the deployer runs it last).
Reconcile the marker ONLY after the service itself is verified healthy — a
@@ -142,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts
numbers — which looks exactly like your change having no effect. Key on
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
See for the full capture
procedure.
## Related
- [[Deployment Flow]] — the deploy pipeline itself
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
- journal capture
@@ -68,7 +68,7 @@ service on the other host.
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
The shared chat template maps to the inference service on the selected target.
The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
The `frank-npu` endpoint is retired () and has no deploy handler
an unrelated large-model deployment never touches it.
## Caddy safety
@@ -125,6 +125,5 @@ routing. See [[Testing Policy]].
## See Also
- [[Testing Policy]]
- [LLM deployment](../LLM/deployment.md)
- [[Caddy]]
- [[SearXNG]]
@@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior.
*arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
sleeping Mac mini must never fail an unrelated deploy. The project joins only
when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`
`shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
that touches the arr links must therefore be paired with a `19-frankbook` run
@@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
else), and accepted traffic returns to Caddy's internal listener on `:8016`,
else), and accepted traffic returns to Caddy's internal listener on `:8016`
which proxies every request whole to the Authentik server with the Host
header preserved; Authentik's embedded outpost routes on that Host against
each proxy provider's `external_host`, serves the login flow in place for
@@ -67,12 +67,12 @@ graph TB
```
Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`):
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`),
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`)
prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta
(`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.`
(the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP
clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP),
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP)
books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts
that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own
auth"). Grafana is **not** exempt even though it signs in through authentik as
@@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through
`Network=searxng.network` (runtime network `systemd-searxng`), while
`searxng.container`, `marreta.container` and `marreta-browser.container` join
the tunnel the modern way — `Network=gluetun.container`
(`dankclaw/searxng/quadlet/searxng.container:18`,
`dankclaw/marreta/quadlet/marreta.container:34`,
(`dankclaw/searxng/quadlet/searxng.container:18`
`dankclaw/marreta/quadlet/marreta.container:34`
`dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet
equivalent of the retired compose `network_mode: service:gluetun`. **Five**
containers share the VPN exit, not three, and Marreta's outbound fetches leave
through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`);
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`)
everything else on Dankclaw uses the normal LAN route. This matches KB Pattern
2 in [[Gluetun]].
See [[Container Network Namespaces]] for which addresses are reachable from that
@@ -36,14 +36,14 @@ done
## 2. Cutover: compose service → Quadlet unit
Checklist (precedents to copy:
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
`dankclaw/observability/quadlet/`):
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
line for line: `Image=` (`localhost/` prefix for locally built images +
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
containers by name), `Network=`, `User=0:0` when the image's own user
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved)
`Exec=` for compose `command:`, `PublishPort=`, health keys,
`[Service] Restart=always` + `RestartSec=10`, `[Install]
WantedBy=default.target`. **Never `systemctl --user enable` the generated
@@ -56,7 +56,7 @@ Checklist (precedents to copy:
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
`~/.config/systemd/user/<svc>.service` that shells out to
`podman compose up` collides BY NAME with the generated unit);
`podman compose up` collides BY NAME with the generated unit)
`podman rm -f <container>` so the unit's container can take the name.
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
@@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever.
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
= the §2 cutover. **Verify the capability, not the status:** `podman exec
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
<name> python3 -c "import torch; print(torch.cuda.is_available())"`
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
@@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
unhealthy but logs look fine, compare the baked
`{{.Config.Healthcheck.Test}}` array against the intended command before
believing the status.
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
- **Probe a new key before shipping:** drop a throwaway `probe.container`
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
"unsupported key"`, `systemctl --user cat probe.service` (shows the
generated ExecStart verbatim), then remove + reload.
@@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con
## See Also
- [[Deployment Flow]]
- [[index]]
@@ -56,8 +56,8 @@ sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed
fixed in the file (`1930` in the art workflow, `2026` in the text one) —
read from `dankclaw/comfyui/workflows/*.json` node `9`.
Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`,
`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`,
Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`
`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`
`SaveImage` (ids 3–11).
| Workflow | File | Shape | Steps | Made for |
@@ -113,9 +113,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`,
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`,
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`,
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
`huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk
yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18).
@@ -126,7 +126,7 @@ yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18)
| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) |
| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) |
Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`
#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
sha256 checks) or a plain `hf download` with the token in the environment —
+3 -3
View File
@@ -43,7 +43,7 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not
## GPU device access (rootless), and why the unit exists
The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`,
The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`
`dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses
`GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman
cannot map the group name into the user namespace. The companion udev rule
@@ -115,7 +115,7 @@ generated units.
The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/`
2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats`
reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The
health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`,
health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`
bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
(`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs
`http://127.0.0.1:8188/` inside the container, and the container reports
@@ -135,5 +135,5 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
- [[Dankclaw Stack]]
- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
- [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
- [[Lemonade and sd.cpp]] — the ggml-family alternative
- the ggml-family alternative
- Knowledge base: [[ComfyUI]] (tools)
@@ -52,11 +52,10 @@ graph LR
The LLM rows are bare-metal systemd services, not containers, and this repo
owns their scripts, units and knowledge base (see
[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on
). Which payload and runtime sits on
each port is era state and lives on the endpoint page, not here. `frank-npu`
(the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon`
on the Dankserver port) are both retired — see [[frank-npu]] and
[[frank-medium-radiance-mxfp4]].
on the Dankserver port) are both retired — see and
## Boot Persistence
@@ -72,7 +71,7 @@ Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-
## See Also
- [[index]] — repository index
- repository index
- [[Network Architecture]] — routing and proxy
- [[Observability]] — Grafana, Prometheus, logs
- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
- the model-serving endpoints on this host
@@ -1,35 +1,36 @@
---
type: reference
description: "The engine behind the fleet's speech endpoint: its runtime, private bridge, and the surface it publishes."
description: "The engine behind the fleet's speech endpoint: its Vulkan build on the iGPU, its private bridge, and what it publishes."
tags:
- dankclaw
- tts
- voice
- quadlet
- cpu
timestamp: 2026-10-18
- vulkan
timestamp: 2026-10-19
---
# Frank TTS Engine
`frank-tts-engine` is the **arm A text-to-speech engine** on Dankclaw: [qwentts.cpp](https://github.com/ServeurpersoCom/qwentts.cpp)'s resident `tts-server` HTTP process, holding the Qwen3-TTS **1.7 B base talker (Q4_K_M)** and the **12 Hz tokenizer (Q8_0)** in RAM once and synthesizing from there. It listens on container port **8880** and is **deliberately not a fleet endpoint** — the endpoint consumers address is `frank-tts` on **7772**, which resolves voice names, re-encodes to mp3 and owns the timeout budget. This page is the engine runbook; the endpoint's contract lives with its own page.
`frank-tts-engine` is the **arm A text-to-speech engine** on Dankclaw: [qwentts.cpp](https://github.com/ServeurpersoCom/qwentts.cpp)'s resident `tts-server` HTTP process, holding the Qwen3-TTS **1.7 B base talker (Q4_K_M)** and the **12 Hz tokenizer (Q8_0)** loaded once and synthesizing from there — **on the gfx1151 iGPU** (ggml's Vulkan backend, `--device Vulkan0`) since 2026-10-19, and in host RAM before that. It listens on container port **8880** and is **deliberately not a fleet endpoint** — the endpoint consumers address is `frank-tts` on **7772**, which resolves voice names, re-encodes to mp3 when asked, and owns the timeout budget. This page is the engine runbook; the endpoint's contract lives with its own page.
**Deployed 2026-10-18.** The chain ran in the order at the bottom of this page — fetch (1.4 GiB, byte + LFS oid verified, 18 s), image build against the pinned commit, unit + network install, container-health gate, warm pass — and the live state still reads the way this page describes it: unit `active` with no drop-ins, container healthy 31 s after the build, **4.0 GiB** resident on a host that kept 17 GiB available, `3 voice(s) registered and warm` in the journal, and no published port anywhere. Two things the first real boot taught: the warm hook's registration body must reach curl through a **file**, not an argument (Troubleshooting), and `STEP 3/13 RUN useradd` is the first honest proof of the image's non-root user.
**Deployed 2026-10-18, cut over to the iGPU on 2026-10-19.** The first chain ran in the order at the bottom of this page — fetch (1.4 GiB, byte + LFS oid verified), image build against the pinned commit, unit + network install, container-health gate, warm pass — and the live state still reads the way this page describes it: unit `active` with no drop-ins, container healthy shortly after the build, `3 voice(s) registered and warm` in the journal, and no published port anywhere. Two things the first real boot taught: the warm hook's registration body must reach curl through a **file**, not an argument (Troubleshooting), and `STEP 3/13 RUN useradd` is the first honest proof of the image's non-root user.
**Host:** Dankclaw (`192.168.0.143`), rootless Podman, user `dank` — **Port:** 8880, **never published** — **Network:** the private `systemd-frank-tts-engine` bridge (10.89.5.0/24) and no other — **Deployer:** `frank-tts-engine` — **Consumers:** `frank-tts` (7772) and, through it, every voice request in the fleet including [[Open Notebook]]'s podcast renders.
## Facts (verified 2026-10-18 against the pinned upstream commit, the Hub, and the CPU trial on this host)
## Facts (verified 2026-10-18 against the pinned upstream commit, the Hub and the CPU trial; the device rows re-verified on the live iGPU shape 2026-10-19)
| Item | Value |
|------|-------|
| Engine | `tts-server` from qwentts.cpp, pinned at commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` (`dankclaw/voiceservice/Containerfile.engine`). The source is **not vendored**: the image clones that commit and its recorded `ggml` submodule, so the build needs outbound HTTPS and the `cfhash` label is what keeps an unchanged re-deploy offline |
| Build line | `cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CCACHE=OFF -DGGML_BACKEND_DL=OFF -DGGML_CPU_ALL_VARIANTS=OFF -DGGML_OPENMP=ON -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=OFF -DGGML_METAL=OFF -DGGML_SYCL=OFF -DGGML_RPC=OFF && cmake --build build -j 8` — the exact line the 2026-10-18 trial ran on this box. `GGML_BLAS=OFF` is **forced**: Dankclaw has `libopenblas.so.0` and no BLAS dev headers, so upstream's own `Dockerfile` and `buildcpu.sh` (both of which pass BLAS on) cannot be copied. No `-DGGML_NATIVE` is passed, and ggml defaults it on, so the binary is `-march=native` for the machine that built it: **build this image on the host that runs it** |
| Image | built locally as `franksdank/frank-tts-engine:latest`, two stages from one digest-pinned base `docker.io/library/ubuntu:24.04@sha256:68ee22004a9109b44973313b02ff7b1de250f1de0f5ecb628d73f579aaff33ac` (index digest resolved 2026-10-18). Runtime stage: `libgomp1` (OpenMP is on), `curl` (the unit's `HealthCmd` runs as `/bin/sh -c`, and the warmer speaks HTTP), uid **10000**, `LD_LIBRARY_PATH=/app` because ggml builds shared by default and the link-time RPATH names the build tree |
| Build line | `cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CCACHE=OFF -DGGML_BACKEND_DL=OFF -DGGML_CPU_ALL_VARIANTS=OFF -DGGML_OPENMP=ON -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_HIP=OFF -DGGML_VULKAN=ON -DGGML_METAL=OFF -DGGML_SYCL=OFF -DGGML_RPC=OFF && cmake --build build -j 8`. **`GGML_VULKAN=ON` since 2026-10-19**, and that door needs three packages in the build stage: `libvulkan-dev`, `spirv-headers`, and **`glslc`** — which Ubuntu's `glslang-tools` does NOT ship (it ships `glslangValidator`, and `find_package(Vulkan COMPONENTS glslc REQUIRED)` fails without the package). The archive's `glslc` is 2023.8: `GL_KHR_cooperative_matrix` compiles, the coopmat2 / integer-dot / bf16 / micro-float shader families do not. `GGML_BLAS=OFF` is **forced**: Dankclaw has `libopenblas.so.0` and no BLAS dev headers, so upstream's own `Dockerfile` and `buildcpu.sh` (both pass BLAS on) cannot be copied. No `-DGGML_NATIVE` is passed, and ggml defaults it on, so the binary is `-march=native` for the machine that built it: **build this image on the host that runs it** |
| Image | built locally as `franksdank/frank-tts-engine:latest`, two stages from one digest-pinned base `docker.io/library/ubuntu:24.04@sha256:68ee22004a9109b44973313b02ff7b1de250f1de0f5ecb628d73f579aaff33ac` (index digest resolved 2026-10-18). Runtime stage: `libgomp1` (OpenMP is on), `curl` (the unit's `HealthCmd` runs as `/bin/sh -c`, and the warmer speaks HTTP), `libvulkan1` + `mesa-vulkan-drivers` (the loader and the RADV ICD — a loader with no ICD enumerates ZERO devices, and the server then dies in device lookup after both GGUFs have been read), uid **10000**, `LD_LIBRARY_PATH=/app` because ggml builds shared by default and the link-time RPATH names the build tree. **No NVIDIA ICD on purpose**: exactly one Vulkan device in the enumeration is what makes the name `Vulkan0` safe to write down |
| Talker | `qwen-talker-1.7b-base-Q4_K_M.gguf`, **1 219 245 248 B**, sha256 `ea393ebaf2167ea23ce9fc18b093822851358a950d7075cd47ab4f6ce23e887d` (the Hub's LFS pointer oid) |
| Tokenizer | `qwen-tokenizer-12hz-Q8_0.gguf`, **291 150 624 B**, sha256 `1883beeed99348fc35e23dd225e9082f93f6f8c109330a33d935baa8acdbfd94` |
| Source of both | `https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF`, fetched by `scripts/download-frank-tts-engine-qwentts.sh` into `/home/dank/models/Qwen3-TTS-GGUF`, mounted read-only at `/models`. It has to be the **base** variant: the engine refuses every voice registration unless the model reports type `base`, so on `customvoice`/`voicedesign` the cloned-voice product does not exist. A 0.6 B talker (628 905 056 B) exists and is **not** the decided artifact |
| Memory | **5.7 GiB peak RSS** with both files resident (2026-10-18 trial). No `LimitMEMORY`/`MemoryMax`: a limit would turn a busy moment into an OOM kill in the middle of an utterance |
| Concurrency | `--max-batch 1`: a serialised FIFO. Warm RTF on this box is 0.55–0.67 against real time 1.0, and that margin is the entire headroom a live stream has; two concurrent syntheses split the same cores and the second caller's first byte additionally waits for the first utterance to finish |
| Device | `--device Vulkan0`: the gfx1151 iGPU, **shared with [[frank-dragoon]] as a co-tenant, never a handover**. Proven on the live pair, not inferred: `fuser -v /dev/dri/renderD129` lists `tts-server` beside `flash_serve`. Both gates (the RADV heap and dragoon's contention) are §5 of `report/tts-npu-spike-verdict-2026-10-19.md`, which also carries the change surface. **Rollback: `QWENTTS_DEVICE=` (empty) in `scripts/frank-tts-engine.env`, redeploy this one service** — the entrypoint then passes no `--device` at all, which is the CPU shape, no rebuild, nothing else restarts |
| Memory | **1.7 GiB host RSS** since the iGPU cutover — the weights ride the device. No `LimitMEMORY`/`MemoryMax`: a limit would turn a busy moment into an OOM kill in the middle of an utterance |
| Concurrency | `--max-batch 1`: a serialised FIFO. Warm RTF on the iGPU measures **0.20** (11.04 s of audio in 2.23 s) with `frank-dragoon` live on the same card, so a live stream has room to spare; a second concurrent synthesis would share the device with the first and make the caller behind it wait out the whole first utterance, so the FIFO stays a decision, not an accident |
| Network | `systemd-frank-tts-engine` (10.89.5.0/24, `dankclaw/voiceservice/quadlet/frank-tts-engine.network`) and **nothing else**, with **no `PublishPort` at all**. `frank-tts` joins the same bridge and dials `http://frank-tts-engine:8880` |
| Voices | `/home/dank/voices` mounted read-only at `/voices`, and it is the **warmer** that reads it, not the server: `tts-server` receives reference audio as base64 in a registration body and keeps its registry in memory only |
| Health | `GET /health` → `{"status":"ok"}`, served by the HTTP thread rather than the synthesis queue, so a busy engine still reads healthy. Probed by the unit (`HealthCmd`, 30 s interval, 120 s start period) **inside the container's namespace**, because there is no published port to probe from outside |
@@ -37,18 +38,11 @@ timestamp: 2026-10-18
## Why this engine, and not the alternatives
The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in [[Local TTS and Voice Cloning]], which is the source of these numbers):
The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. On that criterion the resident `tts-server` (arm A) beat the mainline per-request CLI path (arm B) on this box the same day, and the deciding property was **spread**: the resident server repeats the same first-audio latency every run, while the CLI path varies enough that some runs fall behind real time and a live listener hears the stream stall mid-sentence. That A/B was a CPU measurement and the endpoint has since moved to the iGPU, so the shape of the argument is what is kept here; no page carries its numbers any more, and they are not worth re-deriving.
| Measure | Resident `tts-server` (arm A, chosen) | Mainline per-request CLI path (arm B) |
|---|---|---|
| first audio byte | **86–451 ms** end to end; **94–112 ms** server-side | model load, per request |
| warm real-time factor while streaming | **0.55–0.67** | 0.93–0.99 |
| run-to-run spread | **±0.01** | ±0.25 |
| HTTP surface | serves `/v1/audio/speech` itself | none — façade CLI only |
The rollback to the other backend stays in the repo untouched — `TTS_BACKEND=llama-tts-cli` in `scripts/frank-tts.env` still selects the per-request CLI engine, and that file's header documents what it costs.
The spread is the whole argument: at ±0.01 the engine finishes synthesizing 1.5–1.8× faster than the audio plays, every run; at ±0.25 some runs fall behind real time and a live listener hears the stream stall mid-sentence. The rollback path stays in the repo untouched — `TTS_BACKEND=llama-tts-cli` in `scripts/frank-tts.env` still selects the per-request CLI engine, and that file's header documents what it costs.
## Three laws this service lives under
## Four laws this service lives under
**1. The registry law → the unit warms at boot.** The voice registry is **memory only**: a restart forgets every voice, and re-registering one does **not** warm it — only synthesizing does. A boot that registers nothing leaves the façade answering 400 for every name, which is why `frank-tts-engine.container` carries
@@ -64,10 +58,12 @@ frank-tts-engine-warm: voice narrator1 cold=1402ms warm=1812ms
frank-tts-engine-warm: 3 voice(s) registered and warm
```
**2. The thread law → nothing is ever pinned.** `tts-server` in this build has **no `-t`/`--threads` flag**: `src/backend.h` computes the CPU thread count as `std::thread::hardware_concurrency() / 2` (its comment: GEMM shares SIMD units across hyperthreads). Thread count therefore follows what the kernel *reports*, not what a cgroup *allows* — so `CPUQuota=`, `CPUAffinity=`, `taskset` or a slice silently changes the engine's GEMM tiling and its timings, with no log line. Nothing in this fleet pins this container. If a neighbour starves it, fix the neighbour.
**2. The thread law → nothing is ever pinned.** `tts-server` in this build has **no `-t`/`--threads` flag**: `src/backend.h` computes the CPU thread count as `std::thread::hardware_concurrency() / 2` (its comment: GEMM shares SIMD units across hyperthreads). Thread count therefore follows what the kernel *reports*, not what a cgroup *allows* — so `CPUQuota=`, `CPUAffinity=`, `taskset` or a slice silently changes the engine's pool sizes and its timings, with no log line. The rule did not retire with the iGPU cutover: the binary still sizes those pools from what the kernel says, whatever the device ends up doing with the matmuls. Nothing in this fleet pins this container. If a neighbour starves it, fix the neighbour.
**3. Streaming is not buffered.** At the same seed, `response_format=pcm` (chunked, codec chunks pushed as they decode) and `response_format=wav` (buffered) differ in **92.2 % of samples**, −41.6 dB RMS. Nothing may compare the two paths, cache one against the other, or assume a seed pins output across `response_format`. `frank-tts` always asks for `wav`; the warmer asks for `wav`; the test seam is forbidden from pinning either.
**4. The device law → the card is shared, and the rollback is one line.** Synthesis computes on the gfx1151 **beside** [[frank-dragoon]], so a change to either tenant is a change to both. Two gates decide that pair, and both passed on 2026-10-19 with the other tenant live: the RADV heap a rootless container tenant is actually handed (not the ~2 GiB DRM carve-out), and dragoon's own first-token and decode rates with a synthesis in flight — unchanged. Their numbers, the recipe that measures them, and the full change surface are §5 of `report/tts-npu-spike-verdict-2026-10-19.md`; do not re-derive them here. What belongs on this page is the move: `QWENTTS_DEVICE=` (empty) in `scripts/frank-tts-engine.env`, re-deploy this one service, and synthesis is back on CPU — no rebuild, and dragoon never restarts.
## The voice registry contract
A voice becomes known to the engine through `POST /v1/audio/voices`, and only these shapes are accepted:
@@ -97,9 +93,9 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note
| `dankclaw/voiceservice/Containerfile.engine` | two-stage build of the pinned commit, proven cmake line, digest-pinned base, ships both scripts |
| `dankclaw/voiceservice/tts-engine-start.sh` | image entrypoint (`/app/start.sh`): env → flags, refuses to boot without weights |
| `dankclaw/voiceservice/tts-engine-warm.sh` | boot warmer (`/app/warm-voices.sh`): register from `reference.wav` (body via a temp file), speak a fixed sentence twice, report cold/warm |
| `dankclaw/voiceservice/quadlet/frank-tts-engine.container` | the unit: bridge only, no publish, env file, two read-only mounts, health probe, warm hook, `Restart=always` |
| `dankclaw/voiceservice/quadlet/frank-tts-engine.container` | the unit: bridge only, no publish, env file, two read-only mounts, render access (`AddDevice=/dev/dri` + `GroupAdd=keep-groups`), health probe, warm hook, `Restart=always` |
| `dankclaw/voiceservice/quadlet/frank-tts-engine.network` | the private 10.89.5.0/24 bridge, with the subnet registry |
| `scripts/frank-tts-engine.env` | paths and numbers only, with the thread law and the registry law in its header |
| `scripts/frank-tts-engine.env` | paths and numbers only, with the thread law, the registry law and the device law in its header |
| `scripts/download-frank-tts-engine-qwentts.sh` | weight fetcher: no resume, byte count + LFS oid check, `--verify-only` |
| `scripts/deploy-remote.sh` → `deploy_frank_tts_engine` | weights gate → build → quadlet → restart → container health → warm |
| `tests/frank_tts_engine.bats` | the seam: generator output, entrypoint argv, warmer against a stub engine, planner and routing maps |
@@ -118,6 +114,25 @@ still open.
6. **Then the façade on 7772** (its own page), after the engine answered — that order is the whole point of step 6: publishing the façade in front of a dead engine would make every consumer fail at the first voice line.
7. **Open:** [[Open Notebook]] is still not installed, so nothing renders a podcast yet; the voice leg it waits on now exists.
### 2026-10-19 — the iGPU cutover (PR #34, `6248f02`)
1. **Configure gate before touching anything** — the flag set, configured inside the
digest-pinned base on Dankclaw: `Found Vulkan: … found components: glslc` then
`Including Vulkan backend`. That probe is what caught `glslc` missing from
`glslang-tools`; the Containerfile's flag comment carries the finding.
2. **Deploy** — `./scripts/deploy.sh dankclaw --service frank-tts-engine`: image rebuild
(the `cfhash` moved), unit reinstall, health gate, warm pass
(`3 voice(s) registered and warm`), Playwright e2e 33/33.
3. **Prove the device, not the flag** — `sudo fuser -v /dev/dri/renderD129` lists
`tts-server` next to dragoon's `flash_serve`. Do this instead of reading the log: the
engine prints nothing about which backend it chose, so argv (`--device Vulkan0`) proves
intent and the render-node holder list proves use.
4. **Consumer path** — one `/v1/audio/speech` per voice through the façade on 7772; all
three answered with audio. The RTF figure in the Facts table comes from one engine-direct
WAV, where the duration is exact.
5. **Co-tenancy gate** — §5 of the verdict report: dragoon's first token and decode
unchanged with a 16.9 s synthesis occupying the card.
## Troubleshooting
| Symptom | Meaning | Move |
@@ -128,6 +143,8 @@ still open.
| Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
| `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
| A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see |
| The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this |
| A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces |
| Unit restarts every 10 s and the log ends on a **device** error *after* both GGUFs were read | the image has the Vulkan loader but no ICD, or the container has no render node — the device name then resolves to nothing | `podman exec frank-tts-engine ls /dev/dri` (it must see `renderD129`) and `podman exec frank-tts-engine dpkg -l \| grep -E 'mesa-vulkan\|libvulkan1'`; re-deploy to rebuild the image, or set `QWENTTS_DEVICE=` and re-deploy for the CPU shape |
| `frank-dragoon` slows while TTS speaks | the contention premise broke — a Mesa bump, a third GPU tenant, a bigger synthesis | `QWENTTS_DEVICE=` + re-deploy this service, then re-run the gate recipe in §5 of `report/tts-npu-spike-verdict-2026-10-19.md` before putting it back |
+32 -13
View File
@@ -18,10 +18,10 @@ audio in that cloned voice. Everything else in the voice stack sits behind it
the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers
never learn a path.
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`,
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`
both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
the same change. Probes on the deployed endpoint: `/health` answers
`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`,
`{"status":"ok","backend":"omni-http","voices":["dr.kern","garrickhagon","jeffhays","vaati","wilwheaton"]}`
and two different voices rendered the same sentence into **different** bytes
(24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
pass-through of one default. The operator harness routes its `speech` role here;
@@ -32,7 +32,7 @@ render needs the app installed, not this endpoint.
| --- | --- |
| Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
| Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] |
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only |
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only. The request's `model` field is **ignored** (measured 2026-10-19: `voxcpm2` and `tts-1` both synthesize the same way) — the voice comes solely from `voice`, so a consumer that must name a model may name it anything |
| Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
| Voice packs | `/home/dank/voices` mounted read-only at `/voices` — host state, never repo state |
@@ -42,8 +42,8 @@ render needs the app installed, not this endpoint.
## Naming a voice
A voice **is** a directory name. `voice=narrator1` resolves to
`/voices/narrator1/reference.wav`, and that file — not a manifest, not a config
A voice **is** a directory name. `voice=wilwheaton` resolves to
`/voices/wilwheaton/reference.wav`, and that file — not a manifest, not a config
entry — is what makes the name real. omp's TTS device may send `voice_id`; the
shim accepts either. A call that sends **neither** — the harness's normal shape,
because no harness setting carries a voice for this transport — speaks
@@ -59,10 +59,11 @@ loudly instead of letting it half-work.
Names are user-visible: an unknown voice comes back as
`400 unknown voice 'x'; available voices: …`, and that list is the honest
census. The podcast speakers are `narrator1`, `narrator2`, `narrator3` —
[[Open Notebook]] is configured with exactly those three, so renaming them
breaks episode rendering, not just speech. A personal voice gets the person's
name, lowercase.
census. As of 2026-10-20 the live packs are `dr.kern`, `garrickhagon`,
`jeffhays`, `vaati`, `wilwheaton`. [[Open Notebook]]'s `dank-duo` profile speaks
`wilwheaton` (Frank) and `vaati` (Comrade), and `DEFAULT_VOICE` is `wilwheaton`.
Renaming a pack that a speaker profile names breaks episode rendering, not just
speech. A personal voice gets the person's name, lowercase.
## Voice packs
@@ -111,6 +112,24 @@ own `/health` answers 200 with the backend name and the voice list it resolved
without dialing the engine, so "façade healthy, engine down" is a real state: it
shows up as a failed synthesis, never as a down probe.
## Where the engine computes
The **gfx1151 iGPU**, as a co-tenant beside `frank-dragoon`: the Vulkan build of
`qwentts.cpp` with `--device Vulkan0`, and `/dev/dri` handed to the engine container. That is
the shape since 2026-10-19. The flag, the device name, the render-node trap on this hybrid box
and the one-line CPU rollback are owned by [[Frank TTS Engine]] — read that page before
touching either tenant, because a change to one is a change to both. Co-tenancy was gated with
`frank-dragoon` live on the card (its first token and decode rates unchanged with a synthesis
in flight); both gate recipes are §5 of `report/tts-npu-spike-verdict-2026-10-19.md`.
**The NPU is a door only for a model the vendor has already compiled for it.** XDNA2 runs nothing but
precompiled AIE partition programs, and AMD's own Linux runtime opens `/dev/accel/accel0`, succeeds at the
ioctls, then declines this model family's graph without emitting a single log line. That is why TTS stays on
the GPU — and it is not a general veto on the device: halogen ships its own precompiled catalog (embeddings,
decisions, rerankers) that does run there, which is what [[frank-dragoon]]'s NPU-lane section is about. The
measured record and the model-side contract live in `report/` at the repo root
(`tts-npu-spike-verdict-2026-10-19.md`).
## Files
| Path | What |
@@ -147,7 +166,7 @@ A `/health` probe job would therefore be structurally never-green, and because
`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
Signal every few minutes for an endpoint that is fine. The engine cannot be
scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
scraped at all: it publishes nothing. Status lives in `systemctl --user status`
`podman healthcheck run`, the two unit journals, and the homepage card, which
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
2026-10-18 deploy.
@@ -155,7 +174,7 @@ probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
## The harness speech role
The operator harness reaches this endpoint through
`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`,
`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`
`baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api
`openai-speech` on the **model** row (naming a runner api at *provider* level
fails the whole file's validation and disables every custom provider, which is
@@ -171,7 +190,7 @@ harness bundle and measured the day of the flip:
`speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list,
and it never reaches an HTTP request. So a harness call arrives voice-less by
default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared
pack answers a voice-less request and the journal marks that line `(default)`,
pack answers a voice-less request and the journal marks that line `(default)`
while an **unknown** name still 400s with the list that exists.
- **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not
select a pack; it hands the vocalizer a value outside its enum. (The first
@@ -203,5 +222,5 @@ harness bundle and measured the day of the flip:
| Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
| First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
| The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
| A consumer rejects `wilwheaton` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
---
+1 -1
View File
@@ -33,7 +33,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net
| `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) |
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* |
The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)
The LLM fleet knowledge base:
## Management
@@ -30,7 +30,7 @@ entrypoint does not start the dashboard. The dashboard is a separate process, so
it runs in its own Quadlet unit
([`dankclaw/hermes/quadlet/hermes-dashboard.container`](../../dankclaw/hermes/quadlet/hermes-dashboard.container):
image `localhost/hermes-agent:latest`, `Pull=never`, `Exec=hermes dashboard
--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`,
--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`
`RestartSec=5`). All three Hermes containers mount the same `/data` bind mount.
Unlike its two siblings — `hermes.container` and `hermes-webui.container` both
@@ -40,7 +40,7 @@ from another LAN host: `/health` → 200, `/` → 302) — **Vhost:**
## Why a container, not a host process
The gateway's state files are owned by the container-mapped uid (`109999`,
The gateway's state files are owned by the container-mapped uid (`109999`
from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` →
`dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 =
109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless
@@ -82,12 +82,12 @@ the sibling dashboard on `9119` is not.
- Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994,
shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only
*(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
- (unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
without `sudo`)*; the account can do nothing except
authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel
while the transport stays up for port forwarding.
- sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31)
scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`,
scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`
`X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are
present under `Match User tunnel` (`cat` 2026-10-18).
- Key options on the authorized key pin the blast radius:
@@ -130,7 +130,7 @@ only if inactive (it is never restarted on deploy, so in-flight browser
sessions survive), re-tags the image alias, then restarts
`hermes.service`, `hermes-dashboard.service` and
`hermes-webui.service` in the **user** manager of `dank` (linger must be on for
them to come up at boot — `loginctl show-user dank -p Linger` → `yes`,
them to come up at boot — `loginctl show-user dank -p Linger` → `yes`
2026-10-18), so gateway/dashboard/webui restarts stay isolated from
inference deploys.
Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
+7 -7
View File
@@ -18,7 +18,7 @@ bindings, 2026-10-18). Note the **container name drift**: the unit is
`podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so
`podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no
`ContainerName=` key even though its header comment says "container name:
hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached
`hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A
record; the name resolves from the Pi-hole hosts file,
`dankclaw/pihole/addn-hosts:39`) — **Docs:**
@@ -37,7 +37,7 @@ designed wiring; the remedy for both halves is in the log line
**Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted
at `/data` in all three Hermes containers (gateway, webui, dashboard —
`hermes.container:36`, `hermes-webui.container:46`,
`hermes.container:36`, `hermes-webui.container:46`
`hermes-dashboard.container:24`). Each
container additionally gets an anonymous volume at `/opt/data`, which is
re-created empty on every container recreation — anything that lives only there
@@ -80,7 +80,7 @@ is not durable. The tracked, sanitized configuration baseline is
`frank-dragoon` — verified from the host with `curl
http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
`127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
and is IPv4-only);
and is IPv4-only)
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
2026-10-18, that endpoint serves only `frank-archon`** (`GET
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
@@ -115,7 +115,7 @@ per session; the browser itself lives in a dedicated daemon container, the
digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host
network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows
`127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image
`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`,
`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`
`dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal
gateway and the webui server — run in the host network namespace (`podman
inspect` → `net=host` for both, 2026-10-18), so they can
@@ -248,7 +248,7 @@ What actually breaks, and the fix:
effort: none # mandatory, see below
```
Before copying this: which id is served on which port is endpoint state — see
Before copying this: which id is served on which port is endpoint state —
[[frank-zealot]] and [[frank-archon]]. The provider/model
pairing above is the shape that was landed in 2026; re-derive it from the live
config before reusing it.
@@ -282,7 +282,7 @@ Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth
on match, 401 on miss. No username field. `/health` answers 200
unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with
`{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w
%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via
%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth
those**; probe
login with a deliberately wrong password (expect 401) and the real one
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
@@ -311,7 +311,7 @@ login with a deliberately wrong password (expect 401) and the real one
instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18;
off-network ingress is SSH tunnel only —
user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth
*(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*,
- (unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*
forward remote end
localhost:8787).
+10 -10
View File
@@ -19,7 +19,7 @@ returns a stripped, readable version of the page — no paywall chrome, no
tracking, no cookie walls. It is a Laravel 13 app on the
`serversideup/php:8.4-fpm-nginx` image (`composer.json` requires
`php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` —
<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>),
<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>)
with an optional headless-browser companion for JavaScript-heavy pages.
**Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained)
@@ -31,7 +31,7 @@ with an optional headless-browser companion for JavaScript-heavy pages.
- Fetches a target URL with a cascade of strategies: plain HTTP (curl), a
headless browser (Lightpanda) for JS pages, and the Internet Archive
(Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php`
files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`,
files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`
`WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`).
- Strips paywall/consent/tracking markup and re-renders clean HTML.
- Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` /
@@ -69,7 +69,7 @@ graph LR
| `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) |
Both units are boot-persistent (`WantedBy=default.target`) and self-healing
(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`,
(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`
`marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live
2026-10-18 both report `systemctl --user is-enabled` → `generated` and
`is-active` → `active`, and `podman inspect` reports marreta **healthy**
@@ -87,14 +87,14 @@ The fix is a single env var, verified against the serversideup source:
- The image sets `ENV NGINX_HTTP_PORT="8080"` and the site template is
`listen ${NGINX_HTTP_PORT} default_server;`.
- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build);
- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build)
the entrypoint `10-init-webserver-config.sh` runs `envsubst` over every
`NGINX_*` variable **at container start**.
- So `Environment=NGINX_HTTP_PORT=8085` in the quadlet relocates the listen
port off the collision with no custom build.
Marreta then publishes **nothing** of its own — the netns owner (Gluetun)
publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`),
publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`)
exactly as it publishes `8081:8080` for SearXNG. Caddy dials
`192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the
relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085`
@@ -127,7 +127,7 @@ on the mechanism: `BrowserFetchStrategy.php` only calls
(`dankclaw/Caddyfile:159-163`).
2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies
whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`).
3. **Post-auth routing** — authenticated traffic returns to `http://:8017`,
3. **Post-auth routing** — authenticated traffic returns to `http://:8017`
where `@marreta host marreta.franksdank.cloud` routes to
`reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`).
@@ -164,7 +164,7 @@ Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0
— that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and
this container sets no `TRUSTED_PROXIES`, so the default is what runs
(`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL`
starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`,
starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`
`URL::forceScheme('https')`). With
`APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded
scheme and client IP through the whole chain, so there is no redirect loop
@@ -234,7 +234,7 @@ netns down and would otherwise orphan it).
Deployed via `deploy.sh dankclaw --service gluetun --service marreta --service
caddy` (gluetun first to recreate the tunnel with the new `8082:8085` publish,
which also cycles SearXNG's netns ref). Authentik `ProxyProvider` `marreta`
registered (pk 13, `external_host=https://marreta.franksdank.cloud/`,
registered (pk 13, `external_host=https://marreta.franksdank.cloud/`
`internal_host=http://caddy:8017`). Results:
- Unauth edge → Anubis challenge page (not 502). TLS cert issued via DNS-01.
@@ -245,14 +245,14 @@ registered (pk 13, `external_host=https://marreta.franksdank.cloud/`,
(`GET 127.0.0.1:9222/json/version` → `Lightpanda/1.0`).
- SearXNG unaffected: `:8081` → 200 after the tunnel recreate.
> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`,
> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`
> which 500s every app route (see Secrets above). Fixed by re-keying with the
> `base64:` prefix before any encrypted data was persisted.
## Citations
- Marreta upstream: <https://github.com/manualdousuario/marreta> (README,
`compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`,
`compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`
`lang/` directory listing).
- serversideup/php fpm-nginx port mechanism: `serversideup/docker-php`
`src/variations/fpm-nginx/Dockerfile` (`NGINX_HTTP_PORT="8080"`) and
+74 -27
View File
@@ -6,25 +6,26 @@ tags:
- open-notebook
- podcasts
- quadlet
timestamp: 2026-10-18
timestamp: 2026-10-19
---
# Open Notebook
open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook)) is the notebook layer that turns existing notes and sources into a **spoken episode**: an outline, then a dialogue transcript, then one synthesized clip per speaker turn, concatenated with ffmpeg into an MP3. In the fleet it is the consumer that closes the loop Frank's audio stack was built for — the script is written by **frank-zealot**, the voices are rendered by **frank-tts**, and nothing leaves the LAN.
open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook)) is the notebook layer that turns existing notes and sources into a **spoken episode**: an outline, then a dialogue transcript, then one synthesized clip per speaker turn, concatenated with ffmpeg into an MP3. In the fleet it is the consumer that closes the loop Frank's audio stack was built for — the script is written by **frank-archon**, the voices are rendered by **frank-tts**, and nothing leaves the LAN.
**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Re-censused 2026-10-19 and unchanged: no `*notebook*` unit in `systemctl --user list-unit-files`, nothing installed under `~/.config/containers/systemd/` by that name, no container in `podman ps` — the repo's three quadlet files have never been installed on the host. Deploying it is a deliberate next step, not a pending one.
**Deployed 2026-10-19.** Both units installed and `active`, both containers healthy, SurrealDB migrated to v25 on first boot, and the first episode rendered end to end: a note in, an outline, 23 dialogue turns, 23 clips, and one **747-second MP3 on the host bind, produced in 301 seconds** — about 2.5× real time with every concurrency knob at 1. The download route answers `200 audio/mpeg` with the bearer token and **401 without it**. It is guarded like every other user service — `https://notebook.franksdank.cloud`, Anubis → Authentik → the app's own login — and the API is no longer published on the LAN at all.
**Host:** Dankclaw (rootless Podman, user `dank`) — **Ports:** 8502 UI and 5056 API, published on the LAN, **no vhost and no Caddy route** — **Deployer:** `open-notebook` (one service, two units) — **Docs:** https://github.com/lfnovo/open-notebook
**Host:** Dankclaw (rootless Podman, user `dank`) — **User path:** `https://notebook.franksdank.cloud` (guarded) — **Published:** `0.0.0.0:8502` (the only upstream Caddy dials) and `127.0.0.1:5056` (the deployer's health gate, nothing else) — **Deployer:** `open-notebook` (one service, two units) — **Docs:** https://github.com/lfnovo/open-notebook
## Facts (verified 2026-10-18 against the pinned images, upstream source, and the live host)
## Facts (images and upstream source verified 2026-10-18; live-host columns re-read 2026-10-19 after deploy)
| Item | Value |
|------|-------|
| App image | `docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212` (`open-notebook.container:79`) — the tag upstream's own `docker-compose.yml:27` pins for the multi-container shape. `v1-latest` is a moving tag, so the manifest-list digest (resolved from the registry 2026-10-18) is what makes this reproducible |
| Database image | `docker.io/surrealdb/surrealdb:v2@sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723` (`open-notebook-surrealdb.container:66`) — that digest is **SurrealDB 2.7.0** (image built 2026-09-23; measured by running the digest: `surreal version`) |
| Do NOT use | the `-single` tags (`v1-latest-single`). That variant bundles SurrealDB inside the app image and is deprecated upstream; here it would give the host a second, hidden database |
| Ports | container 8502 (Next.js UI) + 5055 (FastAPI API) → published `0.0.0.0:8502:8502` and `0.0.0.0:5056:5055` (`open-notebook.container:83-84`). **Host 5055 is taken**: seerr publishes `127.0.0.1:5055` (`dankclaw/seerr/quadlet/seerr.container`, and `ss -ltn` on Dankclaw shows `127.0.0.1:5055` live, 2026-10-18). The API therefore leaves on 5056, which is what forces `API_URL` (below) |
| Ports | container 8502 (Next.js UI) + 5055 (FastAPI API) → published `0.0.0.0:8502:8502` and `127.0.0.1:5056:5055` (`open-notebook.container`). **Host 5055 is taken**: seerr publishes `127.0.0.1:5055` (`dankclaw/seerr/quadlet/seerr.container`, and `ss -ltn` on Dankclaw shows it live). The API therefore leaves on 5056, and on loopback only: the vhost is its sole user path and the image's own Next.js layer rewrites `/api/*` to it, so nothing outside the container needs it |
| User path | `https://notebook.franksdank.cloud` → `anubis:8923` → post-auth listener `@notebook` → `192.168.0.143:8502` (`dankclaw/Caddyfile`); Authentik provider `notebook-proxy` (`external_host` the vhost with its trailing slash, `internal_host http://caddy:8017`); LAN record in `dankclaw/pihole/addn-hosts` |
| Database port | 8000 inside its own namespace, **never published** (`open-notebook-surrealdb.container` has no `PublishPort`). Upstream publishes `127.0.0.1:8000` for debugging (`docker-compose.yml:18`); this unit does not, and the seam asserts that it stays that way |
| Data | `open-notebook-data:/app/data` (named volume: uploads, LangGraph SQLite checkpoints, extraction caches) **plus** the interim host bind `/home/dank/media/podcasts:/app/data/podcasts` (`open-notebook.container:87-88`); database in the `open-notebook-db` named volume |
| Networks | app: `dankclaw_default` first, then the private `systemd-open-notebook` (10.89.4.0/24, `open-notebook.network:33`); database: the private bridge only. Order is load-bearing — the generated `--network` flags keep file order, so the default route rides the network where `host.containers.internal` is proven |
@@ -33,6 +34,19 @@ open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook))
| Env | `scripts/open-notebook.env` (repo, knobs) then `/home/dank/Containers/open-notebook/env` (host, secrets, mode 600) — two `EnvironmentFile=` lines per unit, repo file **first** because the last file wins |
| Deployer | `deploy_open_notebook` in `scripts/deploy-remote.sh`: seed secrets → ensure both digest pins are in local storage → install both units + the network → `mkdir -p /home/dank/media/podcasts` → restart database, wait for its container health → restart app → `wait_for_health open-notebook http://127.0.0.1:5056/health 180` |
## The guard chain, and the single origin
`notebook.franksdank.cloud` follows the fleet's exposure policy in this order, and the order is the law ([[Network Architecture]]):
1. **Provider first.** Authentik proxy provider `notebook-proxy` existed before any route pointed at the app: `external_host https://notebook.franksdank.cloud/` (exact vhost, trailing slash), `internal_host http://caddy:8017`, the three flows copied from `embedded-outpost`, Application `Open Notebook` (`notebook-guard`), and the provider attached to the embedded outpost's provider set. Flip the vhost first and authenticated traffic lands on the outpost's error page. Verify after the change that the outpost logged no `Failed to fetch outpost from API` (the null-flow wedge) and that an already-guarded vhost still answers.
2. **Then the route, in one deploy:** the guarded site block (`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}` — the whole request, so no path ever reaches the app unchallenged) plus the post-auth `@notebook` handler on `http://:8017`, plus the LAN record in `dankclaw/pihole/addn-hosts`. The public Porkbun A record is a host action, not a repo file.
3. **One upstream, not two.** `reverse_proxy 192.168.0.143:8502`: the image's own Next.js layer rewrites `/api/*` to the FastAPI beside it (`next.config.ts`, `INTERNAL_API_URL` default `http://localhost:5055`), so the API needs no route and no outward face — hence `127.0.0.1:5056`. Caddy dials the published LAN port rather than a container name because this Caddy lives in root's Podman and cannot see a rootless container at all ([[Container Network Namespaces]]). SSE (`/api/search/ask`, chat) streams through the same bare proxy that carries OpenWebUI's chat at `ai.franksdank.cloud`; no `flush_interval` or timeout directive is needed, and adding a route timeout is precisely what would cut it.
4. **`API_URL` must name the vhost.** `GET /config` (the Next route) answers in priority order: `API_URL` env → autodetect from `Host` + `X-Forwarded-Proto` → `http://localhost:5055`. Autodetect appends **`:5055`**, a port that belongs to seerr on this host, so leaving `API_URL` unset hands the browser an address nothing answers and every call fails under a UI that loaded fine. `CORS_ORIGINS` is set to the same single origin only to keep the app from defaulting to `*`. `scripts/open-notebook.env` carries both.
The app's own login stays on behind the chain. It is not redundant: publishing the API on loopback does **not** hide it from this host's other containers, which can dial `open-notebook:5055` by name on `dankclaw_default`. The chain decides *who from outside* gets in; `OPEN_NOTEBOOK_PASSWORD` decides what anything that already reached the port may do.
**Single tenant — there are no user accounts, so this is an admin-side tool.** Upstream ships one global password and nothing else: `PasswordAuthMiddleware` compares `Bearer <OPEN_NOTEBOOK_PASSWORD>`, and the SurrealDB schema has no user/account/owner table (verified in the running image, 2026-10-20). One shared password therefore means **one shared workspace** — the guard chain can gate the door but cannot partition the data, and Authentik identity is never consumed. Multi-user is *approved but unbuilt* upstream (issue **#712**, "design phase, targeted post-v2"); OIDC/SSO was declined for now (**#607** closed). Real per-person isolation means **one deployment per person**, not a flag to flip. For that reason the homepage lists open-notebook under **admin**, not as a user-facing app.
## What the image actually runs
One container, three processes under supervisord (`supervisord.conf` upstream): `uvicorn api.main:app --port 5055` (`:12`), the job worker (`:25`), and the Next.js frontend (`:39`). Two consequences the config depends on:
@@ -44,38 +58,63 @@ Also from the image itself: `WORKDIR /app` and `EXPOSE 8502 5055` (`Dockerfile:8
`DATA_FOLDER` is hard-coded `"./data"` (`open_notebook/config.py:5`) and `PODCASTS_FOLDER` is `"./data/podcasts"` (`:19`); the database stores each episode's audio path **relative** to that folder (`episodes/<uuid>/audio/<uuid>.mp3`), and the download route is `GET /api/podcasts/episodes/{id}/audio`. That is why covering one subdirectory with a bind mount is sufficient — nothing in the database has to change if the mount moves.
## Provisioning the providers (after first boot; UI or REST, never a unit)
## What is provisioned (REST, 2026-10-19)
Credentials are stored in the database, encrypted, and are **not expressible in env or a Quadlet file**. The sequence is one-time, in **Manage → Models**, and both entries use provider type **OpenAI Compatible** (upstream: any server that implements `/v1/audio/speech` works this way — `docs/5-CONFIGURATION/local-tts.md:5`).
Credentials live in the database, encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`, and are **not expressible in env or a Quadlet file** — the UI (**Manage → Models**) or the REST API are the only doors. Both use provider type `openai_compatible`, because any server that answers `/v1/audio/speech` or `/v1/chat/completions` works through it (`docs/5-CONFIGURATION/local-tts.md:5`).
1. **TTS credential** — base URL **`http://frank-tts:7772/v1`**. The `/v1` is part of the URL: the client posts to `<base_url>/audio/speech`, so a base URL without `/v1` 404s on every dialogue line (`local-tts.md:72-80`, the base-URL table, always carries `/v1`). `frank-tts` resolves by container name because the app is attached to `dankclaw_default`, the network frank-tts's unit joins (`dankclaw/voiceservice/quadlet/frank-tts.container:51-52`). If frank-tts is ever moved off that network, the published-port form is `http://192.168.0.143:7772/v1` — a published port is dialable at the host's LAN address from any netns. The API-key field stays empty; frank-tts does not authenticate.
2. **LLM credential** — base URL **`http://host.containers.internal:7771/v1`**, model row `frank-zealot`. frank-zealot is a **host process** on `0.0.0.0:7771` (`ss -ltn` on Dankclaw shows the listener with no container owning it, 2026-10-18), and on this netavark host a bridge container reaches the host's LAN IP only at a *published* port — which is why Prometheus scrapes the same endpoint as `host.containers.internal:7771` (`shared/prometheus/prometheus.yml:89-111`, "not their own LAN addresses"). See [[Container Network Namespaces]]. `http://192.168.0.143:7771/v1` is the variant to try only if the gateway name ever stops resolving; it is expected to be refused from a bridge netns.
3. **Model rows are created by name.** Discovery is optional (see the next section): type the model id and confirm the add. One base URL per credential, so the TTS and LLM credentials are necessarily two separate configurations.
| Record | Value |
|--------|-------|
| Credential `frank-tts` | modality `text_to_speech`, base URL `http://frank-tts:7772/v1`, no API key (the shim does not authenticate) |
| Credential `frank-archon` | modality `language`, base URL `http://192.168.0.235:7770/v1` |
| Credential `frank-dragoon NPU` | modality `embedding`, base URL `http://frank-dragoon-halogen:8731/v1`, dummy API key (the lane does not authenticate) — a container name on `dankclaw_default`, which is the address shape that clears the URL guard below |
| Model row `voxcpm2` | type `text_to_speech` on the frank-tts credential. The name is decorative: frank-tts ignores the request's `model` field and synthesizes whatever `voice` names ([[Frank TTS]]) |
| Model row `frank-archon` | type `language`; `PUT /api/models/defaults` points chat, transformation and large-context at it |
| Model row `qwen3-embedding-0.6b` | type `embedding` on the NPU-lane credential; `default_embedding_model` points at it, so every note save embeds through the XDNA2 lane ([[frank-dragoon]]) |
| Speaker profile `dank-duo` | two speakers, Frank → `voice_id: wilwheaton`, Comrade → `vaati`, `voice_model` = the TTS row |
| Episode profile `dank-daily` | `num_segments: 4`, `speaker_config` = `dank-duo`, outline and transcript LLM = the archon row, briefing that ends "Never invent numbers." |
**Verify reachability before saving**, from inside the container, because both addresses are netns-dependent:
Order that works over REST: credentials → model rows (`POST /api/models` with `credential: <credential id>`) → `PUT /api/models/defaults` → speaker profile (`voice_model: <model id>`) → episode profile (`speaker_config` / `outline_llm` / `transcript_llm`). `POST /api/models/{id}/test` proves each row: the language row answered with a real completion from frank-archon; the TTS row reached frank-tts and came back `unknown voice 'alloy'; available voices: narrator1, narrator2, narrator3` — the app's own test path hard-codes an OpenAI voice, so that message *is* the proof the leg works.
**`POST /api/podcasts/generate` takes profile NAMES, not record ids.** `episode_profile: "dank-daily"` works; the `episode_profile:…` id returns a bare `500 Failed to submit podcast generation job`, and the actual reason (`Episode profile '<id>' not found`) appears only in `podman logs open-notebook`.
**The scriptwriter is frank-archon, not the planned frank-zealot, because two rules collide on the address.** Measured from inside the app's netns 2026-10-19:
- `192.168.0.143:7771` (frank-zealot, a host-netns listener) and `192.168.0.143:7770` (frank-dragoon, a **root**-published port) both refuse the connection; `host.containers.internal:7771` answers. Full census: [[Container Network Namespaces]].
- The app refuses to **store** that one working address: its URL guard rejects any base URL that resolves inside link-local `169.254.0.0/16` (`open_notebook/utils/url_validation.py`, "Link-local addresses are not allowed (metadata endpoints)"), unconditionally — no env var touches it, and `AddHost=` cannot dodge it because the check runs on the resolved address.
`http://192.168.0.235:7770/v1` — another machine, plain RFC1918 — was, when this was first written, the only fleet LLM address that passed both tests; a container name on `dankclaw_default` passes them as well (the embedding credential below is stored that way), but frank-archon stays the scriptwriter for the reason in the next paragraph. Cost: frank-archon runs `--max-num-seqs 2` and the operator session itself lives there, so a render is a slow-down for whoever else is talking to that endpoint, not a stall; `ESPERANTO_LLM_TIMEOUT=180` covers an outline or transcript call comfortably.
**frank-dragoon can write transcripts, but only greedily — and open-notebook cannot ask for greedy.** Measured 2026-10-19 against the live endpoint: `response_format` as `json_schema` or `json_object` returns schema-valid JSON at `temperature: 0` and a **400** at any sampled temperature — halogen decodes greedy under a schema, and because its sampler default is server-side, omitting `temperature` is sampled, not greedy. That 400 is the trap: esperanto's `is_json_schema_unsupported_error` matches the words *not implemented* in it, so `_SchemaFallbackModel` in `podcast_creator/nodes.py` reads it as "this endpoint has no schema support", downgrades to `json_object`, gets the same 400, and **fails the run** — esperanto's own rescue (drop `response_format` entirely) fires only on the different message `'response_format.type' must be 'json_schema'`. The full law is on [[frank-dragoon]]. And the fix is not reachable from here: a model record is `{credential, name, provider, type}` and a model response adds only timestamps (`GET /openapi.json`, `ModelCreate`/`ModelResponse`) — **there is no per-row config field**, no way to attach `temperature: 0` to one row from the UI or the API. Making `frank-dragoon` the writer would therefore need something in the path that rewrites request bodies, which is a heavier tenant than the contention it removes. **frank-archon stays the scriptwriter.** Reachability stopped being the objection on 2026-10-19 — the endpoint joined `dankclaw_default`, which is exactly why the embedding lane below is dialable by name — but no network change reaches the no-per-row-config wall.
**Embeddings run on the XDNA2 lane — provisioned 2026-10-19.** `qwen3-embedding-0.6b` answers `/v1/embeddings` at `http://frank-dragoon-halogen:8731/v1`, the flash container's own bridge address; a container name on `dankclaw_default` resolves to `10.89.1.39`, plain RFC1918, so it clears the URL guard that ruled out `host.containers.internal` above. The lane's shape (one program, the token ceiling, what it costs the Flash model) is owned by [[frank-dragoon]]; the two shapes that bite on the app side are open-notebook's own:
- **`PUT /api/models/defaults` stores the record ID, not the name.** `"default_embedding_model": "qwen3-embedding-0.6b"` is accepted and echoed back, and then every embed dies in the worker with `Model with ID qwen3-embedding-0.6b not found`. The value has to be the row's `model:…` id — the other defaults in that same object are ids, so mirror them.
- **`POST /api/embed` answers 500 while succeeding.** Upstream's `EmbedResponse.command_id` declares `str` and the worker hands it a `RecordID`, so FastAPI raises on serialization *after* the vector is written. The truth is the worker line `Successfully embedded note … in 0.19s`, never the HTTP code. Saving a note (`POST /api/notes`) submits `embed_note` by itself, so that route is only needed to backfill old rows.
Proof from the app's own paths, 2026-10-19: `POST /api/models/{id}/test` → `Embedding dimensions: 1024`; two notes with nothing in common, then `POST /api/search` with `type: "vector"` (the enum is `text` or `vector`; the score field is `similarity`) — "a yellow waterbird beside a glowing beacon tower" put the grebe note at **0.659** and the boiler note at 0.292, and "quarterly depreciation schedule for industrial boilers" inverted it to **0.906** / 0.318. Embed cost 0.19–0.43 s per note. Semantic search, source ingestion and notebook-context questions are live; podcast generation never needed any of it.
**Secrets never go in a command line.** The API's bearer token *is* `OPEN_NOTEBOOK_PASSWORD`, and a `-H "Authorization: Bearer …"` argument is world-readable in `/proc/<pid>/cmdline`; put it in a mode-600 curl config and pass `-K`:
```bash
podman exec open-notebook curl -sS http://frank-tts:7772/health
podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/models
PW=$(grep -m1 '^OPEN_NOTEBOOK_PASSWORD=' /home/dank/Containers/open-notebook/env | cut -d= -f2- | tr -d '"')
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: Bearer %s"\n' "$PW" > "$cfg"
curl -s -K "$cfg" http://127.0.0.1:5056/api/models | head -c 400
rm -f "$cfg"
```
**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). The decision rests on each endpoint's documented request surface ([[frank-zealot]], [[frank-dragoon]]): zealot's is a known-good target for that call shape, dragoon's structured-output behaviour is **unverified**, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
## Speakers and voices: 1–4, and the IDs are free text
A **speaker profile** carries one to four speakers (`docs/3-USER-GUIDE/creating-podcasts.md:27`), each with a personality and a **Voice ID that is typed exactly and passed through to the TTS request untouched** (`:29`) — there is no whitelist, no enum, no validation against a voice list. So the fleet's voice-pack names work verbatim:
| Speaker | Voice ID to type |
|---------|------------------|
| Speaker 1 | `narrator1` |
| Speaker 2 | `narrator2` |
| Speaker 3 | `narrator3` |
| Speaker 1 | `wilwheaton` |
| Speaker 2 | `vaati` |
| Speaker 3 | `dr.kern` |
Those are the pack names frank-tts resolves from `/home/dank/voices` (`scripts/frank-tts.env` header: a voice is real exactly when `/voices/<name>/reference.wav` exists). Two traps, both from upstream's shipped defaults:
- The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `narrator1`/`narrator2` pair (or a third/fourth) rather than editing the seeded ones in place.
- The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `wilwheaton`/`vaati` pair (or a third/fourth) rather than editing the seeded ones in place.
- Two speakers with the same voice are hard to follow (`creating-podcasts.md:89`). Distinct packs per speaker is the point of the 1–4 shape.
## Two buttons that fail against frank-tts, and why that is cosmetic
@@ -92,9 +131,11 @@ Defaults are `TTS_BATCH_SIZE=5` and `OPEN_NOTEBOOK_WORKER_MAX_TASKS=5` (`docs/5-
The clock next to it: `ESPERANTO_TTS_TIMEOUT` (`:91`) is patience **per dialogue line**, not per episode — a three-minute episode is dozens of separate POSTs, and the tail of a long turn on a CPU engine goes past upstream's 300 s default. It is set to **660, one round-trip outside `SHIM_TIMEOUT=600`** in `scripts/frank-tts.env` (frank-tts's own per-request ceiling): nested budgets go inside-out — the same rule that file states for `LLAMA_TIMEOUT` — so a frank-tts give-up arrives labelled as frank-tts's error instead of the worker's own timeout, and re-measuring frank-tts cannot silently strand this number below it. An abandoned request does not cancel the engine process; it only strands the episode in "generating".
What the guard costs, measured on that first render (the deployed shape): 23 dialogue clips, one POST each, **10–11 s per clip** with both knobs at 1 → 301 s of wall clock for 747 s of speech, i.e. **2.5× real time**, box still responsive. That ratio is the budget to hold any future TTS engine to: halve the per-clip time, halve the episode.
## Secrets, and the difference between blank and unset
`OPEN_NOTEBOOK_PASSWORD` unset means **authentication off** (`environment-reference.md:46`) — every API route, including the episode-audio download, answers anyone who can reach the published port. That is a real exposure on a LAN-published service with no vhost and no guard chain in front of it, which is why `deploy_open_notebook` refuses to start either unit unless the seeded host file carries a value for all four secret keys (`OPEN_NOTEBOOK_ENCRYPTION_KEY`, `OPEN_NOTEBOOK_PASSWORD`, `SURREAL_PASSWORD`, `SURREAL_PASS`). `OPEN_NOTEBOOK_ENCRYPTION_KEY` is separately load-bearing: it is **required to save any provider credential**, and changing it later makes stored keys undecryptable (`:44`).
`OPEN_NOTEBOOK_PASSWORD` unset means **authentication off** (`environment-reference.md:46`) — every API route, including the episode-audio download, answers anyone who can reach it. Moving the API to loopback did not remove that exposure, it only removed the LAN from it: any container on `dankclaw_default` dials `open-notebook:5055` by name, so an unset password is an open episode library to every container on this host, guarded or not. That is why `deploy_open_notebook` refuses to start either unit unless the seeded host file carries a value for all four secret keys (`OPEN_NOTEBOOK_ENCRYPTION_KEY`, `OPEN_NOTEBOOK_PASSWORD`, `SURREAL_PASSWORD`, `SURREAL_PASS`). `OPEN_NOTEBOOK_ENCRYPTION_KEY` is separately load-bearing: it is **required to save any provider credential**, and changing it later makes stored keys undecryptable (`:44`).
The tracked file declares the secrets **empty** and the host file supplies them; podman applies `--env-file` arguments in order and the last one wins (verified against the generated `ExecStart` line, 2026-10-18). Reordering the two `EnvironmentFile=` lines in a unit would blank every credential — that is the trap the seam's ordering assertion exists for.
@@ -102,7 +143,7 @@ The database has the same trap in a different spelling: the app reads `SURREAL_P
## Episode destination: interim, and reversible (issue #7)
**Episode files currently exist in two places by design.** The app's own state stays in the `open-notebook-data` named volume; finished audio additionally lands on the host, because `/home/dank/media/podcasts` is bind-mounted over `/app/data/podcasts` (`open-notebook.container:88`). That is the **interim** answer to where episodes live: the destination is an open decision tracked as **Gitea issue #7, labelled `human-only` — agents must not work it.** Until Frank settles it, episode MP3s are ordinary files in his home directory, readable without `podman unshare`, which the database volume would require (measured with the pinned SurrealDB image).
**Episode files currently exist in two places by design.** The app's own state stays in the `open-notebook-data` named volume; everything the render produces additionally lands on the host, because `/home/dank/media/podcasts` is bind-mounted over `/app/data/podcasts` (`open-notebook.container:88`). The first render confirmed the layout: `episodes/<uuid>/clips/0001.mp3 … 00NN.mp3` (one file per dialogue line, written as the worker goes) and `episodes/<uuid>/audio/<uuid>.mp3` (the ffmpeg concatenation). A half-finished render is therefore visible and greppable from the host without `podman unshare`. That is the **interim** answer to where episodes live: the destination is an open decision tracked as **Gitea issue #7, labelled `human-only` — agents must not work it.** Until Frank settles it, episode MP3s are ordinary files in his home directory.
Reversibility is one line: delete the `Volume=/home/dank/media/podcasts:/app/data/podcasts` line and the episodes go back to living solely inside the named volume. Nothing else in the config assumes the bind. See [[Storage Policy]] for the repo's storage conventions this interim state deliberately bends.
@@ -118,10 +159,11 @@ podman pull docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f7
## Rollback
- **Nothing deployed (today's state):** revert the commits. There is no host state to undo — no unit, no container, no volume, no `/home/dank/media/podcasts` directory beyond what the deployer would create.
- **Deployed:** `systemctl --user stop open-notebook.service open-notebook-surrealdb.service`, then move the three files (`open-notebook.container`, `open-notebook-surrealdb.container`, `open-notebook.network`) out of `~/.config/containers/systemd/` into `~/.AGENTTRASH/` — never `rm` — and `systemctl --user daemon-reload`. The generated units disappear with their sources.
- **Take it off the air, in the reverse of the deploy order:** delete the guarded site block and the `@notebook` handler from `dankclaw/Caddyfile` and deploy `caddy`; then in Authentik delete the `notebook-guard` Application and the `notebook-proxy` provider and drop it from the embedded outpost's provider set. A **route with no provider 502s** while a provider with no route is inert — that asymmetry is why the chain unwinds Caddy-first.
- **Stop the app:** `systemctl --user stop open-notebook.service open-notebook-surrealdb.service`, then move the three files (`open-notebook.container`, `open-notebook-surrealdb.container`, `open-notebook.network`) out of `~/.config/containers/systemd/` into `~/.AGENTTRASH/` — never `rm` — and `systemctl --user daemon-reload`. The generated units disappear with their sources.
- **Kept on purpose:** the `open-notebook-data` and `open-notebook-db` named volumes and `/home/dank/media/podcasts`. Episode audio and the notebook database are user data; removing them is a separate, deliberate act, and episode files are host files precisely so they can outlive the container.
- **Homepage:** the card is `wip: "down"` + `showLink: false`, so retiring the service needs no card surgery beyond setting it back to that state.
- **Provider credentials are database rows**, encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`; no repo file mirrors them. Regenerating that key while the database volume survives makes every stored credential undecryptable and forces re-entry — so a rollback that keeps the volume keeps the key file with it.
- **Homepage:** the card test in `tests/open_notebook.bats` pins the live state (`showLink: true`, link = the vhost), so retiring the service edits that test in the same commit as the card flip back to `wip: "down"`.
## Citations
@@ -130,3 +172,8 @@ podman pull docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f7
- Registry (2026-10-18): `lfnovo/open_notebook` tags (`v1-latest`, `latest`, the `-single` set) and manifest-list digests; `surrealdb/surrealdb` `v2` → 2.7.0
- Local measurements (2026-10-18, workstation, pinned digests): `surreal start --help` (credential and bind options carry `[env: SURREAL_*]`); running the pinned SurrealDB digest with env-only credentials (credential init in the log, correct login authenticates, wrong password refused, `/surreal is-ready` → `OK` exit 0, named volume owned by the mapped uid); the v1-compatible Quadlet generator dry-run of all three unit files (two `--env-file` in order, two `--network` in order, named volume + host bind, both health probes, `--pull never`)
- Host evidence (Dankclaw, 2026-10-18, read-only): `ss -ltn` (`127.0.0.1:5055` held by seerr, `0.0.0.0:7771` a host listener, nothing on 7772/8502/5056/8000), `podman ps -a` (no open-notebook container, `dankclaw_default` present), subnet registry from `dankclaw/open-webui/quadlet/open-webui.network`
- Deploy + provisioning (Dankclaw, 2026-10-19): first-boot log — migrations 1→25, `Credentials were provided, and no root users were found. The root user 'open_notebook' will be created`, `api`/`worker`/`frontend` all RUNNING under supervisord; `GET :8502/config` → the `API_URL` value; `GET /api/providers` (the `openai_compatible` modality list); `GET /openapi.json` schemas for `CreateCredentialRequest` / `ModelCreate` / `SpeakerProfileCreate` / `PodcastGenerateRequest`; the POSTs listed above; the note-save worker error `No embedding model configured.`
- Embedding lane (Dankclaw, 2026-10-19): `POST /api/credentials` → `credential:7fdqz09tt0c4xp68x7cm`, `POST /api/models` → `model:p3ki598dydfinu93tbny`, `POST /api/models/{id}/test` → `Embedding dimensions: 1024`; worker lines `Successfully embedded note … in 0.43s` / `0.19s`; the name-not-id failure `Model with ID qwen3-embedding-0.6b not found`; the `/api/embed` 500 body (`EmbedResponse.command_id`, `input_value=RecordID(table_name=command…)`); `POST /api/search` `similarity` 0.6590/0.2921 and 0.9059/0.3176 for the two-note pair; `podman logs frank-dragoon-halogen` showing `POST /v1/embeddings 200` from `10.89.1.21`. Both probe notes deleted afterwards.
- First render (2026-10-19): job `command:9rdojq5ne7dj9caicq20`, `execution_time: 301.21 s`, `episode:7ft03thj98sn0ek9xf29`; `episodes/d5fed29b-33e9-40ac-b46a-e0537c56c256/clips/0001.mp3 … 0023.mp3`; `…/audio/d5fed29b-….mp3` = 11,957,420 bytes, mp3 24 kHz mono, `ffprobe` duration 747.31 s; `GET /api/podcasts/episodes/{id}/audio` → 200 `audio/mpeg` with the bearer, 401 without.
- Upstream code read inside the pinned image (2026-10-19): `frontend/next.config.ts:26-43` (the `/api/*` rewrite and the one-port claim), `frontend/src/app/config/route.ts:122-173` (`/config` priority order; autodetect appends `:5055`), `frontend/src/app/api/_sse-proxy.ts` (SSE proxied with `no-transform`), `open_notebook/utils/url_validation.py` (link-local refusal, `is_blocked_ip`, no environment override anywhere in the file).
- Guard chain (2026-10-19): provider `notebook-proxy` created through `podman exec -i authentik-server ak shell` with the flows copied from `embedded-outpost` (pk 1), Application `Open Notebook`/`notebook-guard`, embedded-outpost provider set 9 → 10; afterwards the outpost logged zero `Failed to fetch outpost from API` and `seerr`/`grafana` still answered 200. Before the route existed, `notebook.franksdank.cloud` resolved to nothing (`000`).
+18 -18
View File
@@ -111,7 +111,7 @@ The Ollama connection is disabled (`ollama.enable=false`).
- Working dir for imports: `/app/backend`; package name `open_webui`
(underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
BEFORE importing any `open_webui` module (import-time hard requirement);
BEFORE importing any `open_webui` module (import-time hard requirement)
the secret lives at `/app/backend/.webui_secret_key`.
- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
module (`podman exec open-webui python3 -c "import sqlite3;
@@ -122,8 +122,8 @@ The Ollama connection is disabled (`ollama.enable=false`).
pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
`status`/`enabled` attribute — use `is_active` from the DB.
- **API auth for verification:** `create_token` lives in
`open_webui.utils.auth` (NOT `open_webui.utils.utils`);
`get_verified_user` requires an `id` claim (not just `sub`);
`open_webui.utils.auth` (NOT `open_webui.utils.utils`)
`get_verified_user` requires an `id` claim (not just `sub`)
`SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
container:
@@ -148,7 +148,7 @@ The Ollama connection is disabled (`ollama.enable=false`).
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
`host.containers.internal`), not a model problem.
**Durability:** the model table + connections are DB-only (not git-managed);
**Durability:** the model table + connections are DB-only (not git-managed)
a fresh data volume loses them — including the public access grant above.
For reproducibility, add an idempotent first-start seed script or document
the setup in the service doc.
@@ -156,7 +156,7 @@ the setup in the service doc.
## API access (auth-gated)
Every `/api/*` endpoint requires an authenticated session (SSO login JWT)
or a per-user API key (`auth.enable_api_keys=true`; keys are minted per
or a per-user API key (`auth.enable_api_keys=true`; keys are minted
account under Profile → API Keys). Anonymous web visitors get the `pending`
role (`ui.default_user_role=pending`), which cannot chat. Verified
2026-10-07 from loopback `:3005` and over the public vhost:
@@ -166,13 +166,12 @@ bogus bearer token); only `/api/config`, `/api/version` and `/health`
`POST /api/chat/completions` all return **401** with no key (also with a bogus bearer
token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of
them expose the model list. An external program needs an account + API key — there is
no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`,
no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`
`/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and
`POST /api/chat/completions` → 401; `/api/config` additionally publishes
`features.enable_login_form=false`, `features.enable_signup=false` and
`oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside.
## Web search
Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
@@ -186,7 +185,7 @@ Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
The Anubis→Authentik guard chain is a browser mechanism (proof-of-work +
session flow); a plain HTTP client cannot solve it, and the internal port
bypasses Caddy entirely.
- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`,
- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`
a hardcoded Python constant — no config knob). The `pass_ip` list is checked
FIRST and short-circuits the whole botdetection chain, and OpenWebUI's bridge
egress (`10.89.x`) is in `pass_ip`, so the cap never applies. Verified: an
@@ -249,7 +248,6 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
restore the password form: set `ui.enable_login_form` back to `true` in the
Admin Panel (or DB) and drop the two env vars from the quadlet.
## Boot persistence (Quadlet)
The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
@@ -292,11 +290,14 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
## Text-to-speech (frank-tts)
TTS is the fleet's one OpenAI-compatible speech endpoint —
`http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url`
in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That
TTS is the fleet's one OpenAI-compatible speech endpoint. The config DB
`audio.tts.*` is set to engine `openai`, `audio.tts.openai.api_base_url`
`http://192.168.0.143:7772/v1`, `audio.tts.voice` `wilwheaton` (re-keyed
2026-10-20 from the retired `narrator1`; 7772 is a published port so
`host.containers.internal` also reaches it, but the LAN IP is the canonical
on-host form). That
port is the rootless quadlet `frank-tts` (this repo:
`dankclaw/voiceservice/quadlet/frank-tts.container`,
`dankclaw/voiceservice/quadlet/frank-tts.container`
`scripts/frank-tts.env`), and its shim resolves `voice=<name>` against
`/home/dank/voices/<name>/reference.wav` — a name without a reference is a
400 listing the voices that exist, never a silent fallback.
@@ -308,9 +309,9 @@ ships an in-page Kokoro implementation (chunk carrying a frozen voice table
`onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype*
setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
`browser-kokoro` is chosen the page synthesizes locally and rejects any
voice outside its fixed pack — so `narrator1` fails with
`Voice: "narrator1" not found. Should be one of: af_heart, …` **without a
single request reaching the server**. Personal audio settings are per
voice outside its fixed pack — so `wilwheaton` fails with
`Voice: "wilwheaton" not found. Should be one of: af_heart, …` **without a
single request reaching the server**. Personal audio settings are
browser: no `user.settings` row and no container env carries an audio
override, yet the toast still appears ⇒ the choice lives in that browser's
own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**,
@@ -325,7 +326,6 @@ the failing click — that contrast is the instrument: a server-path failure
leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves
neither.
## Retired instances
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -355,7 +355,7 @@ neither.
began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
id-keyed objects had to move in the same window as the alias flip:
`model.id`+`name` (`frank-zealot` → `frank-zealot`, `params`
`{"context":{"length":200704}}` and the `meta.description` kept),
`{"context":{"length":200704}}` and the `meta.description` kept)
`access_grant` (`resource_type='model'`, `resource_id`, principal
`user`/`*`, permission `read`), and `openai.api_configs` per-backend
`model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1
+1 -1
View File
@@ -88,7 +88,7 @@ never enter this repo. See [[Gitea]] and [[Authentik]].
## Boot persistence (Quadlet)
Both units carry `Restart=always` (`gitea-postgres.container:28`,
Both units carry `Restart=always` (`gitea-postgres.container:28`
`authentik-postgres.container:38`) and `[Install] WantedBy=default.target`
(`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed
under `~/.config/containers/systemd/`, and report `generated` / `active` with
@@ -143,7 +143,7 @@ engines:
disabled: false
```
Key per-engine keys (each overwrites the engine-module default; see
Key per-engine keys (each overwrites the engine-module default
`settings_engines.html`):
| Key | Effect |
@@ -164,7 +164,7 @@ Key per-engine keys (each overwrites the engine-module default; see
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
`google2` still returns results, and vice-versa.
**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
**Parser caveat (read before fiddling with `parsers`):** the modern `google`
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
[scrapy_html]` under `name: google` mostly does nothing. To make Google
@@ -376,14 +376,14 @@ done
**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. It answers 200 on this instance today
(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`,
(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`
`categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
[[SearXNG]]). `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.
**3. `check.searx.space`** — the hosted official checker probes every engine of
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
plain HTTP probe misses) and posts results on searx.space. Add your instance via
plain HTTP probe misses) and posts results on searx.space. Add your instance
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
to reach the SearXNG port.
@@ -417,10 +417,10 @@ misleading).
already replaced — checked 2026-10-18; see [[SearXNG]]). It is
also immune to the shared-VPN-exit-IP blocking this page is about: it
authenticates, so reputation of the exit IP is irrelevant.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to
exactly which engines are down and why. Confirm the running SearXNG version
(check whether `parsers`/`rate_limit` exist in the pinned build).
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds
upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
(`:27`), and retries once (`:32`). This addresses
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
+16 -17
View File
@@ -108,7 +108,7 @@ engines:
disabled: true
```
The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas);
The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas)
the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked
`secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is
censused below.
@@ -154,7 +154,7 @@ what keeps the `trusted_proxies` entry meaningful.
## Unblocking Google & Search Providers
The current request policy allows a modest amount of VPN latency (`5.0s` per
The current request policy allows a modest amount of VPN latency (`5.0s`
request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting
retries to one (`:32`). This can recover slow
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
@@ -169,10 +169,10 @@ upstream request volume. SearXNG **suspends** offending engines — which is wha
up as "Google went unavailable". The default suspension is shorter than the folklore:
see the table in [[SearXNG Providers]]. This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
if Google still re-blocks — route the Google engines at a clean-IP proxy via
if Google still re-blocks — route the Google engines at a clean-IP proxy
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
root cause, the suspend-model table, the `engines:` config, and the
`/json` verification recipe.
@@ -212,17 +212,16 @@ root cause, the suspend-model table, the `engines:` config, and the
(`searxng.container:21`), and that is one of the variables SearXNG substitutes at
load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a
config change, out of scope for docs.
- Keep the valkey cache container on the network SearXNG can actually reach — see
- Keep the valkey cache container on the network SearXNG can actually reach —
[[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's
netns), so the network that matters is the one Gluetun and `searxng-valkey` share.
## Kagi (paid API engine)
Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native
`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`),
`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`)
which sets `Authorization: Bearer <key>` (`kagi.py:103`) and issues
`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`),
`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`)
and declares `require_api_key: True` (`kagi.py:63`).
- **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
@@ -231,10 +230,10 @@ and declares `require_api_key: True` (`kagi.py:63`).
`enabled: true`, categories `general, web` (2026-10-18).
- **Cost control**: `paging: false` — exactly one API credit per aggregated
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0)
(`shared/searxng/settings.yml:74-77`; upstream paging default is `True`,
(`shared/searxng/settings.yml:74-77`; upstream paging default is `True`
`kagi.py:67`, so the `false` here is a real override).
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
`deploy-remote.sh` bakes the real key into the live settings.yml via
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`
`deploy-remote.sh` bakes the real key into the live settings.yml
`scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
(owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are
the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`).
@@ -262,15 +261,15 @@ and declares `require_api_key: True` (`kagi.py:63`).
- 262 engines in the build, **75 enabled**, categories of the enabled set led by
`other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7).
- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`,
`github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`,
`arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`,
`wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`,
- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`
`github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`
`arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`
`wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`
`youtube`, `vimeo`, `dailymotion`, `bing videos`.
- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`,
- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`
`brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`.
- The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target
engines that **no longer exist in this build** — `/config` lists no `startpage`,
engines that **no longer exist in this build** — `/config` lists no `startpage`
`startpage images` or `startpage news`, so those overrides are no-ops kept as
intent. *(labelled 2026-10-18)*
@@ -144,8 +144,8 @@ frank-dragoon `health=200` throughout.
The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging
directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only
`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`,
`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`,
`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`
`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`
`20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`.
The old swapfile has therefore been purged; nothing on the host still holds it.
@@ -23,7 +23,7 @@ voice is how a wrong-voice endpoint stays wrong for weeks.
state that stays out of the repo, mounted read-only at `/voices` by the façade
unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in
`scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:**
arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]].
arm A `qwentts.cpp`, decided in .
Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade
container runs as uid 10000 and reads the mount from outside its own uid map.
@@ -137,7 +137,7 @@ invariant to every crop in the tracked pack list.
1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop
offset is a claim about one exact file, so the digest goes in the env with it.
2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once:
`ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`,
`ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`
build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest
on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the
`.rvq` falls under the floor.
@@ -188,15 +188,15 @@ loud, not a bug. Work through it in order:
1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec`
prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , …
codebooks @ … Hz` on every run, and the pinned tree's
`tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`;
`tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`
`codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`.
2. Decide whether the new number is intended. A changed `enc_dim` means the old
`.spk` files are unreadable by the new engine, so **every** pack must be
rebuilt, not just the failing one.
3. Update the four arithmetic lines in `scripts/voice-packs.env`
(`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`,
(`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`
and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and**
`VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`,
`VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`
`VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes
without moving the commit is the mistake to avoid.
4. Rebuild all three packs with `--force` (the retired directories keep the old
@@ -226,7 +226,7 @@ loud, not a bug. Work through it in order:
narrator1 and narrator2). The builder's first real run is therefore a
migration, and it should be a `--force` run per voice so the old directories
are retired rather than left half-updated.
- The three source files (`/home/blank/audiosample.mp3`,
- The three source files (`/home/blank/audiosample.mp3`
`audiosample2.m4b`, `audiosample3.m4b`, digests pinned in
`scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the
builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so
+1 -1
View File
@@ -80,7 +80,7 @@ image's own nginx (the desktop), so the content server runs on internal 8090.
**Auth model (calibre 9.x):** users live in a SQLite userdb whose `pw` column
is **plaintext** — the file is mode 600. The single admin account is
bootstrapped once by the deploy handler
(`deploy_calibre_server` in `scripts/deploy-remote.sh`) via
(`deploy_calibre_server` in `scripts/deploy-remote.sh`)
`calibre-server --manage-users -- add`, the password piped on stdin with
`printf` (a trailing newline from `echo` lands in the stored password and
breaks every login). The desktop's basic auth (`CUSTOM_USER`/`CUSTOM_PASSWORD`)
@@ -83,7 +83,7 @@ The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum co
## Volumes
Live deployment: the whole LUKS array is served at URL `/w` via
Live deployment: the whole LUKS array is served at URL `/w`
`Volume=/mnt/entropy:/w:ro` in the quadlet unit ([[Entropy Volume]]) —
read-only **at the mount level**, so no containerized consumer can write into
the array through this path regardless of account permissions. The ac edition
@@ -111,7 +111,7 @@ convention; see [[#Command-line interface (ac edition)]] above.
```
`${COPAY_ACCOUNT}`/`${COPAY_PASSWORD}` are **never** written into the unit:
they live in the host-local `/home/dank/Containers/copyparty/env` (0600),
they live in the host-local `/home/dank/Containers/copyparty/env` (0600)
loaded via `[Service] EnvironmentFile=` so systemd expands them into
`Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD}` at start. The password still
lands in the container argv — visible to anyone who can run
@@ -178,6 +178,6 @@ Upgrade to v1.20.19+ to cover all known advisories. The dirkey fix (v1.20.17) ma
## Citations
- GitHub Security Advisories, `9001/copyparty` —
https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19);
https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19)
each row's patched version read from `first_patched_version` in the
`https://api.github.com/advisories/<GHSA-id>` response.
@@ -54,5 +54,5 @@ The following services ran on Dankserver and are now retired from this stack:
## See Also
- [[index]] — repository index
- repository index
- [[Network Architecture]] — routing and proxy
+3 -3
View File
@@ -14,7 +14,7 @@ serving music, movies, and shows over LAN. It replaced the earlier research
brief; it runs alongside [[Plex]], which remains the primary server for some
clients.
**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`,
**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
@@ -67,7 +67,7 @@ user to verify the password.
- LDAP provider `Jellyfin LDAP` (Base DN `dc=franksdank,dc=cloud`, bind/search
mode `direct`) + application `Jellyfin`.
- LDAP outpost `Jellyfin LDAP Outpost` → container `authentik-outpost-ldap`
(quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`),
(quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`)
publishing `389`/`636` on the Dankclaw host for cross-host binds.
- Service account `svc-jellyfin-ldap` (DN
`cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud`).
@@ -80,7 +80,7 @@ user to verify the password.
**Jellyfin side** (plugin config, set via admin → Plugins → LDAP-Auth):
- Server `192.168.0.143:389` (plain LDAP over the trusted LAN), Base DN
`dc=franksdank,dc=cloud`, bind user = the service-account DN, search
attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`,
attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`
`CreateUsersFromLdap=true`, all folders enabled.
**Gotcha — the bind credential is a PASSWORD, not a token.** The LDAP bind is
+3 -3
View File
@@ -4,7 +4,7 @@ description: "Plex Media Server is a **containerized media server on Dankserver*
tags:
- dankserver
- plex
timestamp: 2026-09-15
timestamp: 2026-10-21
---
# Plex
@@ -24,7 +24,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Item | Value |
|------|-------|
| Runtime | podman quadlet, `plex.service` (user manager) — migrated from the native `plexmediaserver.service` OS-package unit (2026-09) |
| Image | `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (pinned; verified == the ghcr `latest` manifest digest at pin time) |
| Image | `lscr.io/linuxserver/plex:1.43.5.11029-6bf8af3f0-ls329` (pinned; verified == the registry's `latest` manifest digest at pin time: `sha256:8f655bad…e9e2334`) |
| Data root | `/var/lib/plexmediaserver` bind-mounted **in place** as `/config` (RW) — the migration moved zero bytes |
| Media | `/mnt/entropy` mounted at the same absolute path **read-only** — every library path in the PMS database stayed valid, zero re-scan |
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
@@ -60,7 +60,7 @@ root runs as the `dank` user, and the deployer adopts the data root with
## Integration points
- **Caddy:** Plex is exempt from the Anubis bot challenge
(plain-HTTP legacy clients cannot solve JS challenges) — see
(plain-HTTP legacy clients cannot solve JS challenges) —
[[Anubis]] "Exempt by construction".
- **Tautulli** ([[Tautulli]]) is the companion: it polls PMS on the appnet
for statistics and notifications and makes no sense without Plex.
+2 -2
View File
@@ -32,7 +32,7 @@ Anubis → Authentik chain ([[Caddy]] routes it post-auth to
## Architecture
Current RomM releases require MariaDB plus Redis (or Valkey) for the task
queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit),
queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit)
not valkey.
```mermaid
@@ -114,7 +114,7 @@ scan:
## Deployment (Quadlet)
Since 2026-09-01 the trio runs as repo-tracked quadlet units, deployed with
`./scripts/deploy.sh dankserver --service romm` (any of `romm`,
`./scripts/deploy.sh dankserver --service romm` (any of `romm`
`romm-mariadb`, `romm-redis` triggers the full ordered cutover):
- Units: `dankserver/romm/quadlet/{romm,romm-mariadb,romm-redis}.container`
@@ -45,14 +45,14 @@ Verified: `msg status` returns live JSON state; unknown panel ids return an
error listing all valid panel ids (clipboard, control-center, launcher,
polkit, session, setup-wizard, test, tray-drawer, wallpaper).
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`
`media next|previous|playPause` (graceful no-op, rc 0, with no active
player), `brightness-up [target] [step]`, `brightness-down`,
`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
player), `brightness-up [target] [step]`, `brightness-down`
`settings-open [context]`, `settings-toggle [context]`, `config-reload`
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`
`screenshot-fullscreen`.
**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
@@ -81,9 +81,9 @@ Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
synced. Missing/invalid file → silent fallback to the builtin palette.
Palette JSON shape = the community file's shape: `dark`/`light` maps of
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`
`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`
`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
*generator* format, not this one. Copy an existing community JSON and mutate
@@ -114,7 +114,7 @@ Procedure:
`noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
`color-scheme-get`.
5. Re-render every configured template: `noctalia msg templates-apply`.
Regenerates `~/.config/ghostty/themes/noctalia`,
Regenerates `~/.config/ghostty/themes/noctalia`
`~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
starship, etc. Symlinked-into-dotfiles paths show up as modified files
worth committing.
@@ -216,7 +216,7 @@ Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
scale, transform, logical pos/size) and `niri --version`. Output **names are
connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
the EDID strings printed before them are decoration. On blankpad the live
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`
and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
repo path (that is what gets committed); check with `ls -ld`.
@@ -38,7 +38,7 @@ live call at a time instead of guessing payload shapes:
- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
- `DefaultMetadataProfileId must be > 0` → fetch real ids from
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`)
`GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
profiles.
- `targetType null` → configContract not registered in that build (this
@@ -87,7 +87,7 @@ settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
pipeline hides a miss and you conclude "empty field" when the grep itself
failed.
4. Decision framing: when posture = gated-with-zero-users, present it as a
decision menu (None behind the front-door wall vs External/authentikarr),
decision menu (None behind the front-door wall vs External/authentikarr)
not a silent change — first-account-wins means whoever logs in first owns
the instance.
@@ -70,7 +70,7 @@ http://prowlarr:9696/{indexerId}/download?link=...&file=... (torrent handoff)
The global API key is in `config.xml` (`<ApiKey>`); per-application keys are
only used by Prowlarr's own sync calls, NOT by the feeds. The arr download
clients authenticate to qBittorrent with the permanent WebUI password stored
in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — see
in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver —
[[qBittorrent]] for why that password must be pinned in the conf file.
**Readarr caveat:** Readarr is abandoned upstream; its qBittorrent client
@@ -97,13 +97,13 @@ successor; the indexer row itself is healthy.
- Categories in the indexer row must be a subset of what the feed's `caps`
advertises; extras log "does not support the following categories" and
some feeds reject them.
- **A freshly created indexer row defaults to all-off.** If `enableRss`,
- **A freshly created indexer row defaults to all-off.** If `enableRss`
`enableAutomaticSearch`, or `enableInteractiveSearch` aren't set, the
app logs `Searching indexers for [X]. 0 active indexers` (Lidarr,
verified). Search proves TV + music chains 2026-08-30: Sonarr returned
76 live releases for an anime term, Lidarr 18 for YOASOBI – THE BOOK,
all via the Prowlarr-Nyaa feed.
- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`);
- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`)
GET responses carry `fields: []` — real settings values live in
`prowlarr.db` → `Applications.Settings` JSON (read-only inspection).
- Instance name/`InstanceName=Prowlarr` in `config.xml` does NOT prefix the
@@ -13,7 +13,7 @@ qBittorrent is the torrent download client for the *arr stack. The stack runs on
not on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr,
and Readarr via category tagging.
**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`,
**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`
published by the [[Gluetun]] unit because qBittorrent shares that container's
network namespace (a netns-sharer publishes no ports of its own).
@@ -121,7 +121,7 @@ Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
`podman machine ssh`.
**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
(NOT core — `core` is the podman-machine VM user, reached only via
(NOT core — `core` is the podman-machine VM user, reached only
`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
low MaxAuthTries: when password-bootstrapping use `-o
+80 -39
View File
@@ -3,70 +3,111 @@ type: reference
description: "Alloy is Grafana's log and telemetry shipper."
tags:
- infrastructure
timestamp: 2026-09-11
timestamp: 2026-10-20
---
# Alloy
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below.
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life
as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It
forwards the host journal — including the rootless Quadlet containers' logs —
to Loki. Unlike its observability siblings it runs as a **root system service on
the host network**, because a rootless container cannot read the journal (see
*Why root* below).
**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest
**Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345
(loopback) — **Docs:** https://grafana.com/docs/alloy/latest
## Facts
| Item | Value |
|------|-------|
| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) |
| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) |
| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` |
| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) |
| Inputs | host journald socket, podman container logs, `/var/log` |
| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) |
| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) |
| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" |
| Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) |
| Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) |
| Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy |
| Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish |
| UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) |
| Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) |
| Output | `http://127.0.0.1:3100/loki/api/v1/push` |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) |
| Successor to | Promtail (EOL 2026-03-02) |
| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs |
## Why Alloy and not Promtail
## Why root (the load-bearing decision)
Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`).
A rootless Alloy delivered **zero** lines, silently and with a green health
check. Two independent causes, both proven on the host 2026-10-20:
1. **The journal is not readable by a rootless container.** The files are
`root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a
rootless container has no supplementary groups, so opening them returns
`DENIED`. `loki.source.journal` then sees an empty view and the counter just
sits at 0 — no error logged.
2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless
Quadlet containers log via the journald driver into `user-1000.journal`; the
old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`)
point at a root-storage layout that does not exist here.
**Root fixes both at once.** As root the container opens every journal file
under the machine-id directory — `system.journal` *and* `user-1000.journal` —
so a single `loki.source.journal` carries host services and the rootless
containers' logs together. Validated with a throwaway root probe before writing
the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero
drops**, and Loki's label set went from empty to populated.
### The two mounts that make it work
- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal
directory by the *local* machine id. Without it the container opens nothing and
the journal source reads exactly 0 lines — silently, health green. This single
mount is the whole difference between a working and an empty pipeline; do not
remove it. (A root probe without it reads 0; with it, 31,825.)
- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge
container — rootless or root — can reach the host's loopback, so Alloy must
share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds
`127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before.
## Configuration
`shared/alloy/config.alloy` is the source of truth:
- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`).
- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`).
- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
`shared/alloy/config.alloy` is the source of truth — one `loki.source.journal`
(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write`
endpoint. There is deliberately **no** container-file source: container stdout is
in the journal.
## Volumes
| Container path | Host path |
|----------------|-----------|
| `/etc/alloy` | `/opt/alloy/config` |
| `/run/systemd/journal/socket` | host journald socket (read-only) |
| `/var/log` | host logs (read-only) |
| `/var/lib/containers` | podman storage (read-only) |
| `/etc/alloy` | `/opt/alloy/config` (ro) |
| `/etc/machine-id` | host machine id (ro) — **required** |
| `/run/systemd/journal/socket` | host journald socket (ro) |
| `/var/log/journal` | persistent journal files (ro) |
| `/run/log/journal` | runtime journal files (ro) |
## The UI port is published on purpose
## The UI stays on loopback
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
## Delivery state (2026-10-18): healthy, shipping nothing
The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
Alloy 1.x refuses to configure the `ui` service from a config block, so the UI
address is set on the command line instead. Under the old rootless unit that
meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under
`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's
loopback, reachable from the host and from nothing off-host. Alloy has no
authentication, which is exactly why it is never published wider. The homepage
card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`.
## Gotchas
- The journald socket mount is required for journal scraping.
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up.
- **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes.
This is the failure mode that hid the broken pipeline; check
`loki_source_journal_target_lines_total` and the Loki label query, never
`/healthy`.
- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a
rootless user unit — a system unit cannot order against a user unit. Alloy
retries its push until Loki answers, so ordering is unnecessary.
- **The image must be in the ROOT store.** The deployer builds rootless and
mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a
rootless tag fails at create.
- One Alloy per host; do not run a single instance that tries to reach both
hosts' logs.
## See Also
@@ -93,7 +93,7 @@ and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
it is exempt by absence — and therefore unauthenticated on the LAN at
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`
`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
(`dankclaw/Caddyfile:283-423`).
@@ -140,7 +140,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
instead of allowed, the browser's fetch throws and it renders its own
error page (iOS/macOS browsers phrase it as "the request failed and the
interceptors did not return an alternative response") instead of the
login flow. Verify with a JS-capable client end-to-end — mint via
login flow. Verify with a JS-capable client end-to-end — mint
`make-challenge`, solve, submit with browser-exact query params — before
declaring a challenge chain healthy.
7. **Single-file bind mount + git pull = stale policy.** The policy is
@@ -158,7 +158,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
answers any request lacking `X-Real-Ip` with a 500 whose log line reads
`[misconfiguration] X-Real-Ip header is not set`. The real chain always
sets it (every guarded Caddy listener carries `header_up X-Real-Ip
{remote_host}` — the bats seam enforces the line), so users never see
{remote_host}` — the bats seam enforces the line), so users never
this. But any direct probe of `:8923` (scratch container, curl, a new
reverse-proxy hop) MUST set the header or it looks like a policy failure.
Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`.
@@ -23,8 +23,8 @@ read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
members in `authentik_outposts_outpost_providers`), all with
`internal_host = http://caddy:8017`, all carrying the **same three flow
UUIDs** (`default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
UUIDs** (`default-authentication-flow`
`default-provider-authorization-implicit-consent`
`default-provider-invalidation-flow`), and each backed by an Application
(the `slug` column below):
@@ -67,7 +67,7 @@ Two provisioning rules, each verified live by an outage:
reload** — including hosts whose providers were fine (verified 2026-09-02
when three providers provisioned without any flows took down every guarded
vhost after a server restart). Backfill all three flows (copy from pk 1:
`default-authentication-flow`,
`default-authentication-flow`
`default-provider-authorization-implicit-consent` (the implicit-consent
authorization flow, `default-invalidation-flow`) and the retry loop
recovers on its next 3-second cycle. The silent killer: while the outpost
@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
appears. Create the Application with `provider_id` set (the UI does this
automatically; raw API/ORM drives must do it explicitly).
## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -94,29 +94,38 @@ implementation diverges from upstream in ways that bit us; all verified live:
contains `"authorization_code"` (enum value with underscore), the token
endpoint rejects every exchange with `invalid_request`.
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
`parse_token_request` see no `grant_type` → `unsupported_grant_type`;
`parse_token_request` see no `grant_type` → `unsupported_grant_type`
POST-body client secrets are not accepted — authenticate the confidential
client with `Authorization: Basic base64(client_id:client_secret)`.
- **Provider resolution is by `client_id` query param** on the global
`/application/o/authorize/` route; per-slug authorize paths do not exist in
this build. Token/userinfo live at `/application/o/token/` and
`/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
`PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
`BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
becomes the *expression source* instead of the context; every call raises a
swallowed `PropertyMappingExpressionException` and the claim is skipped.
Scope-grant filtering has the same class of bug: `ScopeMapping` has no
provider FK (linkage lives in the `ProviderPropertyMapping` through model),
so `filter(provider=...)` always yields an empty queryset and tokens asked
to carry scopes get zero. Net effect: userinfo/introspection answers with
`{sub}` only — email/groups/entitlements claims cannot be delivered by this
build until the evaluator is patched or upgraded. Clients consuming this
provider must key identity off `sub`.
- Scope mappings for this provider exist — `grafana-email-scope` and
`grafana-profile-scope` are both rows in `authentik_core_propertymapping`
(read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
the evaluator bug is fixed; no config change is needed then.
`/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
- **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
object has no attribute 'slug'`.** A provider created via raw `ak shell`
`create()` leaves `authorization_flow` unset; the authorize view then crashes
building the flow planner (`flows/planner.py`) and returns "Server Error —
Powered by authentik" before any claim is built. This was previously
mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
`authorization_flow = default-provider-authorization-implicit-consent` (the
flow every other provider in this instance already uses).
- **A scope mapping that returns a non-dict emits no claim.** authentik merges a
granted scope mapping's evaluated result into the token **as a mapping**: a
dict contributes its keys as claims; a bare string is keyed under the
mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
bare string `user.email`, so no `email` claim landed — the evaluator was
never broken (the dict-valued `grafana-profile-scope` mapping has always
delivered, and the stock OpenID mappings return dicts). Fix:
`return {"email": user.email, "email_verified": True}`. With no email, Grafana
v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
InternalError".
- **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
- **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
clients. Both IdP fixes are idempotent in
`scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
the provider or a blueprint import, which revert them). Grafana's half plus the
verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].
## LDAP provider objects (provisioned via `ak shell` ORM)
@@ -156,7 +165,7 @@ Hard-won rules (the first is now contested by the live database):
and retest.
- `ak shell` import path for the provider model in 2026.8:
`from authentik.providers.ldap.models import LDAPProvider`.
*(unverified 2026-10-18: importing it outside a configured Django shell fails
- (unverified 2026-10-18: importing it outside a configured Django shell fails
with `ImproperlyConfigured`, and no `ak shell` was driven against the live
auth service for this check)*
@@ -188,13 +197,13 @@ is a `password` field on the `default-user-settings` prompt
Two ORM gotchas: `Flow.stages.add()` fails on this through model
(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
MTI), so create the `FlowStageBinding` row directly with an explicit `order`;
MTI), so create the `FlowStageBinding` row directly with an explicit `order`
and a prompt field is a `Prompt` row linked through `PromptStageField`
(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).
**A blank password must not set an empty password.** `set_password("")` would
make the account log in with no password, so a validation policy
(`default-user-settings-no-empty-password`, bound to the prompt via
(`default-user-settings-no-empty-password`, bound to the prompt
`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
is `""` and allows it when set. Verified with the `PolicyEvaluator`
(empty → deny, set → allow, absent → allow) and end-to-end by driving
@@ -213,7 +222,7 @@ three `*_flow_id` columns; the proxy table is
`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
table in 2026.8), the LDAP table is
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`
`authentik_core_application` attaches to a provider through `provider_id`
(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
@@ -15,20 +15,20 @@ application-level authentication with its proxy outpost. Not wrapped: Gitea,
OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
they handle their own session. Grafana is the exception that is both: it is a
guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`
`dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]).
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`
2026-10-18)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack: five quadlet units, no compose.** `authentik-postgres`,
`authentik-server`, `authentik-worker` (`deploy_authentik`,
- **Stack: five quadlet units, no compose.** `authentik-postgres`
`authentik-server`, `authentik-worker` (`deploy_authentik`
`scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`
`deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`
`scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
`active`, `Restart=always`, `WantedBy=default.target`, with linger on
(`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
@@ -36,7 +36,7 @@ relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.
**Split off from this page:** [[Authentik Processes]] owns the per-container
facts for `authentik-postgres`, `authentik-server` and `authentik-worker`;
facts for `authentik-postgres`, `authentik-server` and `authentik-worker`
[[Authentik Outpost Reference]] owns the provider/flow/token lookup material —
the live proxy-provider inventory, the proxy-provider provisioning rules, the
OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow
@@ -136,7 +136,7 @@ routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.
The live proxy-provider inventory (the per-vhost `external_host` → app mapping),
The live proxy-provider inventory (the per-vhost `external_host` → app mapping)
the two provisioning rules the outpost enforces, and this fork's OAuth2
relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
@@ -144,7 +144,7 @@ relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
single-platform digest**
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
@@ -159,7 +159,7 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
.digest'` and pin that digest.
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
(`PublishPort=389:3389`, `636:6636`,
(`PublishPort=389:3389`, `636:6636`
`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
@@ -196,7 +196,7 @@ every outpost session (see the ORM playbook).
core quadlets, starts postgres health-gated (`wait_container_healthy
authentik-postgres 120`), then the server (300 s health wait) and the worker.
The outposts are separate handlers: `--service authentik-outpost` and
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`
dispatch at `:1579-1588`).
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
creates the superuser (email + password) and generates the DB schema.
@@ -210,7 +210,7 @@ every outpost session (see the ORM playbook).
4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
the app with the correct redirect URI, and paste the `client_id` /
`client_secret` into the app's SSO settings. The relying parties that exist
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14)
**Gitea has no provider row at all** (live provider table, 2026-10-18) — it
authenticates itself and stays off the chain by design
(`dankclaw/Caddyfile:110-120`).
@@ -243,7 +243,7 @@ every outpost session (see the ORM playbook).
The embedded outpost does not re-evaluate group membership per request. Each
completed authorize writes one row to `authentik_providers_proxy_proxysession`
whose `session_data` JSON holds the claims (`groups` included) that get served
for that browser session until the row expires (observed TTLs 1–24 h per
for that browser session until the row expires (observed TTLs 1–24 h
row). A session authorized **before** a user was added to a group keeps
serving the old group list — Caddy-side RBAC header gates then 403 despite
correct membership, and no authentik event is generated (the 403 happens at
@@ -267,6 +267,12 @@ The silent re-auth path was proven end-to-end: after deleting
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
state and passed with no login prompt (2 tests, 2.9 s).
## Login-session lifetime (how often you re-authenticate)
How long a login stays valid is the **`default-authentication-login` stage's `session_duration`** (`UserLoginStage`) — not a global setting, and not `SESSION_COOKIE_AGE` (that is the *pre-login* session). Raised to **`days=7`** on 2026-10-20 (was stage value `0` = the 24 h framework default). It is an **absolute** window, never sliding: authentik implements `SESSION_SAVE_EVERY_REQUEST` nowhere, so activity does not extend it and no supported knob makes it one.
**`session_duration` is a STRING** in authentik's idiom (`"days=7"`, `"seconds=30"`, `"hours=12"`). A Python `timedelta` stores `"7 days, 0:00:00"`, which `authentik.lib.utils.time.timedelta_from_string` rejects with `ValueError` and login then crashes computing the expiry — so set it with **`scripts/authentik-set-login-session-duration.sh`** (validates the form first). Any non-zero value applies to *every* login (the "remember me" box only adds `remember_me_offset` on top) and a positive `set_expiry` also overrides `SESSION_EXPIRE_AT_BROWSER_CLOSE`, so closing the browser no longer logs you out; **existing sessions keep their old expiry** — delete the user's `authentik_core_session`/`…proxysession` rows and re-login to force it now. This is **DB state, not repo-managed**: a blueprint re-import or rebuild reverts the stage to `0`, so re-run the script after one. The other two login stages (`default-source-*`) drive external-source/enrollment logins and stay at the framework default.
## Guard-chain flip recipe (adding a guarded vhost)
Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
@@ -309,7 +315,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
000 (connection refused) — the post-auth listener lives in the caddy
container's netns; probe inside it or grep the config. Non-browser clients
cannot solve the JS challenge — before flipping, audit service-to-service
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
traffic: ingestion/push paths must target internal addresses (`loki:3100`)
never a public hostname of a guarded service. Exempt-by-construction
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
endpoints, services with their own complete auth — document the exemption
@@ -336,14 +342,14 @@ EOF
`sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
the exact banner text and the 229-object count — no `ak shell` was driven
against the live auth service for this audit)*.
- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
- Import paths (each wrong guess costs a failed exec): `User`, `Group`
`Application`, `ApplicationEntitlement`, `PropertyMapping` →
`authentik.core.models`; `ProxyProvider` →
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
`LDAPProvider` → `authentik.providers.ldap.models`;
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`)
`LDAPProvider` → `authentik.providers.ldap.models`
`authentik.core.models.application` does NOT exist as an import path; there
is no `authentik.policies.group` module in this build.
*(unverified 2026-10-18: these module paths were not re-executed — importing
- (unverified 2026-10-18: these module paths were not re-executed — importing
outside a configured Django shell fails with `ImproperlyConfigured`)*
- The live database's table and column names (the 2026.x renames, the MTI parent
keys, the read-only `psql` audit recipe) are reference lookup:
@@ -376,7 +382,7 @@ downstream —
`category: user` panels from `services.json`.
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
**"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
`Open WebUI` application (`authentik_core_applicationentitlement`,
`Open WebUI` application (`authentik_core_applicationentitlement`
2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
expression were not read; the mapping itself is not visible under that name
@@ -392,7 +398,7 @@ downstream —
artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
and every one carries the `<user>@franksdank.cloud` address the template
prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
*(unverified 2026-10-18)*. Template: `username` +
- (unverified 2026-10-18)*. Template: `username` +
`<user>@franksdank.cloud` + display name + `is_active=True` + the password
set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
`openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
@@ -400,7 +406,7 @@ downstream —
the row carried no MFA device, token, entitlement, or proxy session, and
events survive a user delete because `Event.user` is a denormalized JSON
field, not a FK).
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
Plex, Seerr, Marreta) need no per-user group: any authenticated user
@@ -429,8 +435,8 @@ More field gotchas (each cost a failed exec): `ApplicationEntitlement.applicatio
can be null (guard with `getattr`); `LDAPProvider` has
`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
`organization` — the backing table
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`
`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
(scopes are linked through `ProviderPropertyMapping`, which is why
@@ -484,5 +490,5 @@ podman run --rm --network container:<container> --entrypoint /bin/busybox \
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison
*(the file is gone from the repo and from the host checkout as of
- (the file is gone from the repo and from the host checkout as of
2026-10-18)*
@@ -47,7 +47,7 @@ torrent.
WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s
The private key is hashed into the VM env file and never enters the repo.
NAT-PMP inbound forwarding is available on Proton (gluetun supports it via
NAT-PMP inbound forwarding is available on Proton (gluetun supports it
`FIREWALL_INPUT_PORTS` + `NATPMP=true`) — deliberately deferred: the e2e
gate passes outbound-only, and swarm peers needing an inbound port is a
seeding-era concern, not a grab-only one.
@@ -86,7 +86,7 @@ are unaffected.
got `Unauthenticated`); the live conf was patched and
`scripts/deploy-remote.sh` fixed to write the `_PBKDF2` key. Gotcha
(live, 2026-09-01): after qB 5.x ran, its conf rewrite left a hash line
4.6.7 rejects (`invalid credentials` for every login, including the arrs);
4.6.7 rejects (`invalid credentials` for every login, including the arrs)
rewriting the same `@ByteArray` line fresh (same password, 100k
iterations) restored login. Any future version flip: regenerate the line.
- `HostHeaderValidation=false` + `ServerDomains=*`: the socat forwarder
@@ -98,7 +98,7 @@ are unaffected.
path (no auto-TMM), so the default itself points at `Finished Torrents`:
every grab lands there regardless of client behavior; human review moves
files onward; nothing writes to the entropy volume.
- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`,
- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`
`tv-sonarr` (all → `Finished Torrents`).
- **Download clients** (gateway `10.88.0.1:8080`, user `admin`): all five test
green against live qB — sonarr (`tv-sonarr`), radarr, lidarr, readarr
@@ -109,12 +109,12 @@ are unaffected.
as "Authentication failure" — that kills runtime auth too, not just API
creates; the fix was pinning qB itself (below), after which readarr tests
200 like the rest.
- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`,
- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`
not `latest`): `latest` resolved to 5.x, whose `204` login breaks
Readarr's frozen client; 4.6.7 answers the legacy `200` + `"Ok."` that
every client accepts. Pinned in
`frankbook-pro/qbittorrent/quadlet/qbittorrent.container`.
- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`,
- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`
gateway ports). Readarr deliberately omitted: prowlarr's indexer-push
test to readarr's `/api/v1/indexer/test` 500s (EOL schema drift).
Legacy `Prowlarr-*` indexer rows carried in the migrated dankserver DBs
+25 -16
View File
@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
tags:
- infrastructure
- grafana
timestamp: 2026-09-11
timestamp: 2026-10-20
---
# Grafana
@@ -46,7 +46,7 @@ The Prometheus datasource is declared in git, not clicked into `grafana.db`:
address Caddy proxies to, never the loopback-published port — and carries
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
alongside it by `dashboards/provider.yml` (provider `fleet`,
alongside it by `dashboards/provider.yml` (provider `fleet`
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
A Loki datasource is still **not** provisioned — the tree contains exactly one
@@ -64,22 +64,31 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
- Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
no provider token (`token=None`), so no identity header ever reaches Grafana;
RP mode is topology-independent. The full verified contract (provider pk 9,
client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
Basic client auth, form-encoded token body, and the fork's userinfo-claims
bug that leaves identity at the `sub` hash only) is in
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
users land as Viewer with `login = <sub hex>`; no email/groups/role claims
reach Grafana until the evaluator bug is fixed — do not write role mappings
that depend on those claims. The local admin account stays the only Admin.
(The Viewer-default outcome and the `<sub hex>` login are not readable from
the host without the database or an admin password. *(unverified 2026-10-18)*)
no provider token (`token=None`), so no identity header reaches Grafana; the
login is Grafana's own "Sign in with Authentik" button against the `grafana`
confidential-client provider. **Verified end-to-end 2026-10-20** after four
independent breakages had made it fail. The two Authentik-side causes (a NULL
`authorization_flow` → authorize 500; an email scope mapping that returned a
bare string so no `email` claim was emitted) and the full wire contract live in
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
an ak-shell re-creation of the provider or a blueprint import, both of which
revert it silently). The Grafana half is the unit's own env (`grafana.container`):
`SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
(safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
when the token carries no `email` claim — so it is the email claim itself (an
IdP fix, above) that makes the login land, not any Grafana toggle.
- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
`ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
`email`. The local admin account stays the only Admin. The landing that the
edge-only 13-sso spec never checked is now asserted by
`tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
an email).
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
for the service, podman then looks for its configuration under `/root`,
for the service, podman then looks for its configuration under `/root`
cannot read it, and the unit dies at start with
`stat /root/.config: permission denied`. The unit reads
`/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
@@ -51,7 +51,7 @@ See [[Container Host Reachability]] for the netns rules.
published nothing → structurally unprobeable from the host).
3. Probe from the **correct vantage**: from a bridge-net container the host
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`)
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
they expand **inside** `podman exec bash -c`.
@@ -83,7 +83,7 @@ python3 -m http.server 8899 --bind 127.0.0.1
```
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
(panels render after the async fetch; checking immediately returns 0 panels),
(panels render after the async fetch; checking immediately returns 0 panels)
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
@@ -22,7 +22,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
**Docs page shipped (2026-09-17):** `/docs` serves a static user documentation
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`,
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`
`frank-zealot`) moved from admin-only to the **user** page with live model
descriptions and a `docs` link to `/docs#models`.
@@ -59,7 +59,7 @@ without which the picker showed "No results found" for every family account
"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra
(same era)", frank-zealot "equivalent to last year's frontier level
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic)
pinned in `homepage_config.bats`.
**Protoss rename + Model Information component (2026-10-08):** the model
@@ -91,7 +91,7 @@ same renames, api-id mapping, supervision and compaction notes.
app card now carries hex-clipped art beside its name (`services.json` `art`
field, files under `img/apps/`): 19 brand marks tinted amber from the
Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard)
node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
cards render in the order zealot → dragoon → archon (registry order is the
@@ -143,8 +143,8 @@ per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`)
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path)
`model.compaction` (the amber "set this as compaction model" callout
frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
@@ -175,7 +175,7 @@ titles are chamfered plates with the diamond glyph inside the plate.
Fonts: body is **Share Tech Mono** (legible retro terminal), headings
are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
three pages load the pair from Google Fonts. Four tiles that had
reverted to initials got pictograms: Calibre-Web an open book (orange),
reverted to initials got pictograms: Calibre-Web an open book (orange)
RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
@@ -197,7 +197,7 @@ state — the only animation left is the speaking-voice pulse. The result
is flat navy panels on crisp 1px borders: same Protoss direction, half
the pixels.
**Model ids official (2026-10-08):** the card ids `frank-zealot`,
**Model ids official (2026-10-08):** the card ids `frank-zealot`
`frank-dragoon`, `frank-archon` are now the harness's official model ids
in `models.yml`/`config.yml` (see [[OMP Harness]]); the
systemd services keep the `frank-archon`/`frank-dragoon`/
@@ -205,7 +205,7 @@ systemd services keep the `frank-archon`/`frank-dragoon`/
one name per model.
**Api ids renamed on the wire (2026-10-09):** the promise in
`docs.html` is kept — the served model ids are now `frank-zealot`,
`docs.html` is kept — the served model ids are now `frank-zealot`
`frank-dragoon`, `frank-archon`, so the docs page no longer carries
`api id:` annotations, and the curl/opencode/Pi examples send
`frank-zealot`. The systemd services keep the old names. `docs.html`
@@ -216,7 +216,7 @@ ids, which did not change.
clip-paths replaced with `border-radius` (the clipped corners read as
gaps in the 1px border); section-title diamonds moved from absolute
positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
WIP items grey out (dim type, desaturated art, muted border via
WIP items grey out (dim type, desaturated art, muted border
`:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
(`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
beat; the two SC2 source oggs are trashed); privacy notes became
@@ -188,7 +188,7 @@ sudo systemctl restart systemd-resolved && resolvectl flush-caches
```
The `~` (route-only) prefix is load-bearing: it routes only those domains to
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`
`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
@@ -236,7 +236,7 @@ Endpoints (all POST, JSON body):
`.content.records[]` as some docs claim — parse with
`jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`
`type`, `content`, `ttl`, `prio`).
Quirks that bite (verified live 2026-08-31):
+1 -1
View File
@@ -55,7 +55,7 @@ The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `
The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`
`chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
`find /opt/loki/data -type f` counted **8 files**, all WAL segments,
cluster-seed and delete-request bookkeeping, **no chunk objects**
@@ -21,7 +21,7 @@ committed.
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
Targets and datasources reference **container names** (`grafana:3000`,
Targets and datasources reference **container names** (`grafana:3000`
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
loopback-published ports; `alloy` is **not scraped at all** (no job in the
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
@@ -47,7 +47,7 @@ from `deploy_observability`:
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
`shared/grafana/**` to `grafana` (`:261-264`); changes under
`dankclaw/observability/` select the touched unit and its dependency consumer
@@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint.
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
Current delivered set: **4 alert rules + 2 recording rules** —
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`)
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
@@ -101,7 +101,7 @@ curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.to
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and `up` for
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
listed in [[Prometheus]]. The Grafana query needs the local admin password and
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
@@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect).
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
via `host.containers.internal`) → signal-cli, addressed by number
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
`curl -s http://192.168.0.143:18093/health` → `ok`, and
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
@@ -148,7 +148,7 @@ state lives in Prometheus' process, not on disk — a restart re-stamps
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
`--storage.path=/alertmanager` on a host volume keeps silences and the
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
`/opt/alertmanager/data`.)
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
@@ -2,18 +2,18 @@
type: reference
title: "Observability on Dankclaw"
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-09-11
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
timestamp: 2026-10-20
---
# Observability on Dankclaw
## Summary
Six containers and one host-side process: Prometheus (metrics), Grafana
Seven containers and two host-side processes: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
carrying every firing group to Signal. All of it is Quadlet-managed,
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
boot-persistent and loopback-only, deployed by
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
end to end; the log half is configured, healthy, and carries no data.
@@ -34,6 +34,8 @@ that proves it today.
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,20 +46,22 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
the same day, and `homepage-status.timer` — the root unit that probes all of
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
Dankserver runs one member of this family, node-exporter, as its own unit
(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
`gpu-exporter` (a user unit installed the same way as `signal-relay`).
## Exposure
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
inside, and several carry lifecycle or query APIs. Browser access goes through
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
`anubis:8923`, and the post-auth handlers route by container name on
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
@@ -70,46 +74,82 @@ and the podman bridge subnet `10.89.0.0/16` before the DROP
## Deployment
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
which installs the unit, copies that service's config
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
`copy_alloy_config`, `copy_grafana_provisioning`, plus
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
`grafana` after `prometheus`, `alloy` after `loki`
(`grafana.container:32-33`, `alloy.container:38-39`). See
[[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs.
## Current state (2026-10-18)
## Current state (2026-10-20)
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
eight up targets; the four downs are structural (the `comfyui` and `services`
HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
4 alert rules + 2 recording rules (`/api/v1/rules`, matching
`shared/prometheus/rules/llm-fleet.yml`).
eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
and both node exporters. The four downs are structural (the `comfyui` and
`services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
time-to-first-token), each keyed only to the namespace its runtime exports.
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
relay, and `journalctl --user -u signal-relay` records successful
`sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** One provisioned Prometheus datasource (uid
`prometheus`) and the `llm-fleet` dashboard, both installed from
`shared/grafana/provisioning/`.
* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
(8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
no label names.
See [[Alloy]] → Delivery state and [[Loki]] → Contents.
* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
structural downs have no real uptime signal), no Alertmanager vhost.
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
`prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
`containers`, `observability` and `logs-overview` dashboards, all
installed from `shared/grafana/provisioning/`. The Loki datasource is how
you read logs: Loki has no web UI of its own, so Grafana Explore is the
human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
per-container CPU/memory here, and on this rootless-Podman host it works on one
condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
namespace, so without that flag cadvisor sees only its own cgroup root and
exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
names nothing itself — the libpod socket resolved only 2 of 34 rootless
containers — but the quadlet unit name is carried in the cgroup path
(`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
`container` label. Network and filesystem are NOT exported for these containers
(no netns/classid attribution under the host-PID layout), so the dashboard
graphs only CPU, memory and OOM and does not fake the rest.
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
`deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
`mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
°C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
`power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
a rootless Alloy cannot read the `0640` journal files, and the container-log
globs point at a root-storage layout that does not exist here. The fix is built
and tested in-repo (2026-10-20): Alloy is now a **root system service on
`Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
lines and pushed 26,966 with zero drops. The **live** instance is still the old
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
not yet deployed (see *Logs* above).
## Citations
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
@@ -4,7 +4,7 @@ description: "Pi-hole is the LAN DNS server for the homelab."
tags:
- infrastructure
- pi-hole
timestamp: 2026-08-28
timestamp: 2026-10-19
---
# Pi-hole
@@ -46,6 +46,8 @@ The file format is hosts-style (`<IP> <name> [<name> ...]`), one A record per Ca
**Pitfall (2026-08-24):** the first quadlet mounted the file into `/etc/dnsmasq.d/`. FTL never read it; names only resolved from the query cache. A fresh name returned NXDOMAIN. Mount into `/etc/pihole/hosts/`, not `/etc/dnsmasq.d/`.
**Pitfall (2026-10-19): FTL can come up with the record file mounted and still serve none of it.** A plain `systemctl --user restart pihole` (16:25, inside the open-notebook deploy window) left every `franksdank.cloud` name NXDOMAIN for LAN clients while **external names kept answering** — from FTL's cache, which is exactly what makes the failure read like a client or an upstream problem. **The trigger is a container restart, not a boot**, and that matters for where the check belongs: `uptime -s` reads 2026-10-18 01:24 (two days of uptime) while `journalctl --user -u pihole.service` shows `Stopping pihole.service` at 16:25:23 and the container's `StartedAt` at 16:25:27 — so verify after **any** pihole restart, deploy-triggered or manual, not only after a reboot. The instrument is on the box: `getent hosts grafana.franksdank.cloud` (or `dig +short @192.168.0.143 <name>`) answering empty means FTL is holding no local records at all; here the file was present and readable inside the container (`-rw-rw-r-- … /etc/pihole/hosts/addn-hosts`, 44 records) and `dns.hosts` in `pihole.toml` was intact. `podman exec pihole pihole restartdns reload` had every name answering within seconds. The mechanism is **not established** — the same mounts and the same config read fine before and after the reload — so treat it as a load failure of the record sources at FTL start, not as a lost or unmounted file. What it cost: the whole Playwright gate, 18 specs across every vhost failing `ERR_NAME_NOT_RESOLVED` while the fleet itself was healthy ([[E2E Suite]]).
## Upstream
The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `DNS1=1.1.1.1`, `DNS2=1.0.0.1` (mirrors the retired `dnsmasq.conf` `server=` lines). Every query that is not a local record goes to those resolvers. The zone apex and unknown subdomains fall through to Porkbun.
@@ -74,6 +76,8 @@ Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
| Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong |
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line |
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
| Every `franksdank.cloud` name is NXDOMAIN for LAN clients while external names still answer | FTL holds no local records: `getent hosts <name>` on Dankclaw comes back empty → `podman exec pihole pihole restartdns reload` (the 2026-10-19 pitfall under **Records**) |
| One client gets nothing while every other client is fine | FTL rate-limiting that client: `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`. A `10.0.2.x` client address is a pasta container's DNS forwarder, not a machine |
## See Also
@@ -100,7 +100,7 @@ The DFlash drafter proposes, then ngram-mod runs as an additional implementation
`--spec-type` takes a comma list, and the gate is `ctx_dft` in
`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`
`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
- A chain may hold ONE context-backed drafter plus any number of ngram types.
@@ -201,7 +201,7 @@ Speculative decoding is lossless under rejection sampling: the target verifies e
On the mid-August 2026 Dankclaw shape `draft-dflash` was the benchmarked winner
at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those
three figures are that retired llama.cpp era's property ([[frank-dragoon]] owns
the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see ).
## Draft Models
@@ -257,7 +257,7 @@ DSpark took `ROCm error: an illegal memory access` inside the shared draft
context with two concurrent 16k prompts; DFlash corrupted multi-slot into
premature EOS and was retired first. The timeline, the crash class, the
"concurrency is the trigger, not size" finding and the `PARALLEL=1` workaround
are on [[qwen3.8-spec-matrix]]; the shape they were measured on is
are on ; the shape they were measured on is
[[frank-dragoon]]'s mid-August 2026 era.
## A/B protocol on a live llama.cpp endpoint
@@ -303,7 +303,7 @@ Protocol:
Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
`predicted_per_second`; cumulative `/metrics`
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`
per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
while tokens flow = the proposer is NOT engaged, whatever the flags say. A
zero-draft config is a useful negative control: it proves the other config's
@@ -367,7 +367,7 @@ states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
**Step 4 — acceptance reality checks.** Treat 40–60 % as the honest band for a
matched pair; the arm that anchors the bottom of it on this fleet is the
depth-7 `draft-dflash` row of [[frank-medium-llamacpp-dflash2]]'s spec table
depth-7 `draft-dflash` row of 's spec table
(accepted share, rate and VRAM for every depth are in that table). Cheapest
proxy for `p` at temp 0 when both models are greedy and share a tokenizer:
greedy continuation agreement on identical prompts, two API calls, no weights
@@ -381,10 +381,10 @@ drafting.
**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
engines refuse external drafters by design: halogen-flash-server's
`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
name is a deliberate 400 (a silent downgrade would make comparisons lie);
name is a deliberate 400 (a silent downgrade would make comparisons lie)
there is no device routing in its loopback token protocol, so a drafter can
never sit on another card. Pure llama.cpp DOES route the draft:
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`
`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
One process sees only the devices its backends were built with — CUDA+Vulkan
is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
@@ -411,20 +411,16 @@ measurement that would settle it.
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
- The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by [[RAM CPU Offloading in VLLM]].
-The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by .
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
- Use `rejection_sample_method` default `strict` for lossless output.
## See Also
- [[Quantization by Hardware Architecture]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[deployment]]
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- where the drafter's KV actually lands
- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle

Some files were not shown because too many files have changed in this diff Show More