Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.
cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.
cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.
Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:
- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
upstream would stay invisible - the exact 17-hermes class. Added to the 14-
guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
new 21- that proves authenticated reachability to open-notebook's own login
through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
answers {status:success}.
Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
llm-fleet.json already owns LLM capacity+spec-decode. These three cover the
domains that had NO dashboard:
- host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both
Dankclaw and Dankserver (templated by job node-dankclaw|node-dankserver)
- logs-overview: Loki volume + per-container error rate + a sample log panel;
container is parsed from the JSON line at query time (journal streams have
no container_name stream label)
- observability: the stack's own health - Prometheus target up/down + scrape
interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem.
Panel queries verified live against Prometheus/Loki; provisioned dashboards
confirmed via /api/search. Alertmanager is running but NOT scraped by
Prometheus - noted in the dashboard, no empty panels built on it.
The login lands once authentik emits the email claim (the IdP fix); Grafana needs
no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in
the literal-secret seam. Correct the two SSO notes accordingly.
The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
The vault claimed the fork property-mapping evaluator was broken and userinfo
returned only sub, so email could never reach Grafana (outcome unverified). That
was a misdiagnosis: the real causes were the NULL authorization_flow (the
NoneType.slug crash) and the email scope mapping returning a bare string rather
than a dict. Record the four verified causes and fixes, point to the repair
script and the new landing spec, and drop the unverified caveat.
13-sso proves an authenticated session reaches Grafana login page but never
follows the OAuth button, so the relying-party login could be dead (500, or an
email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives
/login/generic_oauth through the real Anubis+authentik round-trip and asserts a
live session (/api/user 200 with a non-empty email). Wire 20- into the standard
project (qa-e2e session).
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.
scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
The seams encoded alloy-as-rootless (default.target, dankclaw_default,
loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update
them to the root/system/host shape and add a test pinning the
load-bearing properties: Network=host, the /etc/machine-id mount, the
journal mounts, loopback UI, and the system-install + root-mirror
deploy path.
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.
As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.
deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a
Loki datasource is provisioned beside Prometheus.
Observability.md: current-state census refreshed (three LLM targets up,
two rule groups, Loki datasource); logs still carry no data.
Alloy.md: replace the 'wrong glob' theory with the proven cause - a
rootless container cannot read the 0640 systemd-journal files (a rootless
busybox read returns DENIED), and rootless quadlet logs land in the user
journal the system-only source never reads. Closing it is a
permission/architecture decision, not a config glob.
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
radiance exports vllm:* metrics (verified on the live endpoint), not
llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a
vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache
collapse, preemption, p95 time-to-first-token) keyed to vllm:*.
Split the observability seam so llm-fleet rules bind llamacpp:* and
vllm-fleet rules bind vllm:*, and neither may evaluate the other's
namespace. Drops the retired no-metrics-for-archon doctrine the seam
still asserted and the vacuous namespace grep.
The Text-to-speech section recorded audio.tts base_url host.containers.internal
and voice narrator1. Re-keyed the live config DB 2026-10-20: voice wilwheaton
(narrator1 retired), api_base_url the canonical host LAN IP 192.168.0.143:7772.
7772 is published so host.containers.internal also reaches it, but the LAN IP
matches this page's own on-host law. Browser-kokoro example voice updated to the
current pack; the dated 2026-10-18 verification is left as history.
Frank replaced the narrator1/2/3 packs with cloned character voices
(wilwheaton, vaati, dr.kern, garrickhagon, jeffhays). The three narrator
slots are now carried by real packs: narrator1=wilwheaton, narrator2=vaati,
narrator3=dr.kern.
- frank-tts.env DEFAULT_VOICE narrator1 -> wilwheaton (a voice-less request
400'd on the dead name; wilwheaton is narrator1's successor).
- Docs: correct the /health census and the podcast-speaker references in
Frank TTS.md and Open Notebook.md to the live packs. Dated 2026-10-18/19
measurement logs are left as history.
Open Notebook's live dank-duo speaker profile was repointed over REST
(Frank->wilwheaton, Comrade->vaati) - app-side, not in this commit.
Bats fixtures keep narrator1 as a self-contained sample pack; untouched.
open-notebook has no user accounts upstream (PasswordAuthMiddleware = one
shared password; SurrealDB has no user/account/owner table — verified in the
running image). One shared password = one shared workspace; the guard chain
gates access but cannot partition data, and Authentik identity is never
consumed. Multi-user is approved-but-unbuilt upstream (#712, post-v2); OIDC
declined (#607). So move the homepage card from the user grid to the admin
tier and add the single-tenant boundary to the vault page. Fix the card's
stale 'frank-zealot' -> frank-archon. Update the seam's incidental category
pin to the new intentional value. Password rotated on the host (not repo).
The login-session lifetime (raised to days=7 on 2026-10-20) is owned by the
default-authentication-login stage's session_duration, not a global setting and
not SESSION_COOKIE_AGE. Authentik exposes no SESSION_SAVE_EVERY_REQUEST, so the
window is absolute. session_duration is a STRING idiom; a timedelta stores a form
timedelta_from_string rejects, crashing login. Add scripts/authentik-set-login-session-duration.sh
that validates the form before writing (DB state, not repo-managed, re-run after
a blueprint re-import). Trim the page back under the OKF 500-line cap. tests/run.sh green.
Point the mattpocock-lineage skills (triage, to-tickets, to-spec,
wayfinder) at repo facts instead of a parallel docs/ tree: Gitea is the
issue tracker (credential-safe curl shape, token-only writes), the five
canonical triage roles keep their default label strings (created on the
tracker today), and GLOSSARY.md/docs/adr/ stay lazy with this vault
owning operational facts. AGENTS.md gains the ## Agent skills pointer
block; indices regenerated.
The 2026-10-19 write-up inferred a host restart from FTL's 16:25 start line. It was wrong: uptime -s reads 2026-10-18 01:24 (two days up) and the user journal shows 'Stopping pihole.service' at 16:25:23 with StartedAt 16:25:27 — a plain container restart inside the open-notebook deploy window. That moves where the verification belongs: after every pihole restart, not just after a reboot.
open-notebook resolves frank-dragoon-halogen to 10.89.1.39, GET :8731/v1/models answers, and the engine logs the app's POST /v1/embeddings 200. That makes the 2026-10-19 dial census a pre-move record: pasta's forwarding rule is unchanged, the endpoint was moved out of pasta, so the 'menu is frank-tts + frank-archon' line is now past tense. Frontmatter date was still 2026-09-02 against October content; bumped.
The 2026-10-19 reboot left every franksdank.cloud name NXDOMAIN for LAN clients while external names kept answering from cache; the file was mounted and readable and dns.hosts was intact, and 'pihole restartdns reload' restored every name in seconds. Mechanism not established. Instrument and remedy now on the owner page, plus a failure-mode row for the FTL rate-limit case seen beside it (10.0.2.x = a pasta container's forwarder).
open-notebook's embedding default now points at qwen3-embedding-0.6b on frank-dragoon's XDNA2 lane, proven through the app's own paths (model test 1024 dims, worker 'Successfully embedded', vector search ranking two unrelated notes 0.659/0.292 and 0.906/0.318). Records the two app-side traps: defaults store the model ID and a name is accepted then fails in the worker, and POST /api/embed 500s on an upstream RecordID typing bug after the vector is already written. frank-dragoon's lane section moves from 'off until the mount fix' to armed-and-serving with the boot-ledger contract and the measured idle RSS; the page's stale 'not on dankclaw_default' reachability claims are retired.
The armed boot reached the engine and died in its own preflight:
halogen npu: /dev/accel/accel0 is here, but this process cannot open it
(crw-rw---- nobody:nogroup).
The node is root:render 0660 and a rootless container maps exactly one uid/gid,
so `render` is unmapped inside it and `--group-add keep-groups` — the vendor's
advice, and the ComfyUI pattern for /dev/kfd — cannot grant a group the
namespace does not map. Measured 2026-10-19 from a `dank` shell already in
`render`: without the flag denied, with the flag denied, with the udev mode
0666 the open succeeds (OPEN_OK). Same wall 99-comfyui-gpu-access.rules already
solved for the GPU compute nodes, so the accel node gets its counterpart:
dankclaw/udev/98-halogen-npu-access.rules — KERNEL=="accel[0-9]*",
SUBSYSTEM=="accel", MODE="0666"
The deployer installs the rule, reloads rules, re-triggers the node and only
then checks the mode; the serve script refuses any node that is not
other-writable, naming the rule, so this cannot return as a crash loop (189
restarts while chat was down is the failure mode being closed).
Both checks read the permission bit directly: `[ -o FILE ]` is not
"other-writable" — it means "exists" and `-o` doubles as the OR operator inside
test, which is why the first cut of the gate refused everything.
Dragoon seam: 33 -> 35 tests. The fixture node is 0666 like the installed one,
a 0660 node is refused with the rule named and nothing rendered, and the rule
file's shape plus the deployer's install/trigger are asserted.
The containment gate added in #44 parsed scripts/frank-dragoon.env through a
$CHECKOUT variable that does not exist in deploy-remote.sh, so the very next
deploy died at `set -u` before it touched the unit:
./scripts/deploy-remote.sh: line 1445: CHECKOUT: unbound variable
deploy-remote.sh runs from the checkout root and every other path in that block
is a parity-twin default (LLM_DRAGOON_CKPT/_VISION/_TOK_DIR), so the payload dir
gets the same treatment: LLM_DRAGOON_MODEL_DIR, defaulted to the deployed path,
compared directly. The seam now asserts that twin against the env file next to
the NPU twins it already checks, and the containment comparison itself is
verified both ways (inside accepted; the old outside path and a shared-prefix
sibling both refused).
frank-dragoon's lane section: the staging dir must live inside the payload dir
(the image ships no /models, so a nested :ro bind has no mountpoint to land on),
the entrypoint's rejection of HALOGEN_NPU_DIR, and the fabric-clock paragraph
corrected — the vendor's root host unit is installed and reports `held (manual)`
as of today, so "nothing is held" is retired along with the hand-write it
replaced.
dankclaw-iommu-translation-off: the deferred `pp_dpm_fclk` pin is no longer
deferred; the fact is owned by [[frank-dragoon]] and this page links it.
The lane's first boot died at container init, not at the engine:
runc create failed: ... create mountpoint for /models/npu mount:
mkdirat .../merged/models/npu: read-only file system
The image ships no /models, and the serve script binds the payload dir
there read-only, so a nested `-v $NPU_DIR:/models/npu:ro` had to create
its own mountpoint underneath a read-only parent — impossible. The
engine's own entrypoint confirms the container path is fixed: it rejects
HALOGEN_NPU_DIR outright ("our NPU models live under /models/npu/<id>/").
Fix is the mount shape, not the engine: FRANK_DRAGOON_NPU_DIR now lives
inside FRANK_DRAGOON_MODEL_DIR, so the payload's single read-only mount
puts the files at /models/npu/<id>/ by itself and the nested bind is
gone. Verified on the running container: after the move, `podman exec
frank-dragoon-halogen ls /models/npu/qwen3-embedding-0.6b` answers with
the staged tree while the lane is still armed off.
Both gates refuse the old shape by name (serve + deployer parity) so a
future payload swap that moves the model dir cannot reintroduce a crash
that lands after the deployer has already stopped the running unit.
The 2026-10-19 legs table blamed the frank-dragoon refusal on a root/rootless
netavark rule-set split. It is not that: the container is rootless (the root
unit's User=dank runs podman run), it is on pasta, and pasta installs no DNAT
for rootless-bridge traffic while refusing a second network outright. The
gateway alias does reach it (200) and is link-local, which is the same dead end
the Open Notebook URL guard hit. Measured with the move onto the bridge now in
the deployer, including the /proc/net/tcp line that shows 8730 is loopback-bound
and so stays private after the move.
Co-Authored-By: omp <omp@dankclaw>
An endpoint on podman's rootless default (pasta) has no address another
container can use, measured on 2026-10-19 from inside open-notebook:
- 192.168.0.143:7770 connection refused (pasta forwards the
published port in the host netns; no netavark
DNAT exists for traffic arriving from a
rootless bridge)
- host.containers.internal:7770 200, but the alias resolves to 169.254.1.2 —
link-local, which open-notebook's URL guard
refuses to store
- frank-dragoon-halogen:8731 name unknown: different netns
- --network pasta --network dankclaw_default
"cannot set multiple networks without bridge
network mode" — pasta cannot be added to
So reaching the NPU endpoints from an app container means moving the container
onto dankclaw_default (FRANK_DRAGOON_NETWORK; empty keeps today's shape). This
is not an exposure change: the engine's unauthenticated token port binds the
container's own loopback (/proc/net/tcp: 127.0.0.1:8730 against 0.0.0.0:8731),
so only the API port becomes name-resolvable — the same trust the published
7770 already carries on the LAN. The deployer refuses a network that does not
exist rather than starting a unit that restart-loops on it.
Co-Authored-By: omp <omp@dankclaw>
halogen-flash-server already ships an NPU path on the pin we run: for each
model named in HALOGEN_NPU_MODELS it loads the vendor's precompiled AIE
programs (XDNA2 executes only AIE-compiled graphs) and answers them on
/v1/embeddings inside the SAME container behind port 7770. That gives the
fleet an in-house embedding endpoint without touching the iGPU trunk.
The lane is opt-in and one line wide: HALOGEN_NPU_MODELS blank is the
pre-lane endpoint byte for byte (no device, no mounts, no engine env).
The three host preconditions the vendor names are gated in the serve script
and mirrored in the deployer (the 2026-10-01 twin law), each refused by name:
- the staged model files. /models is read-only and HALOGEN_DOWNLOAD stays
unset so the container opens no outbound connection at boot, so
scripts/download-frank-dragoon-npu.sh stages them from the pinned
revision with the catalog's own byte counts and sha256s (the engine
re-checks both at boot);
- the distro XRT's libxrt_coreutil/libxrt_core/libxrt_driver_xdna, each
mounted twice — where the image looks for XRT and at its own path —
because those libraries find each other where they were installed and
an /opt/xilinx/xrt of symlinks dangles inside the container;
- the GPU fabric clock HELD. On Strix Halo the iGPU and the NPU co-running
hang the machine (and corrupt NPU results) while FCLK changes speed, and
halogen refuses to start in that state; it cannot read host sysfs from
inside, so the hold is proven host-side. dankclaw/host/ carries the
vendor's halogen-fabric-clock{,.service} verbatim at tag v0.16.2 (sha256
a3a4f7f4e0142e71… / 5f9f1cc4c85a1c02…); the deployer installs, enables
and gates it only while the lane is named, and releases the clock plus
drops the boot hook when it is not.
After health, the deployer's lane gate is a real embedding through
/v1/embeddings — a 200 on /health proves nothing about the NPU programs.
Planner: dankclaw/host/halogen-fabric-clock* re-dispatches frank-dragoon
(the deployer installs it); dankclaw/host/grub and the fetch script plan
nothing.
Seam: 11 new tests — the off arm really renders nothing lane-specific, the
on arm renders the device/mounts/env, every missing precondition refuses by
name, the deployer twins and the host unit shape are pinned.
Co-Authored-By: omp <omp@dankclaw>
- frank-dragoon: measured the response_format contract. json_schema and json_object answer
schema-valid JSON at temperature 0 and draw a 400 at any sampled temperature, and because
the sampler default is server-side an absent temperature is sampled. The page claimed
"response_format is unsupported", which sends the next agent past a working feature.
- frank-dragoon: new "The NPU lane" section. The pinned 0.16.2 image already ships the
five-program AIE catalog including qwen3-embedding-0.6b, so /v1/embeddings is one env var
plus the accel device, the distro-XRT mount and the fabric-clock hold; embeddinggemma-2
needs 0.18.3+, the stock-kernel-driver memory-budget fix 0.18.8+. Host census recorded:
no /opt/xilinx/xrt, distro XRT 1:2.21.75 with libxrt_driver_xdna, clocks unheld.
- Open Notebook: scriptwriter probe verdict — greedy-only, and open-notebook's model record
is {credential, name, provider, type} with no per-row config, so it cannot ask for greedy;
frank-archon stays the writer. The embedding gap now has a named door instead of "build a
new llama.cpp slot".
- Frank TTS: "the NPU is not a door at all" was too broad. It is not a door for the TTS
family; halogen's precompiled catalog is a door on the same device.
- System Specs: NPU runtime row corrected to the distro XRT actually installed, and the
stale "frank-zealot serves 27B" line retired in favour of [[frank-zealot]] as sole owner.
Tests: ./tests/run.sh, 455 passed.
The page described an undeployed service whose scriptwriter was going to be
frank-zealot. Both halves are now false, and the reason is worth more than the
correction.
- Status, ports, coordinates: deployed 2026-10-19, guarded vhost, API published
on loopback only. The first episode is the proof: 747 s of speech in 301 s of
wall clock (2.5x real time, knobs at 1), download route 200 with the bearer
and 401 without.
- New section for the guard chain: provider-before-route, one upstream because
the image rewrites /api/* itself, and the API_URL law — autodetect appends
:5055, which is seerr's port on this host, so an unset API_URL yields a UI
that loads and an app that cannot call itself.
- Provisioned records table (credentials, model rows, dank-duo, dank-daily) and
the two contract traps measured on the live API: generate takes profile
NAMES, and the fleet has no /v1/embeddings, so every note save fails its
embed_note command while renders from content work fine.
- Why frank-archon writes the scripts: frank-zealot is reachable from this
netns only at 169.254.1.2, and open-notebook's URL guard refuses to store a
link-local base URL with no override. Census lands on [[Container Network
Namespaces]], which also gains the root-vs-rootless publish rule — a
root-published port is invisible to a rootless container, so "publish the
port" was the wrong remedy as written there.
- Frank TTS: the request's model field is ignored; only voice selects a pack.
Tests: ./tests/run.sh, 455 passed.
The service came up internal-only: UI on the LAN at 8502, API at 5056, both
published on every interface, with only its own shared password in front. That
put the episode library outside the fleet's auth chain and one port scan away
from anything on the network.
- Caddy: guarded vhost notebook.franksdank.cloud (Anubis -> Authentik; provider
notebook-proxy, internal_host caddy:8017) plus the post-auth route to the
published UI port. One upstream is enough, because the image's own Next.js
layer rewrites /api/* to the FastAPI beside it — so the API needs no outward
face at all, and 5056 moves to loopback, where only the deployer's health gate
dials it.
- scripts/open-notebook.env: API_URL and CORS_ORIGINS become the single guarded
origin. Left to itself the app auto-detects <host>:5055, the port seerr holds
on this host; a LAN-origin API_URL would also take the browser around the
guard chain, where its authenticated session does not ride.
- addn-hosts: the LAN A record. The homepage card goes live and links the vhost.
- Seam: the API origin is derived from whatever the Caddyfile routes to 8502, so
renaming the vhost moves the test instead of leaving it blessing a stale name,
and a new test proves the vhost hands the WHOLE request to Anubis.
Frank
dank
