observability.json: stale 'structural downs' panel description fixed,
target census matches the post-blackbox job set. logs-overview.json:
panels grouped by the promoted container label instead of the absent
container_name, so the board stops reading empty beside live traffic.
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.
copy_grafana_provisioning named llm-fleet.json alone, so a new board in
shared/grafana/provisioning/dashboards/ silently never reached
/opt/grafana/provisioning. Measured 2026-10-21: the containers board was
merged, seam-tested, and documented as working since 2026-10-20, yet was
NEVER provisioned — its panel fixes were undeployable; the other four
boards existed on disk only because someone hand-copied them once (and
host-system.json there had gone stale against the repo). The copier now
globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run
the copier for real into a sandbox (red against the old hand-listed
body, green against the glob). Vault: the copier law + the containers
incident on Observability Provisioning, the corrected dashboards claim
on Observability.
Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the
dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).
- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
non-matching pairs and the right side collapses to {job} while the left
carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
so the '> 0' filter can never match. Rewritten to share-of-host-total via
label_replace join of the node exporter's MemTotal by host (the parens
form is a PromQL parse error: nothing may follow group_left); 62 series
live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
(ALERTS carries only firing/pending series); description now says so,
per the Stale Signal rule that 0-series-on-ALERTS is not a defect.
logs-overview and observability: all targets already returning data.
Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):
- Observability Provisioning: destroyed the 63-line 'Current delivered
state' block — chain topology/relay/delivery restates [[Observability]]
and [[Alertmanager]] (those are the owners). Its three sole-custody laws
MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
down-beside-healthy -> read lastError; Prometheus alert state in process
vs Alertmanager's --storage.path persistence; candidate-arm metrics are
never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
never reuse a B/token ratio across geometries; pin pool bytes
floor-sized), era-labelled the rates, destroyed the measured ladder
table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.
check-okf 109/0, vault links 0 unresolved, docs seams green.
The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/
as well; our mounts filled only /opt/xilinx/xrt/lib and the install path.
Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming
the missing prefix ('No such library
/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine
supervisor takes the WHOLE container down on a dead lane — measured
2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the
GPU lines looked healthy.
Each XRT library now mounts THREE times from the same real file (never a
symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename
so the bats fixture proves the derivation. Proven end-to-end before commit:
production-shaped scratch boot + the three extra mounts answers the full
boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling.
A dead lane is not survivable on this engine — arm it only with all three
prefixes (page law updated).
Shape + harness contract on the endpoint page, AGENTS.md, and the OMP
Harness worked example recomputed to the live pair; the 400-zone law now
names the 85 %-comments vs 95 %-config disagreement as the recurring
cause and points at the seam that enforces the arithmetic.
The 200,704 ceiling squeezed the harness contextWindow down to 155,648
(the 400-zone fit law needs ~45k of gap under the ceiling), which is why
long dragoon sessions kept hitting the 400 zone: the harness clamps
max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %,
and config.yml compacts at 95 % while the pair was designed for 85 %.
Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays
401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx
still holds). The harness window becomes 200,000 with a 62,144 gap.
The fit-law seam now reads the declared window from the vault page (the
one owner) and checks the arithmetic at the live 95 % trigger, so a
ceiling/window pair that drifts back together fails tests/run.sh
instead of a live session.
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:
- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
but never accept()ed, loopback deploy gate passed, every scrape that
reached the other bind timed out on the SYN backlog. Now one thread per
extra bind. Regression test spins --serve on two loopback binds and
requires BOTH to answer /metrics.
- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
Prometheus container: a bridge container cannot dial its own host's LAN
address at all (hairpin refuses published ports too); the podman gateway
answers. Target moved to host.containers.internal:8080; the seam that
pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
gateway plus a negative pin against the own-host LAN dial. Observability
page records the measured reachability law replacing the inferred one.
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.
Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029
is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex
releases behind. Digest-verified at pin time: the tag's manifest digest
equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the
same law the page records. Quadlet restart runs PMS's usual one-minor
schema migration on first boot; the deployer's /identity HTTPS gate
tolerates the first-boot migration up to 600 s.
Rollback: revert this commit (one line).
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.
scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.
It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the
repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate
400 from the engine's own window check — which reads as a capacity-contract
failure on a healthy endpoint. One ~4k-token request now prices the ratio, and
the leg sizes to 97 % of the deployed window.
The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB
(78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions,
`--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host`
8.99 GiB a rank.
The KV Capacity law is rewritten from the pinned source instead of inference:
`core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`,
`core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never
funding), and `core/place/planner.cpp:100-104` makes the expert floor fall
one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool
bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %,
never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling
case, not a mystery. Slot-cost law carries the four-slot price (state +
checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot
era is one row in Eras with its rollback commit. Date stamps in the trio
comments corrected to 2026-10-20.
Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it
exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is
800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of
evidence on the endpoint page), which clears 700k, and the VRAM has to come
from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with
it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly
plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60
GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to
19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the
leftover returned to the experts.
Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per
rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when
the router picks a host-resident expert. The boot ledger's vram_kv line is the
referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and
this redeploys.
TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 +
prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB =
161,848 MiB), and the bats ledger guard now counts the checkpoint store because
it grows with the slot count. The two env-value assertions in that seam are
bumped with the values they guard.
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.
BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.
Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.
cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.
cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.
Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
The replay law named `--checkpoint-policy floor`; config.cpp admits exactly
geometric-backoff (default), tip-only, keep-all and refuses anything else at
boot, so the recorded arm could never be deployed. Replaced with the real
semantics, priced from core/mem/prefix.{h,cpp}: the default keeps a geometric
ladder (9 snapshots for a 200k session at interval 2048 -- exactly the live
ledger line -- so a turn diverging d behind the tip can resume at tip-3d:
diverge at 120k, replay 21.7k); retention is not the write path, so keep-all
buys a bounded replay for pinned host RAM alone (196 slots x 111.98 MiB = 21.4
GiB against the store's eighth-of-host budget of 31.5 GiB, hard bound = room);
and shortening --checkpoint-interval cannot fix the deep case -- the coarsest
ladder scale is interval x 2^k under the depth, so the deepest gap stays about
half the prompt at every interval. §9 keeps only the three-name locator.
frank-archon Laws: add the measured interference census (decode-only step
15.5 ms vs 166 ms chunk steps; 4.7% of steps hold 34% of step time; median
decode 166 -> 13 t/s above 5k prefill tok/s, per-stream 6.7 t/s = one token
per shared step; TTFT p90 2.4 -> 18.0 s for arriving requests), the lag test
that kills the residency theory, the busy_cap arithmetic, and the correction
that --prefill-interleave is the decoder dial and auto is not.
Facts that change how an operator reads this box: /diagnostics exists and
reports link-below-max at 100% busy; latency metrics live under vllm:, cadence
under radiance:; preemption is vllm:num_preemptions_total (264 this boot, all
bursts) while the radiance gauge is instant depth; both root ports are x8 Gen5
max by board (31.5 GB/s ceiling, not 64); DDR is back at 4800 MT/s while the
4600 corruption fix is recorded -- verify before blaming garbage tokens.
§9 (LLM Performance Diagnosis): method corrected to the phase-ratio counters,
dead 1.0.13 numbers destroyed, four PromQL traps added (metric prefix, 1 s EMA
gauges vs 15 s scrape, counter resets across builds, 2-minute mean diluting the
chunk), instruments that cannot be reached from Prometheus, and the knob list
priced from the pinned 1.8.1 source instead of the stale guide.
The env comment claimed BUSY_PREFILL_CHUNK=auto protects decode rows; the 1.8.1
source (core/sched/scheduler.cpp busy_cap) reserves 1+spec rows per decoding
sequence and 1/8 of the budget per waiting request, and refuses to shrink the
chunk for decoders alone -- auto is a prefill-speed dial. The interleave comment
carried a 512 that is not this build's auto value (0 until the held-step
counter asks; 64 in help mode). The serve header still described three decode
slots; MAX_SEQS has been 2 since the 2026-10-19 reslice. No behavior change.
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:
- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
upstream would stay invisible - the exact 17-hermes class. Added to the 14-
guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
new 21- that proves authenticated reachability to open-notebook's own login
through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
answers {status:success}.
Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
llm-fleet.json already owns LLM capacity+spec-decode. These three cover the
domains that had NO dashboard:
- host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both
Dankclaw and Dankserver (templated by job node-dankclaw|node-dankserver)
- logs-overview: Loki volume + per-container error rate + a sample log panel;
container is parsed from the JSON line at query time (journal streams have
no container_name stream label)
- observability: the stack's own health - Prometheus target up/down + scrape
interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem.
Panel queries verified live against Prometheus/Loki; provisioned dashboards
confirmed via /api/search. Alertmanager is running but NOT scraped by
Prometheus - noted in the dashboard, no empty panels built on it.
The login lands once authentik emits the email claim (the IdP fix); Grafana needs
no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in
the literal-secret seam. Correct the two SSO notes accordingly.
The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
The vault claimed the fork property-mapping evaluator was broken and userinfo
returned only sub, so email could never reach Grafana (outcome unverified). That
was a misdiagnosis: the real causes were the NULL authorization_flow (the
NoneType.slug crash) and the email scope mapping returning a bare string rather
than a dict. Record the four verified causes and fixes, point to the repair
script and the new landing spec, and drop the unverified caveat.
13-sso proves an authenticated session reaches Grafana login page but never
follows the OAuth button, so the relying-party login could be dead (500, or an
email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives
/login/generic_oauth through the real Anubis+authentik round-trip and asserts a
live session (/api/user 200 with a non-empty email). Wire 20- into the standard
project (qa-e2e session).
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.
scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
The seams encoded alloy-as-rootless (default.target, dankclaw_default,
loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update
them to the root/system/host shape and add a test pinning the
load-bearing properties: Network=host, the /etc/machine-id mount, the
journal mounts, loopback UI, and the system-install + root-mirror
deploy path.
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.
As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.
deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a
Loki datasource is provisioned beside Prometheus.
Observability.md: current-state census refreshed (three LLM targets up,
two rule groups, Loki datasource); logs still carry no data.
Alloy.md: replace the 'wrong glob' theory with the proven cause - a
rootless container cannot read the 0640 systemd-journal files (a rootless
busybox read returns DENIED), and rootless quadlet logs land in the user
journal the system-only source never reads. Closing it is a
permission/architecture decision, not a config glob.
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
Frank
dank