iptables -I prepends, so the install block's final -I of the blanket DROP
landed it at position 1, above all three ACCEPTs: every packet to
18093/tcp died there (live 2026-10-21: 159 packets on the DROP, 0 on each
ACCEPT). The host's own health probe timed out beside a LISTENing socket
and the whole dankclaw drain aborted at the alertmanager gate twice.
Install now rebuilds the tail: delete any blanket DROP wherever it sits,
keep the ACCEPTs at the head, append the DROP last - converges on hosts
poisoned by the old order. New bats case runs install_signal_relay for
real against an iptables sandbox seeded with the poisoned order (red on
the old code, green on this one).
Per Frank, frank-archon is public alongside dragoon/zealot: three-leg DB
trio (conn enable, model row ctx 200000, access_grant principal='*'),
verified as role=user. Token-mint traps recorded (cwd, PID-1 secret).
Closing the command-center security queue #70/#71/#72/#73/#74: rolling tags
let a rebuild move under a running service with nobody approving it, and
make a CVE audit ungradeable. 36 quadlet units across dankclaw, dankserver
and frankbook-pro now carry tag@sha256; localhost/* builds stay exempt
(their build is the pin). The pihole and copyparty pins pull the fixed
builds; valkey 9.1.2 (already running) and redis 7.4.11 are pinned in
place, vendor fixes TBD.
New gate tests/quadlet_image_pins.bats: no unpinned registry image, no
generator-rejected key anywhere. Three seams that pinned the old exact-tag
law (frankbook readarr, open_webui, seerr) now require tag AND digest -
strictly stronger. Quadlet Lifecycle docs: batch-abort law, entrypoint
flags-only law, pin law.
Three stacked lessons, each proved live 2026-10-21:
- the image's ENTRYPOINT is /bin/blackbox_exporter, so Exec= repeating the
binary made it a positional arg: 'unexpected /bin/blackbox_exporter'.
- the podman gateway 169.254.1.2 is NOT bindable from the host netns - it
exists only inside each rootless netns: 'bind: cannot assign requested
address'. The scrape path does not need it bound: a bridge container
dialing the gateway lands on the host LOOPBACK (proved 200 from the
prometheus container), so the exporter binds 127.0.0.1 + LAN.
- HealthCmd shell form ('/bin/sh -c "..."') is passed as one quoted token
and quadlet folds the following --health-interval/--health-retries (and
the image) into the command string; the plain exec form the sibling units
use is correct.
Prometheus.md bind law corrected (gateway is a forward-to-loopback, not a
bind address); deploy-remote gate stays loopback, which is now honest.
Third boot failure: the exporter exited 100 ms in with 'unknown long
flag --listen-address' (proved against the pinned image, plus --help).
The bind semantics were right; the flag name was invented. --web.listen-address
is repeatable - one per address - so the dual gateway+LAN bind stands.
Proven again pre-deploy with the real argv: the run starts and dies
only on the deliberate /dev/null placeholder config.
The deploy died at 'Unit blackbox-exporter.service not found': a
.container without Image= is not generated at all, and the header
argued for Network=host it never declared. Completes the unit (digest
pinned prom/blackbox-exporter, Network=host) and fixes the bind
topology the first cut got wrong:
- Prometheus is bridge-networked and the netavark hairpin gap bars it
from its own host's LAN IP, so a LAN-only prober bind is unscrapable
and a loopback-only bind was claimed to ride the gateway on faith.
The prober now binds 9115 on the podman gateway (scrape path) AND
the LAN (healthcheck/diagnostics); the listen_address line leaves
blackbox.yml so config and Exec cannot disagree.
- comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188:
bridge container names do not resolve in the host netns; the
published 0.0.0.0:8188 answers on loopback there.
- New seam: every observability unit must declare Image= (red on the
shipped file by name); copier test pinned module presence instead
of the removed listen line. Docs updated to the real topology.
PR #69 wired the copier, planner entry and health gate but missed the
three dispatch gates: the client valid_service, the remote
service_is_allowed, and deploy_service's case list. The client gate
rejected --service blackbox-exporter outright; even past it the
dispatcher's catch-all hard-fails on dankclaw (no compose since
2026-09-16). New seam pins the class: every dankclaw service the
remote gate allows must carry a deploy_service case - proven red by
planting a gated, undispatched name.
Measured 2026-10-21 with frank-dragoon live: MemAvailable 15.3 GiB,
iGPU busy 1.75 GiB. The Qwen-Image weight sets (13.3-14.2 GB) do not
fit that budget, and the page's 122 GB /system_stats figure is the
aperture, not headroom. Adds the fit table and the
stop-mask-render-verify procedure (the 3090 bench law, applied to the
APU); ComfyUI.md gotcha links it.
Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
- caddyfile seam learns the documented exception: URL-shaped targets
are legal only inside a prober job (state is per-job; the first cut
leaked it and passed a planted URL).
- observability seams: blackbox unit + planner entries, both copiers
executed for real in sandboxes, alloy engine-validation refusal
(image-gated), dashboard panel invariants.
- AGENTS.md: structural target downs retired.
copy_alloy_config now validates the rendered config with the very
alloy image that will run it, before installing: a misplaced attribute
crash-loops the unit behind a green deploy, and no textual seam can
see it. copy_blackbox_config ships the prober config; the planner maps
the new unit and shared/blackbox/** to the blackbox-exporter service,
ordered before prometheus.
Loki metrics queries group only by indexed stream labels, so the
journal-as-JSON stream made every per-container view structurally
empty (measured 6.5k lines/5min flowing while the board read zero).
A loki.process relabel stage now extracts CONTAINER_NAME into a
'container' label. alloy 1.21's json stage has no on_error attribute
(validated against the pinned image) — format_as_json guarantees the
line shape, so no error branch exists.
observability.json: stale 'structural downs' panel description fixed,
target census matches the post-blackbox job set. logs-overview.json:
panels grouped by the promoted container label instead of the absent
container_name, so the board stops reading empty beside live traffic.
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.
copy_grafana_provisioning named llm-fleet.json alone, so a new board in
shared/grafana/provisioning/dashboards/ silently never reached
/opt/grafana/provisioning. Measured 2026-10-21: the containers board was
merged, seam-tested, and documented as working since 2026-10-20, yet was
NEVER provisioned — its panel fixes were undeployable; the other four
boards existed on disk only because someone hand-copied them once (and
host-system.json there had gone stale against the repo). The copier now
globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run
the copier for real into a sandbox (red against the old hand-listed
body, green against the glob). Vault: the copier law + the containers
incident on Observability Provisioning, the corrected dashboards claim
on Observability.
Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the
dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).
- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
non-matching pairs and the right side collapses to {job} while the left
carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
so the '> 0' filter can never match. Rewritten to share-of-host-total via
label_replace join of the node exporter's MemTotal by host (the parens
form is a PromQL parse error: nothing may follow group_left); 62 series
live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
(ALERTS carries only firing/pending series); description now says so,
per the Stale Signal rule that 0-series-on-ALERTS is not a defect.
logs-overview and observability: all targets already returning data.
Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):
- Observability Provisioning: destroyed the 63-line 'Current delivered
state' block — chain topology/relay/delivery restates [[Observability]]
and [[Alertmanager]] (those are the owners). Its three sole-custody laws
MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
down-beside-healthy -> read lastError; Prometheus alert state in process
vs Alertmanager's --storage.path persistence; candidate-arm metrics are
never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
never reuse a B/token ratio across geometries; pin pool bytes
floor-sized), era-labelled the rates, destroyed the measured ladder
table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.
check-okf 109/0, vault links 0 unresolved, docs seams green.
The lane's child dlopens libxrt_core.so.2 from /opt/xilinx/lib/<multiarch>/
as well; our mounts filled only /opt/xilinx/xrt/lib and the install path.
Since the host's libxrt2 was reconfigured 2026-10-20 the child dies naming
the missing prefix ('No such library
/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2') and the engine
supervisor takes the WHOLE container down on a dead lane — measured
2026-10-21: frank-dragoon boot-looped 88 times with chat dead while the
GPU lines looked healthy.
Each XRT library now mounts THREE times from the same real file (never a
symlink); the multiarch name derives from FRANK_DRAGOON_XRT_DIR's basename
so the bats fixture proves the derivation. Proven end-to-end before commit:
production-shaped scratch boot + the three extra mounts answers the full
boot ledger, npu_api buckets u0..u9, and /health ok at the 262144 ceiling.
A dead lane is not survivable on this engine — arm it only with all three
prefixes (page law updated).
Shape + harness contract on the endpoint page, AGENTS.md, and the OMP
Harness worked example recomputed to the live pair; the 400-zone law now
names the 85 %-comments vs 95 %-config disagreement as the recurring
cause and points at the seam that enforces the arithmetic.
The 200,704 ceiling squeezed the harness contextWindow down to 155,648
(the 400-zone fit law needs ~45k of gap under the ceiling), which is why
long dragoon sessions kept hitting the 400 zone: the harness clamps
max_tokens to window - est - 4096, its chars/4 est under-counts ~3 %,
and config.yml compacts at 95 % while the pair was designed for 85 %.
Ceiling -> checkpoint's native 262,144 (no RAM move: the KV pool stays
401,408 and is what the ledger bills; the band ctx <= pool <= slots*ctx
still holds). The harness window becomes 200,000 with a 62,144 gap.
The fit-law seam now reads the declared window from the vault page (the
one owner) and checks the arithmetic at the live 95 % trigger, so a
ceiling/window pair that drifts back together fails tests/run.sh
instead of a live session.
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:
- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
but never accept()ed, loopback deploy gate passed, every scrape that
reached the other bind timed out on the SYN backlog. Now one thread per
extra bind. Regression test spins --serve on two loopback binds and
requires BOTH to answer /metrics.
- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
Prometheus container: a bridge container cannot dial its own host's LAN
address at all (hairpin refuses published ports too); the podman gateway
answers. Target moved to host.containers.internal:8080; the seam that
pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
gateway plus a negative pin against the own-host LAN dial. Observability
page records the measured reachability law replacing the inferred one.
deploy.sh's valid_service and deploy-remote.sh's service_is_allowed are
two gates for the same door, and they drifted: the remote gate gained
cadvisor (2026-10-20) and gpu-exporter (2026-10-21) while the client gate
never did, so `--service cadvisor|gpu-exporter` died with "is not deployed
on <target>" at the client — the first-ever deploy of a new service is
exactly the case that needs the client gate to know it. dankserver:prowlarr
(planner can emit it for dankserver) was remote-only too.
Adds the four missing pairs to the client gate and a deployment_flow
contract test that diffs the two extracted lists per target — the test the
cadvisor PR should have carried. searxng-network stays the one sanctioned
client-only exception (network unit, no remote handler), documented inline.
ls329 is the newest build on the linuxserver channel and Plex 1.43.5.11029
is their newest release; the live pin (ls325 / 1.43.4.10903) is two Plex
releases behind. Digest-verified at pin time: the tag's manifest digest
equals the registry's current 'latest' (sha256:8f655bad...e9e2334), the
same law the page records. Quadlet restart runs PMS's usual one-minor
schema migration on first boot; the deployer's /identity HTTPS gate
tolerates the first-boot migration up to 600 s.
Rollback: revert this commit (one line).
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.
scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.
It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
The stored 3.28 chars/token (measured 2026-10-11) had drifted to 3.20 as the
repo's docs grew, so leg B sent ~201,900-token prompts and took a legitimate
400 from the engine's own window check — which reads as a capacity-contract
failure on a healthy endpoint. One ~4k-token request now prices the ratio, and
the leg sizes to 97 % of the deployed window.
The shape block and boot ledger now quote the live 1.8.1 boot: experts 20.72 GiB
(78 %) / kv cache 5.64 GiB (21 %) = 747,492 tokens, 3.7 full-length sessions,
`--checkpoint-slots 40`, `queue_cap 32`, sampler 12 rows, `experts-from-host`
8.99 GiB a rank.
The KV Capacity law is rewritten from the pinned source instead of inference:
`core/mem/vram_budget.cpp:146-156` funds the cache as `elastic - expert_floor`,
`core/mem/kv.cpp:266` sets its ceiling from `max_seqs x max_ctx` (demand, never
funding), and `core/place/planner.cpp:100-104` makes the expert floor fall
one-for-one with `--host-pool-mib` — which is why +4,224 MiB of pinned host pool
bought the pool and cost 3.01 GiB of residency a card. The old "funds ~90 %,
never 100 %" phrasing was inference; the byte-identical 45,568 row is the ceiling
case, not a mystery. Slot-cost law carries the four-slot price (state +
checkpoint store + rows, still under the 25-row exact-wire bound). The two-slot
era is one row in Eras with its rollback commit. Date stamps in the trio
comments corrected to 2026-10-20.
Frank's order: 4 slots, a 700k pool, round UP if the resolver cannot land it
exactly. The cache target is MAX_SEQS x MAX_MODEL_LEN, so the demand is
800,000 fp8 tokens; the resolver funds ~91-93 % of demand (three boots of
evidence on the endpoint page), which clears 700k, and the VRAM has to come
from the expert plane -- which has to go somewhere, so HOST_POOL_MIB moves with
it. expert_vram_floor (core/place/planner.cpp:100-104) is exactly
plane + 2 units - host_room, and the live boot ledger pins that plane at 67.60
GiB a rank (23.73 GiB floor at 44,928 of room), so 49,152 drops the floor to
19.61 GiB and leaves the cache ~6.9 GiB: enough for the whole target, with the
leftover returned to the experts.
Costs, both real: pinned host RAM 87.75 -> 96.0 GiB (the pool is pinned once per
rank) and ~3.2 GiB a card of expert residency, read over the x8 Gen5 link when
the router picks a host-resident expert. The boot ledger's vram_kv line is the
referee; if it comes back under 700,000 tokens HOST_POOL_MIB goes up 1024 and
this redeploys.
TTM_MIN_MIB 150,000 -> 162,000 for the new pin ledger (2 x pool + PLE 48,824 +
prefix 8,192 + MNBT 2,048 + the auto checkpoint store, now 40 slots of 112 MiB =
161,848 MiB), and the bats ledger guard now counts the checkpoint store because
it grows with the slot count. The two env-value assertions in that seam are
bumped with the values they guard.
A prefill chunk that shares a step with decoding sequences holds the device for
the chunk's whole time: measured on this shape, decode-only steps 15.5 ms
against chunk steps 166 ms, and 4.7 % of the steps carry 34 % of the engine's
step time. Harness-visible decode falls 166 -> 13 tok/s (per stream 6.7 = one
token per shared step) once concurrent prefill passes 5k tok/s, and TTFT p90 for
an arriving request goes 2.4 -> 18.0 s.
BUSY_PREFILL_CHUNK stays `auto`: it is a prefill-speed dial, and the source
refuses to halve the chunk for decode rows alone -- which is why
engine_steps_held_total read zero for the whole 34.7 h boot. PREFILL_INTERLEAVE
is the knob that acts on the measured mechanism: N decode-only steps follow any
chunk step taken beside decoders. N=1 is the conservative first arm -- a stream
takes its token from the chunk step and one from the held step (~2 tokens per
182 ms instead of one per 166 ms); the cost lands on the prefilling prompt's
TTFT and on requests still queued, not on anything decoding.
Rollback: PREFILL_INTERLEAVE=0. Verification: the flag must reach argv (the
serve script only passes it when > 0), engine_steps_held_total must leave zero,
then the §9 bucket pair must move before N rises.
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.
cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.
cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.
Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
dank
Frank