Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):
- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
container name, not the loopback publish, with a fixed uid so dashboards
reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
rate, e2e latency p95, and decode/prefill throughput, with an endpoint
variable covering both frank-medium and frank-large. Every series name was
checked against the live /metrics endpoint: this fleet already shipped one
comment naming a metric that does not exist, and a dashboard full of those
would be worse.
Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.
Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
The capacity work in this runbook was done with ad-hoc probes; the same signals
now have history in Prometheus, so the page says which series to watch and
admits the honest gap — this stack has no alerting, so a saturated pool shows up
on a graph after the fact rather than paging anyone.
vllm:prefix_cache_hit_rate does not exist in this build; the exported pair is
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give
the hit rate. A comment naming a series nobody can query is worse than no
comment.
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.
frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.
The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
Two more hypotheses, both measured on the same three-family workload (one
~80k-token family plus two ~50k-token families, each turn re-sending its
document):
1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt
sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too
(22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%"
is an average across every prompt that replica ever saw, not a verdict on
any one request, and the pinning policy is safe.
2. Serialising large requests does not fix the overload. Admission control
(at most one >16k-token request per card, small requests untouched) gave no
clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums
34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was
reverted like the placement change before it.
The arithmetic now closes: with three families on two cards, one card holds two
families and must carry 2 x 50k of cached prefix plus the 50k of the sequence
generating — 150k against 122,636 tokens. Those co-tenants evict each other
regardless of concurrency limits. The wall is capacity, and the runbook says so
with the numbers.
The obvious next lever for cache thrash was placement: send a new conversation
to the card carrying less context instead of alternating. Implemented it
(decayed prompt-token estimate per replica, continuations still pinned) and
A/B'd it against the counter on the workload it targets — one ~80k-token
family plus two ~50k-token families, each turn re-sending the same document:
counter heavy 13 turns median 25.4 s (23% cached)
medium 34.7 s / 45.2 s (14% / 15% cached)
balanced by load heavy 5 turns median 46.7 s (20% cached)
medium 23.4 s / 34.3 s (0% cached, small sample)
No improvement, so the change is reverted and routing.py keeps the counter.
The reason is arithmetic: a running request's live sequence and its cached
prefix draw on the same pool, so a card running two 50k-token generations has
already spent its 122,636 tokens before any cache entry is considered.
Placement chooses which card is over capacity, not whether one is.
Third hypothesis measured and rejected tonight (with SPEC depth kept and CHUNK
disproved); the runbook records all three so the next operator starts from
evidence rather than from these ideas looking obvious.
Rebooted dankserver unattended and verified without any manual step: all three
units active 59 s after boot, both replicas healthy ~3.5 min after boot, real
completion through :7770 — the failure mode this repo has been bitten by before
(a service that only ever ran from compose and stayed dark after a reboot).
Records why it works: [Install] WantedBy=default.target wired into the
generator's default.target wants, plus Linger=yes, plus a disk-backed compile
cache so a cold boot needs no warm-up beyond the model load.
A 25-minute soak with three concurrent large-context agent loops (each turn
re-sending a ~30k-token document, growing history) came back clean — 155
requests, 155 answers, no errors, no failover, both replicas steady — but only
25% of turns hit the prefix cache, the rest paying a 13.9 s median re-prefill.
That is the capacity budget showing: 122,636 KV tokens per card is about two
30k-token conversations with their cached prefixes, and a third family on the
same card evicts.
Chasing the obvious lever disproved it. Available KV per card is 5.44 GiB at
CHUNK=8192, 4.22 GiB at 16384 and 4.16 GiB at 4096 — where the engine refuses
to start. The budget is not monotonic in the chunk size, so 8192 is a sweet
spot rather than a compromise and freeing KV needs another card, not a knob.
The proxy trusted its health table, which the watcher refreshes every 10 s. A
replica that restarts — and a deploy restarts both — therefore produced a
window in which requests routed to the dead card got a 502 back, even though
the other card was serving the whole time.
A failed connection now demotes that replica immediately and retries the
request on another live one, before anything is written to the client.
vLLM's OpenAI endpoints are stateless, so re-sending a request that never
reached an engine is safe; a failure after streaming has begun is still not
retried. The failover is logged, because a silent one hides a card that is
down.
Measured with card 1 stopped mid-flight: six requests, six 200s, one failover
line in the log, no client-visible error. Before this commit that burst
returned 502s.
Upstream ships dflash depth 7 as a content-dependent default measured for TP=2
on their corpus. Re-measured on one card here, two runs per setting, 16
repo-derived prompts at 512 output tokens:
SPEC=7 conc1 46.2 tok/s, TPOT 20.8 ms, 3.86 tok/update
conc4 118.7 / 127.0 tok/s
SPEC=5 conc1 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update
conc4 136.1 / 129.4 tok/s
Step time is nearly identical at both depths (~79 ms vs ~80 ms single stream):
7 wins by landing more tokens per step, 5 wins at conc 4 by ~8% because more
narrow verify steps pack better. Depth 7 stays — this fleet is interactive
agent loops where per-turn latency is what is felt — and the runbook now
records the numbers and the one-line switch for a batch-first workload.
Raising the prefill chunk to 16384 grows the arena enough that the KV cache
falls to 4.22 GiB against the 5.21 GiB a 114688-token context needs, so the
engine refuses to start. Measured single-card prefill at 8192 is ~2.5k tok/s
(~15k-token prompt, 5992 ms TTFT); the trade would mean cutting context for
unmeasured prefill gain, so the default stands.
The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.
Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.
So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.
The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.
ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.
The single-GPU replicas died with torch.OutOfMemoryError during KV
allocation (2.16 GiB requested, 116 MiB free, 29.34 GiB already held by
PyTorch) because the entrypoint blanked KV_MEM=0 into an omitted flag. The
image's patched KV sizing reads an explicit 0 as 'profile this shape'; with
the flag absent vLLM sizes the cache from --gpu-memory-utilization alone and
overshoots. Passing 0 literally is the shape the bench numbers were measured
on, and MAXSEQS gets its default back under set -u.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling;
decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the
root-cause evidence chain, tuning levers, and co-tenancy traps
- AGENTS.md: fleet topology row + Active Services section for the ik
deployment; halogen demoted to rollback; bench policy unit type fixed
- LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE,
halogen-flash-server retired)
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).
- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records
the hardware swap, the self-contained gfx1201 build, the measured layer-vs-
tensor decision (llama-bench multi-device syntax gotcha included), the
tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm
posture. AGENTS.md, README, docs indexes, Dankserver System Specs,
deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename;
freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED.
Work log entry added.
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
Follow-up (same session, measured): llama-bench on the host with the correct
multi-device syntax (-dev ROCm0/ROCm1) overturned the tensor choice:
single R9700 pp4096 1160.9 / tg128 19.42 (d32768: 512 / 18.4)
layer 2x pp4096 1991.2 / tg128 18.76 (d32768: 859 / 17.6)
tensor 2x pp4096 271.8 / tg128 26.26 (d32768: 229 / 25.1)
This build has no RCCL, so tensor mode falls back to a "meta-backend
butterfly" allreduce and prefill collapses 7x. Served path with MTP:
layer 35.7-46.2 tok/s vs tensor 28.7. SPLIT_MODE=layer, draft pinned to
ROCm1 (the card holding the last target layers). Served prefix cache with
the draft head verified live: turn 2 reprocessed 22 tokens instead of 949.
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger
printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous
2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also
documents the ~45-60 s first-request PING silence after a restart (the
healthcheck's HealthStartPeriod covers it; do not restart the unit).
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).
Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.
Answering "does the FreeToken runtime use sampling arguments?" -- yes, and
nothing in our config revealed it. serve-frank-small.sh passes no sampling
flags; the behaviour is the runtime default:
freetoken/server/args.py:31 sampling_defaults: str = "model"
which fills temperature/top_k/top_p from the checkpoint's
generation_config.json -- this one ships do_sample:true, temp 1.0, top_k 20,
top_p 0.95. Proved live: three identical requests with no sampling params
produced three different reasoning traces (487/540/577 chars).
The asymmetry with frank-large is the part that will bite someone: a silent
client samples on frank-small and decodes greedy on frank-large, so the same
request is reproducible on one endpoint and not the other. Documented with the
determinism options (explicit temperature:0, or --sampling-defaults none).
Also recorded two request-level traps found during the probe: thinking eats the
whole max_tokens budget and returns empty content with finish_reason length,
and the API accepts unknown fields with a 200, so a 200 proves nothing about
whether a key was understood.
Ledger confirms the change landed: "3 slot(s) over one 524288-position pool",
"slots: 1 -> 3 POOLED, 111.0 MiB each", held 103.5 -> 103.4 GiB.
Recorded the oversubscription nuance that the old row missed: the slots were
never the binding constraint on context. 4 x 262144 exceeds the 524288 pool,
so admission was pool-bound all along -- two full-length conversations, or
four at 131k. Lowering the cap trims simultaneous requests, not capacity.
Concurrency cap trimmed to the real fleet size. Each slot is 111.0 MiB of
O(1) state, so this hands ~111 MiB back on a host sitting at 7.1 GiB left
for everything else.
Note the slots already oversubscribed the pool (4 x 262144 > 524288), so
admission was pool-bound regardless: at most two full-length conversations,
or four at 131k. This does not change how much context fits -- it caps
simultaneous requests at three, and a fourth now queues.
Raised /swap.img from 8 GiB to 64 GiB on Dankclaw. Host-level state with no
repo file, so it was invisible to the repo until now: fstab line 12 drives the
generated swap.img.swap unit, ext4, fallocated hole-free, swappiness 60,
zswap off. 748 GiB still free on /.
The reusable part is the ordering. The old file held 4 GiB of live pages and
MemAvailable read 73 GiB while only ~5 GiB was genuinely free -- the pinned
model weights are counted as reclaimable page cache, so the headline lies by
the size of the model. Swapping off before the replacement is online would have
pushed those pages into RAM that does not exist. Creating and enabling the new
swap first let them migrate instead: swapoff took 65 s and the host never
wobbled.
New page serverAppConfigs-docs/Dankclaw/Swap Configuration.md, linked from
the index, plus a pointer in the AGENTS.md memory-model section so nobody
reasons about headroom without checking swapon.
Old 8 GiB file staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/
pending approval to purge.
Seven restarts measured. New section "The reserve cannot buy air": reserve 16
was deployed and held exactly what 15 holds — 103.5 GiB, 7.1 GiB left to the
host. The flag is a term in the fit-check inequality, not a reservation, so it
has only two regimes: below the pool's requirement it changes nothing
observable, above it the pool silently disappears.
Sizing rule now defers to the engine's printed requirement (budget >= 36.5 for
524288 at max_tok 32768, from "need ~35.0 GiB plus 1.5 of margin") over my
modelled 37.0, and the ledger table carries the reserve-16 row.
Tested 16 live. Pool stayed 524288 (no clamp), server held 103.5 GiB and
reported "7.1 GiB left for everything else" at BOTH 15 and 16 — identical
footprint. HALOGEN_HOST_RESERVE_GIB feeds the fit-check budget only; it is not
an allocation, so raising it cannot give RAM back to the co-tenants. Its sole
observable effect above 16 is to take the KV pool away.
16 therefore cost half the fit margin (1.0 vs 2.0 against the engine's 36.5 GiB
requirement) and bought nothing. 15 is strictly better at the same cost.
Recorded the ceiling (16) and the real air levers in the unit comment: pool
262144 frees 7.2 GiB, max_tok 16384 frees 8.4.
Operator asked for 18. 18 is not available: the engine's own fit-check line,
read from a clamped boot rather than modelled, requires budget >= 36.5 GiB to
keep 524288 at max_tok 32768 ("524288 positions need ~35.0 GiB (plus 1.5 of
margin)"). budget(18) = 35.5, so 18 silently un-doubles the pool we just paid
the reserve for. 17 fits by 0.02, inside the noise of the 67.7 resident-weights
term. 16 is the largest reserve that survives the check with real margin:
37.5 budget vs 36.5 required.
Recorded the ceiling and both remedies in the unit comment. The reserve is now
spent, so a future stall is answered by the pool or the arena, not here.
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.
Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.
Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
Neither lever alone could give both features. At reserve 24 the device budget
is 29.5 GiB and max_tok 32768's 37.0 GiB need always lost the doubling to the
fit-check clamp; at max_tok 16384 the doubling survived but prefill paid 8-9%.
Dropping the reserve to 15 moves the budget to 38.5:
121.2 host - 67.7 resident weights - 15 reserve = 38.5
need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB spare
So the payment is page cache for the n-gram lookup table rather than either
feature. 15 GiB, against a table of 47.7 GiB read through the file cache.
Two reasons this is a survivable risk rather than a cliff:
- The engine's own ledger reported only 12-17 GiB left at the previous 24
setting, so the declared reserve was never held literally; 15 may cost
less real cache than it reads as.
- The failure mode is legible: long prompts fault heavily against a busy
disk, and raising the reserve is the documented remedy. Not the silent
clamp, and not the 65536 no-boot mode.
1.5 GiB of headroom is thin and deliberate. Documented in the unit as a
matched pair with the pool: raise the reserve back to 24 and the doubling
quietly goes away again.
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:
kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all
so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.
Two corrections to what I wrote earlier today:
1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
spare it" line, print ONLY when the fit check had to clamp something. The
09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
of a warning in old logs was never evidence of headroom.
2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
stated budget -- and has been the whole time. It boots and serves with
12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
floor, not a wall.
The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
Operator call. Prediction from the measured model: 524288 positions cost
35.0 GiB against a 29.5 budget, so HALOGEN_KV_POOL_FIT clamps the pool to
262144 (27.8). Rollback is a single revert; 16384 was the config that made
the doubling real.
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
Operator call, measuring what the prefill arena actually costs against
upstream's do-not-raise warning. Rollback is one revert: the settled
config was 16384, which made the 524288 pool fit at 26.6 GiB.
The first draft of these notes estimated the doubled pool at 103.5 GiB
held / ~5 GiB host remainder. Both numbers were wrong — they costed the
KV alone and ignored that the arena shrinks when MAX_TOK drops. Measured:
pool 262144 / max_tok 32768 -> 96.3 GiB held, 12.1 GiB host left
pool 524288 / max_tok 16384 -> 94.6 GiB held, 13.8 GiB host left
The doubled pool is 1.7 GiB cheaper overall. Also records the failed
first attempt verbatim, since the silent-revert failure mode is the thing
a future operator most needs to recognise, and adds the HALOGEN_MAX_TOK
row to the deployed-flags table (the "baked 32768" line it overrode was
stale).
The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT
measured the device budget at startup and lowered the pool straight back
to 262144, while the deploy still reported healthy in 12s.
The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0
reserved for the n-gram page cache and the OS = 29.5 GiB of device
budget, and that single budget carries the KV pool AND the prefill arena.
At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions
need 35.0 GiB against 29.5 available.
HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the
524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is
about 8-9% of prefill throughput, per upstream. This is the engine's own
prescribed fix, quoted verbatim in its refusal message.
Lowering HALOGEN_CTX was the other option the engine offered; rejected
because both clients declare 262144 as the context window, so it would
break the client contract rather than pay for the pool.
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Documents the root-cause chain for the marreta.franksdank.cloud login wall:
null invalidation_flow (outpost serde crash) -> missing flows -> and the real
blocker, ProxyProvider.application=None so the embedded outpost's
filter(application__isnull=False) never loaded the provider. Notes the
providers are live-only (no repo blueprint) as a provenance gap.
16-marreta.spec.ts (standard project) asserts the user homepage's Marreta
open link actually reaches the Marreta app through the full guard chain
(Anubis PoW -> Authentik SSO -> app), and explicitly NOT the authentik
login flow. Guards the 2026-09-12 incident where marreta-proxy was created
without a linked Application, so the embedded outpost never loaded it and
the homepage link parked on a login wall.
Wires spec 16 into the standard project testMatch. Verified passing live.
Adds the Marreta content-reader tile (http://marreta.franksdank.cloud,
probe 127.0.0.1:8082 via gluetun's publish) to the user grid so regular
authenticated users get a link to it. Registry-driven e2e picks it up
automatically.
The 2026-09-12 homepage outage slipped through because the live-fleet
Playwright suite was a manual 'MUST invoke' that deploys skipped. Wire it
into deploy.sh as the [3/3] gate: after a behavior-affecting deploy,
run tests/e2e/run.sh against the live fleet; a failure exits DEPLOY
INCOMPLETE. Add --skip-e2e as an emergency escape hatch. Update the
AGENTS.md deployment-completeness contract and flow diagram to match.
Verified: bash -n clean, --skip-e2e parsed, unknown flags rejected,
seam suite green (only the pre-existing comfyui test 73 fails).
Deployed live at 91e9410 (gluetun+marreta+caddy), authentik provider pk 13,
guard chain verified end-to-end. Documented that Laravel APP_KEY must carry the
base64: prefix — a bare base64 key 500s every app route with 'Unsupported
cipher or incorrect key length' while /up still passes.
The scoped runner (deploy-remote.sh) already knew marreta; the top-level
deploy.sh valid_service() gate did not, so --service marreta was rejected
before reaching the planner.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.
Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.
AGENTS.md's boot-persistence gap note is replaced by the resolution.
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.
The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
Hermes hardcodes response_format=json_schema on its title call. No local
back-end implements constrained decoding — frank-small and frank-large both
answer 400 'response_format json_object/json_schema is not supported' — so
every session logged a title failure and fell back to a message-derived
title. The NPU endpoint that used to absorb this call was retired on
2026-09-11.
The call now goes out over the Responses transport (the only one that does
not forward response_format) with thinking off. A 64-token cap cannot hold a
thinking pass plus a title: with the default reasoning effort the model spent
all 64 tokens thinking and returned empty content, which is why disabling
reasoning is part of the fix and not decoration.
Verified against the live endpoint with the real prompt and the real
extraction path: raw {"title": "Friendly greeting"} -> title
'Friendly greeting' in 3.4s, where the previous configuration returned 400.
Rebooting Dankclaw for the amd_iommu=off change exposed five services that
exist only as compose.yaml services with no Quadlet unit (prometheus, grafana,
loki, alloy, node-exporter): they did not come back and had to be restored by
hand. The boot-persistence rule already requires units for them; this records
the concrete gap so the next reboot is not a surprise.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
The unit comment always claimed the webui drives the same agent state as
the gateway, but the image bakes HERMES_HOME=/opt/data and no unit
overrode it. Podman auto-creates a fresh anonymous volume at every
image-VOLUME point on container recreation, so each restart reset the
webui's config home to the image-default template (provider: auto ->
openrouter.ai with no API key), while the gateway read the corrected
/data/config.yaml. Pin HERMES_HOME=/data so both processes share one
config/state root and config edits propagate without a recreate.
Live config was still routing on pre-rename aliases (frank-default-large,
frank-default-small) after the 2026-09-03 frank-large/frank-small/frank-npu
rename cutover; hermes-webui resolved those ids to nonexistent models and
requests failed. Live config fixed in place (backup kept on-host as
config.yaml.bak-20260905); this backup now mirrors the live shape: named
providers for aux routes, title_generation/triage/kanban routed to their
actual models, moa + mcp sections, signal disabled.
Wget-style (no-JS) clients must never receive application bytes from guarded
services: copyparty serves authenticated traffic with anonymous read perms, so
the guard chain is the only boundary between an unauthenticated client and the
file bytes. Assertions target leakage markers in app code (Copyparty UI title,
homepage panel grid, services.json JSON shape), not gate internals, so any
guard-route regression fails the suite while any gate implementation passes.
Also route 15-*.spec.ts into the guest project (empty storage state).
Flip the homepage Sonarr card from the user category to admin and repoint
its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded
public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its
post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A
record with it. No public Porkbun record ever existed (NXDOMAIN at the zone),
so after this there is no WAN path to Sonarr: users request TV via Seerr
(guarded), admins reach Sonarr's own UI from the LAN only.
Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
Rootless podman on dankserver already maps container root (PUID=0) to
the dank user (uid 1000) by default — /proc/<pid>/uid_map reads
0->1000, 1..->100000+. Verified across plex, tautulli, jellyfin,
calibre-web, copyparty, romm. userns_mode: host is unnecessary, and
podman-compose 1.5.0 cannot apply it anyway (compose services run in
pods; --userns and --pod cannot be set together). The real fix for the
never-booted arr stack was chowning the root-owned /opt/<svc>/config
dirs to 1000:1000.
The arr services were committed (b9de886) with PUID=0 but no userns_mode.
Under rootless podman's default subuid mapping, container-root cannot write
host-root-owned bind mounts, so every app died in logger init
(UnauthorizedAccessException on /config/Sentry) before binding its port —
the stack 502'd since it was first brought up (2026-09-04 00:33).
Per the fleet convention (AGENTS.md, plex/tautulli migration): PUID=0 +
UserNS=host maps container-root to dank (1000), which owns /opt data roots
and /mnt/entropy. Config dirs chowned 1000:1000 at bring-up.
Also drops a duplicate 'networks: [appnet]' key on prowlarr.
The arr stack (prowlarr, radarr, sonarr, lidarr, readarr, qbittorrent)
had A records in addn-hosts but no Caddyfile blocks. This adds:
- Six public vhost blocks in the dankserver section, each proxying to
192.168.0.235:<port> (exempt by construction: own auth, no Anubis)
- Six @arr named handlers inside :8017 for authenticated traffic routing
All 233 bats tests pass.
Boot tag flips from qwen3.8-distilled-heretic:9b to nuextract3:4b (verified
live: per-request routing already serves every registered tag; the boot tag
only decides which model is resident at start, so the first request after a
swap pays the load cost). FLM_CTX_LEN=-1 drops the global --ctx-len override
so each pack loads with its own trained context (nuextract3 32768, gemma4
131072).
Live run against the fleet showed the outpost redirects direct navigation to
auth.franksdank.cloud with the guarded host preserved as redirect_uri, rather
than rendering the flow UI in place. The security property under test is that
an unauthenticated visitor never sees app content; assert that shape (uidField
visible + no leak markers + gate is either in-place or a redirect carrying the
return path) instead of hostname preservation.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
14-guard.spec.ts: every authentik-guarded vhost (www, search, romm, files,
seerr, grafana, prometheus, loki) must challenge an unauthenticated direct
navigation; plex is asserted exempt (PMS answers directly, no IdP form).
11-admin.spec.ts: admin direct-nav to grafana/prometheus/loki passes through
with host preserved and no uidField challenge.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Post-change doc reconciliation for the hermes vhosts + DNS deletion batch:
- Hermes-Dashboard.md: access is now the hermesadmin.franksdank.cloud vhost
(direct Caddy proxy to 192.168.0.143:9119, no guard chain — exempt because the
dashboard enforces its own basic-auth login), plus the raw LAN port. The old
text claimed an Authentik-guarded public vhost, which was decommissioned.
- Hermes-WebUI.md: description updated — the webui is now also reachable via the
unguarded hermes.franksdank.cloud vhost (its own password auth is the gate).
- LAN-DNS.md: "add a name in two places" corrected to a two-or-three-step
decision; internal-only names (prometheus/loki/grafana/seerr) deliberately get
no public A record and resolve on the LAN only, and DNS-01 issuance is
unaffected because the challenge is a TXT record in the same zone.
- Seerr.md: DNS row corrected — addn-hosts only since 2026-09-02; NXDOMAIN on WAN.
- hermes-dashboard.container: unit comment no longer claims Authentik termination.
No behavior change in this commit; tests 233/233 green (docs-completeness seam included).
Both Hermes services get direct (unguarded) vhosts: each authenticates itself
(webui password; dashboard basic_auth in /data/config.yaml), so no Anubis
challenge and no Authentik proxy provider is required. Upstreams are the host
LAN IP because Caddy itself sits on dankclaw_default, where the gateway
address 10.89.1.1 has nothing serving 8787. Internal-only by DNS absence:
hermes.franksdank.cloud already resolves via Pi-hole addn-hosts; hermesadmin
is added in this same commit so the caddyfile/addn-hosts cross-check (test 178)
stays green.
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.
Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
/home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
daemon itself is a host user unit (signal-cli-daemon.service) bound to
0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
GET /health from another LAN machine returned 200) — the SSH tunnel is an access
path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
tunnel via the shared named network searxng.network, not service:gluetun.
Docs only — deploy is a no-op for this change.
PMS 1.43 serves TLS-only on 32400: a plain-HTTP GET gets a TLS alert and
curl reports "empty reply from server" (exit 52, code 000), so the card read
down while Plex answered 200 at /web/index.html over HTTPS. TCP connect is the
honest liveness signal for this port; the public URL stays HTTP because Caddy
terminates TLS and re-keys the path.
The Hermes card probed 127.0.0.1:8090, a port retired when the hermes
stack moved from compose (8090 web dashboard) to quadlets: the dashboard
now publishes 9119 and the webui listens on 0.0.0.0:8787 in the host netns.
Nothing has listened on 8090 since, so the card reported "down" while the
gateway, dashboard, and webui were all running.
Probe 127.0.0.1:8787 instead — the webui runs HERMES_HOME=/data, the same
agent state as the gateway, and /health answers only when that server is
alive (uptime, sessions, active runs). The prober runs on Dankclaw, so the
loopback probe hits the host netns where the webui binds.
The C5 probe failed for the wrong reason: plex.container lost its Network=
key in the same edit churn that ate tautulli's ContainerName. Plex ran on
the default podman network — published 32400 still worked (clients fine)
but tautulli could not resolve the plex alias, breaking the seeded
PMS_Web_Host=plex. Restore appnet membership; structure now parallel to
tautulli.container. Lesson recorded: after multi-hunk line edits on these
unit files, diff the whole file — key-order churn has eaten three keys
already.
Without ContainerName= the v1 generator names the container
systemd-tautulli; the fleet references containers by their plain names
(podman inspect/exec, deployer health checks). Container was healthy but
renamed — caught by post-cutover verification, not by the seam; the seam
only asserts file keys, so pin the name here and keep plex/tautulli
structures parallel.
- tautulli.container: HealthCmd switches to the repo's proven curl pattern
(jellyfin unit) — quadlet's systemd-style value parsing mangles embedded
double quotes; PublishPort=8181:8181 restored (a mis-aimed line edit
replaced it; the PublishPort seam test caught the regression)
- deploy_plex readiness gate: 36x5s -> 120x5s; the 2026-09 cutover showed
the first container boot of a years-old library runs ~10 min while PMS
pre-binds 32400 and empty-replies; 180s failed the gate twice with a
healthy, still-booting service
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- runtime rows: podman quadlet under the systemd user generator, pinned
image digests, in-place bind mounts, identity model, appnet aliases
- cutover/rollback section on Plex.md (mask-native-unit lever, decommission
gate); drop the false 'updates are apt-only' and 'snap refresh' gotchas
- update path is now deploy.sh dankserver --service plex|tautulli
- plex: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322 (verified ==
ghcr latest digest; plan's ls9 pin superseded by the re-verify clause),
/var/lib/plexmediaserver bind-mounted in place as /config, /mnt/entropy
mounted ro at the same path, PUID=997/PGID=984 + UsernsMode=host so the
container's file identity is byte-compatible with the native unit
- tautulli: lscr.io/linuxserver/tautulli:v2.18.1-ls242 (verified == ghcr
latest digest), fresh /home/dank/Containers/tautulli config root,
PUID=1000/PGID=1000, python3 /home healthcheck
- both on dankserver_appnet with aliases; 32400/8181 published as before
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
test-changed cutover case, quadlet seam enumeration + new
userns/PUID=997/PGID=984 ownership-contract test
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
- qB 5.x's 204 login breaks Readarr's retired client at runtime; 4.6.7 answers
legacy 200+Ok. Pin recorded in the unit and documented.
- Prowlarr download client created via schema template; all five clients test
200 through the podman gateway.
- PBKDF2 gotcha: qB 5.x conf rewrite left a hash 4.6.7 rejects; fresh rewrite
of the same line restored login.
- E2E: torrent through the Panama tunnel at ~17 MB/s, completed into Finished
Torrents, deleted with data.
Frank