DFlash2's sidecar (1.92 GiB + buffers on ROCm1) kept the 393216 pool off the table; the MTP head lives in the target file, so its cost is one nextn layer's KV (4096 B/pos) plus small buffers. Fit: ROCm0 29.77 / ROCm1 28.32 GiB, over the floor, 34:31 the balance point at this pool. n_max 3 is the measured MTP winner. The seam's fit is now drafter-aware, the slack is the deployed integer gap (1483 MiB), the ctx cap 393216 with the rope caveat (top 131k extrapolates on the base rope).
CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained
window. Measured fit arithmetic on the 3090: 1 full-context slot idles
at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed.
A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool
allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit
on purpose: fit maximises the TOTAL pool and would divide a smaller
window across slots at parallel>1. The slot-grid seam now pins both the
full-window division and the budget inequality so a future capacity
bump fails pre-deploy, not in the boot allocation.
Audio verified while measuring: the mmproj's audio encoder transcribes
input_audio parts verbatim (JFK WAV sample). Header comments updated.
The Qwen3.8-27B IQ3_S file carried its own MTP head; Gemma4's ships as a
separate assistant file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant,
4 nextn layers) that loads via --model-draft and shares the target's last
layers' KV. The draftless ngram-mod matcher chains behind it in the same
--spec-type value — two context-backed drafters cannot share one process,
so this chain is the only shape that carries both.
KV is q8_0 (16 KiB/token/layer vs 32 at f16) and CTX_SIZE is unset so
--fit sizes the pool against what the 7.4 GiB of payloads leave on the
24 GiB card. build-cuda was rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON,
which quantized KV on CUDA requires. The three seams that pinned the Qwen
contract (MTP-in-file, the Qwen mmproj name, the f16 KV arithmetic) now
pin the Gemma4 contract instead, and the deploy handler's presence check
points at the new payload.
The 36:29/327680/vision deploy came up healthy and text-only: image requests
answered 500 'image input is not supported - hint: ... provide the mmproj'
with no mmproj line in the load log, so llama-server fb27a525 does not
autoload mmproj*.gguf from the model directory. Name the file, the way
frank-small's serve script already does, and pin the flag in the seam.
The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV,
and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point,
computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82
GiB free, both above the 1.5 GiB floor.
36:29 is one layer over the 35:30 balance on purpose: at this context the
balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0
to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so
the seam's slack is that gap rounded up (3225 MiB) with the trade written down,
and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries
no scaling; positions past the trained 262144 extrapolate on the base rope).
Vision: NO_MMPROJ=0, the serve script now refuses to start without the
projector when vision is on (a missing file is a deploy error, not a silent
text-only server), the download script fetches and byte-verifies it, and the
deploy handler's preflight checks for it. The projector is already on the host,
byte- and sha256-verified against the Hub.
The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell
grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau:
ubatch 256 384 512 768 1024 1280 1536 1792 2048
tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078
The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out
1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the
shape, not noise. The logical batch is throughput-neutral (2048..8192 all
within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized
by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell.
So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB
more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2):
36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed)
36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone)
35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed
The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1
holds the output layers and the drafter's graphs -- so it shifted the balance
toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is
one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split
can be tuned finer than a layer, which is why 34,31 is not better.
Re-fitted the seam with both ubatch coefficients measured at this shape and
re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to
0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 ->
0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB.
Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the
shape it actually cares about -- two shares that add up to the model's 65
blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer
breaks a test about parallelism. The numeric contract lives in the seam test.
DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.
The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:
39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48
38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05
36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.
Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
--verify-only fail on a payload that was in fact correct. Repointed at z-lab
DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
point; it now passes 36:29 and fails every other split measured, including
1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
are gone; the build's defaults apply.
The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card
floor, and the context maximised inside it.
Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric:
ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid
~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78.
One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement:
41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free.
That balance is also what maximises the context -- an unbalanced split would have
to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's
trained window (262144), now split over two slots of 131072. Verified under two
concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2,
n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots
(acceptance 0.38/0.22 and 0.52/0.43).
8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to
allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in
the env's sizing table.
The unit seam now models both cards (share + draft + ubatch-scaled buffers,
calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor
on each, and asserts the two cards stay within 0.5 GiB of each other. Both
assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and
a 1:1 split) rather than pass by construction.
DSpark declares no sliding window, so its draft context -- built with
n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8
kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where
DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at
33.9 GB of 34.2 GB and the first draft graph capture died inside
common_speculative_impl_draft_dflash::process with "ROCm error: out of memory"
-- after the model had loaded and /health had answered, which is why the deploy
looked like a crash-loop rather than a sizing failure.
Handing three target layers to ROCm0 restores headroom on both cards at the full
window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free
(was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam
now models the busier card as a skewed share plus the drafter's full-context KV,
calibrated to the rocm-smi figure.
AGENTS.md gains the durable lesson: read sliding_window out of the drafter's
GGUF header before sizing a spec-decode deployment.
DSpark (Q8_0, 1.27 GiB) replaces z-lab DFlash2 (1.92 GiB): same 5-layer
block-diffusion shape, Markov head instead of the candidate selector, and
block_size 7, so the draft depth drops 7 -> 6 (llama.cpp asserts
n_max <= block_size - 1).
The serve script's dspark branch never passed --spec-draft-model (only the
dflash branch did), so dspark was not deployable as written; the model file
gate now covers both types. Download script, env and the unit seam track the
new file and its exact byte count.
hermes-webui is Network=host, so 8787 is a bare host-netns bind, and netavark
DNATs only *published* ports. From Caddy's bridge netns 192.168.0.143:8787 and
10.89.1.1:8787 are both refused, while host.containers.internal:8787
(169.254.1.2) answers 200 — measured live 2026-09-16 from inside the Caddy
container. The dashboard's published 9119 answers on that same LAN IP, which is
what made the old upstream look sound.
The vhost comment asserted "192.168.0.143:8787 is open" from that bridge; it is
not, so the claim is replaced with the measurement, and the hermes-webui quadlet
comment carries the same correction. The seam guarding this invariant accepted
any numeric upstream, so the LAN IP passed it; it now pins the measured-working
form and fails on the LAN IP (teeth verified by substitution).
ngram-mod is draftless (added to the spec config vector unconditionally, while
every model-backed drafter needs ctx_dft), so unlike MTP it can share the
process with DFlash2. Chained at the fleet's 48/64 window, over three shapes:
prose 1k/512 dflash 43.77 | ngram 17.72 (0 drafts) | chain 43.65
echo 1.5k/400 dflash 76.82 | ngram 102.50 | chain 200.58
structured/400 dflash 91.08 | ngram 17.80 (0 drafts) | chain 91.22
identity dflash 70.37 | ngram 17.77 | chain 70.40
On prose the chain is byte-identical to the drafter alone (same 119 drafts/49
accepts) -- that is what makes it a safe default -- and on prompt-echo traffic
it is 2.6x. VRAM is unchanged: the matcher is a 64 MiB host-RAM hash table.
The window sweep says keep the long one: 48/64 beats 16/32 (161.0) and 8/16
(113.8) on echo, and shortening it costs prose (43.9 -> 36.0) because the
matcher then proposes on ordinary fragments and the rejections still have to be
verified. That is the 'ngram hurts' result this fleet hit twice before.
Two documentation traps recorded with it: ngram-mod prints NO startup banner in
this build (its drafts are only visible behaviourally), and the harness's
'structured' shape is a negative control because its lines differ.
Four configurations of the same binary, identical prompts at temperature 0, the
unit stopped between runs, on the deployed payload:
none 17.7 tok/s card1 26.07 GB
draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all
four produced IDENTICAL greedy text, so the 'lossless verification' claim in the
runbook is now a check rather than an assumption. MTP is not the cheaper option
either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper
drafting loses (n=7 accepts a third of its proposals, n=3 more than half).
The two CANNOT be chained, and the failure is structural rather than tunable:
'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested
but model doesn't contain MTP layers', because a chain has ONE draft context and
MTP builds against whatever model that context holds -- here the DFlash2 file.
The serve script therefore passes --spec-draft-model in the dflash branch ONLY,
and the seam now asserts exactly that (the wiring whose violation is this
error), plus that the deployment stays on the measured winner.
scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking
policy wants the runtime's own harness for this, and the numbers above were
otherwise unreproducible one-offs.
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.
Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.
The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.
omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
The 2026-09-16 halogen restore failed exactly here. The ik root unit stopped,
systemd logged "Deactivated successfully", pasta tried to bind 7770 about a
second later, reported "Address already in use", and the quadlet exited 126.
Only Restart=on-failure recovered it -- minutes later, which is not a deploy.
The handler now waits for every socket on the port to clear (listener, accepted
and TIME_WAIT) before installing and starting the quadlet, and fails loudly with
the socket table attached if it never does. The retirement also reports when the
unit file was already gone, because this host's `disable` deletes it.
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.
Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.
Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.
The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
Two gaps behind the same near-miss.
1. **No spec covered hermes.franksdank.cloud** — the one guarded vhost whose
upstream is a host address (`reverse_proxy 192.168.0.143:8787`) rather than a
container name, because its service is host-networked. That is how a
loopback-only bind could take the vhost down with a 502 and leave every gate
green. tests/e2e/specs/17-hermes-vhost.spec.ts asserts what a dead upstream
violates: no 5xx through the guard chain, no Caddy error page, and no app
bytes to an unauthenticated raw client. It does not assert SSO-silence, since
this app has its own password login.
2. **Specs are matched by number prefix, not globbed.** A spec file whose prefix
is in no project silently never runs — which is exactly what happened to the
new one until it was registered. The prefix is registered now, the comment in
playwright.config.ts says why, and tests/e2e_scaffold.bats fails if any
spec file matches no project.
The hermes-webui seam assertion is corrected too: it asserted the loopback bind
I wrongly imposed, and now asserts the real contract — the unit's bind must serve
the address the Caddyfile fronts it on.
Two things, found while checking whether the hermes trio still needed migrating.
1. **The phone-facing webui was exposed to the LAN.** Its unit runs with
Network=host, so there is no Podman port forwarder and the app's own bind is
the exposure. The compose service it replaced published
`127.0.0.1:8787:8787` (loopback); the unit bound `HERMES_WEBUI_HOST=0.0.0.0`,
and the LAN address answered 200 — password-gated, but on every interface,
which is not the documented "SSH tunnel is the sole ingress" posture. It now
binds 127.0.0.1, which serves the tunnel just as well (sshd dials
localhost:8787 from the host). The unit comment claiming "LAN-facing as
before" was wrong about "before", and is corrected.
2. **Dankclaw's compose.yaml is retired.** Auditing ownership showed every
running container on that host reports a PODMAN_SYSTEMD_UNIT label — the
hermes trio's units have owned their containers since 2026-09-12 — so the
three service blocks left in compose.yaml were dead weight that could only
create a duplicate container fighting a unit for a name. deploy_hermes was
already fully unit-based. The file is staged in .AGENTTRASH/; the planner
plans nothing for it, an unknown service on that target now errors instead of
silently no-oping through `podman-compose`, and DANKCLAW_COMPOSE_SERVICES is
empty.
The hermes-webui seam now reads the unit — it read compose, which is exactly how
the bind drifted while the test kept passing against a file nobody ran.
ComfyUI had failed 9,146 times and had been down ~7 hours when this was found:
its container was created 2026-09-15 14:12, BEFORE the GPU declarations were
added to the compose service, and `restart: unless-stopped` kept restarting
that stale container — no /dev/kfd, no /dev/dri, no keep-groups — so it died
every time on `RuntimeError: No HIP GPUs are available`. Its "unit" was a
hand-written ~/.config/systemd/user/comfyui.service that shelled out to
`podman compose up -d`, which is not repo-tracked state and collides by name
with the Quadlet the generator produces.
- dankclaw/comfyui/quadlet/comfyui.container: image comfyui-rocm:gfx1151
(built from Containerfile.comfyui), /dev/kfd + /dev/dri, keep-groups,
the four comfyui-data volumes, loopback-only publish (an unguarded service
is internal-only here; the homepage card probes 127.0.0.1:8188), Restart=
always, WantedBy=default.target.
- the compose service is deleted — one owner per container name.
- deploy_comfyui builds the image, installs the unit, retires the compose-era
unit into ~/.AGENTTRASH, removes the stale container so the unit's own can
take the name, and waits on :8188.
- the healthcheck is a file, because the podman 5.x v1 generator splits
HealthCmd on spaces and an inline `python3 -c "..."` becomes a quoting
puzzle that fails silently.
The comfyui seam now reads the unit instead of compose for devices, command and
groups (and resolves the Containerfile from the deployer's declaration, not
from a directory listing).
Dankserver's node-exporter has been dead since a reboot: `podman ps -a` showed
it `Exited (143)` and Prometheus' `node-dankserver` job has been scraping
nothing. It ran only from `dankserver/compose.yaml`, which is the failure mode
this repository's boot-persistence rule exists for (rootless Podman does not
restart bare containers after the stop that shutdown performs), and the compose
form bound the wildcard address — the unauthenticated-metrics-API-on-every-
interface posture already removed on Dankclaw.
- dankserver/observability/quadlet/node-exporter.container — mirrors the
Dankclaw unit, listening on exactly 127.0.0.1:9100 (host-local probes) and
192.168.0.235:9100 (Prometheus on Dankclaw), with Restart=always and
WantedBy=default.target.
- the compose service is deleted: one owner per container name.
- deploy_observability now installs from "$TARGET/observability/quadlet"; it
hardcoded the Dankclaw path, so a Dankserver unit could not be installed by
the shared dispatcher at all.
- planner: "node-exporter" joins the Dankserver service set, with a path rule
for dankserver/observability/**; the dispatcher's allowlist gains
dankserver:node-exporter.
Seams cover the silent halves: both listen addresses present and the wildcard
absent, the compose service gone, the allowlist entry present, and the planner
mapping the path to the service.
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
This test failed on a clean tree for many sessions, and the working belief
became "test 73 is a known failure, ignore it" — which is exactly what a
permanently red suite teaches. The cause was a selector bug: `find
dankclaw -name 'Containerfile*' | head -1` returned whichever Containerfile the
directory listing happened to put first (dankclaw/alloy/Containerfile), so the
assertion read a file that never claimed to be ComfyUI's. It would equally have
passed on a wrong file.
The test now resolves the Containerfile the way the build does — build.context
joined with build.dockerfile, both read from dankclaw/compose.yaml — so it
checks the file the deploy actually builds (dankclaw/Containerfile.comfyui, the
one compose, deploy-remote.sh and the ComfyUI doc all name) and follows a future
rename instead of guessing.
Suite: ALL TESTS PASSED (310), no exceptions to explain away.
The card probing 192.168.0.235:7770 — frank-medium's endpoint since the
radiance cutover — was still called "frank-small", a service retired on
2026-09-15. The probe target was right; only the identity was wrong, which is
the worse half: a green card named after a service that no longer exists teaches
the wrong thing about the fleet. It is now frank-medium, described as what it
actually is (Qwen3.8-27B Uncensored MXFP4, vllm-radiance, two R9700s behind one
endpoint), and frank-large's description stops crediting the retired halogen
runtime for work ik_llama.cpp does.
The seam added to homepage_config.bats pins the cross-file contract that let
this drift: a card's probe target and its name must agree with the fleet, and a
retired service must not keep a card.
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):
LlmKvPoolSaturated kv pool > 90% for 10m — the 122,636-token wall
LlmRequestsQueueing waiting > 0 for 10m — MAXSEQS=4 reached
LlmPrefixCacheCollapse reuse < 20% for 15m with traffic — the thrash signature
LlmEndpointDown scrape target down 2m — both replicas gone
LlmLatencyHigh p95 end-to-end > 60s for 10m — what an operator feels
job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
dashboard, so the two cannot drift
Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.
Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):
- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
container name, not the loopback publish, with a fixed uid so dashboards
reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
rate, e2e latency p95, and decode/prefill throughput, with an endpoint
variable covering both frank-medium and frank-large. Every series name was
checked against the live /metrics endpoint: this fleet already shipped one
comment naming a metric that does not exist, and a dashboard full of those
would be worse.
Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.
Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.
frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.
The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.
Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.
So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.
The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.
ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).
- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
16-marreta.spec.ts (standard project) asserts the user homepage's Marreta
open link actually reaches the Marreta app through the full guard chain
(Anubis PoW -> Authentik SSO -> app), and explicitly NOT the authentik
login flow. Guards the 2026-09-12 incident where marreta-proxy was created
without a linked Application, so the embedded outpost never loaded it and
the homepage link parked on a login wall.
Wires spec 16 into the standard project testMatch. Verified passing live.
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.
The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Wget-style (no-JS) clients must never receive application bytes from guarded
services: copyparty serves authenticated traffic with anonymous read perms, so
the guard chain is the only boundary between an unauthenticated client and the
file bytes. Assertions target leakage markers in app code (Copyparty UI title,
homepage panel grid, services.json JSON shape), not gate internals, so any
guard-route regression fails the suite while any gate implementation passes.
Also route 15-*.spec.ts into the guest project (empty storage state).
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
Live run against the fleet showed the outpost redirects direct navigation to
auth.franksdank.cloud with the guarded host preserved as redirect_uri, rather
than rendering the flow UI in place. The security property under test is that
an unauthenticated visitor never sees app content; assert that shape (uidField
visible + no leak markers + gate is either in-place or a redirect carrying the
return path) instead of hostname preservation.
14-guard.spec.ts: every authentik-guarded vhost (www, search, romm, files,
seerr, grafana, prometheus, loki) must challenge an unauthenticated direct
navigation; plex is asserted exempt (PMS answers directly, no IdP form).
11-admin.spec.ts: admin direct-nav to grafana/prometheus/loki passes through
with host preserved and no uidField challenge.
Frank