The note described DSpark as the deployed drafter and 40:25 as the split; both
were true for a few hours. It now leads with DFlash2 at 36:29 and keeps the
excursion as the record it is:
- the drafter comparison table with DSpark's REAL size (1,455,376,576, not the
1,359,284,737 that went into the fetch script), and why it lost: no sliding
window, so its draft KV spanned the whole context.
- the split's two moves as one story: the batch pushed it to 40:25, the drafter
swap pulled it back to 36:29, with both measurement tables.
- the layer-size law (0.43 GiB DeltaNet, 1.43 GiB attention) as the reason the
balance is measured and not interpolated.
- the verified-behaviour section now carries the LIVE numbers: 4.71 / 4.91 GiB
free (4.43 / 4.61 under load, 0.18 GiB gap), the DFlash2 init lines, and
draft acceptance from both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0).
- the fault note scopes its own evidence: 39:26 was loaded, not exercised.
DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.
The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:
39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48
38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05
36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.
Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
--verify-only fail on a payload that was in fact correct. Repointed at z-lab
DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
point; it now passes 36:29 and fails every other split measured, including
1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
are gone; the build's defaults apply.
The note still described DFlash2, one slot at 262144 and a 1:1 split -- all of
which the day's work replaced. It now carries:
- the DSpark swap with the table that matters (1.27 vs 1.92 GiB, block_size 7 vs
8 so n_max 6 vs 7, a Markov head instead of the candidate selector, and NO
sliding window) and the rule behind the sizing: the draft context is built
with n_ctx = llama_n_ctx(ctx_tgt), so a drafter without a window caches the
whole target context -- 5.0 GiB here, against DFlash2's ~40 MiB.
- the failure that rule produces when ignored: load succeeds, /health answers,
and the first draft decode dies with an OOM at 1:1.
- why the split moved a second time: tripling the batch costs ROCm1 about twice
what it costs ROCm0, with the 38,27 -> 41,24 -> 40,25 measurement table.
- the deployed flags, the 2-slot shape (n_slots = 2, n_ctx_slot = 131072), the
DSpark init lines, and balanced 29.98 / 29.98 GiB cards.
- honest scoping on the older measurements: the throughput, drafter-worth and
ngram tables were taken on DFlash2, and the agent-loop numbers on one slot.
- the 39:26 page-fault signature as a thing to watch, marked as NOT an OOM.
The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card
floor, and the context maximised inside it.
Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric:
ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid
~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78.
One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement:
41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free.
That balance is also what maximises the context -- an unbalanced split would have
to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's
trained window (262144), now split over two slots of 131072. Verified under two
concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2,
n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots
(acceptance 0.38/0.22 and 0.52/0.43).
8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to
allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in
the env's sizing table.
The unit seam now models both cards (share + draft + ubatch-scaled buffers,
calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor
on each, and asserts the two cards stay within 0.5 GiB of each other. Both
assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and
a 1:1 split) rather than pass by construction.
Correcting a claim I made from a bad query: Q4_K_S does exist for this finetune.
`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` ships the large quants as
`.partNofM` fragments — Q4_K_M 37.00+37.00+36.97 = 110.97 GiB (which is how the
deployed file was assembled) and Q4_K_S 35.00+35.00+34.09 = 104.09 GiB — while
the smaller rungs are single files. A tree listing filtered on `.gguf` makes
Q4_K_S invisible, which is exactly how I concluded it did not exist.
Also records the fetch trap that hid it for a moment: `curl -o FILE URL` exits 0
on 401/403, so a gated repo produced three 153-byte "shards" with identical
sha256 values and a cheerful DOWNLOAD_DONE in 20 s for 104 GiB. -f plus the
Hub's declared size plus the LFS sha256 is the standard the fleet's own
download-frank-medium-gguf.sh already meets.
The 38:27 skew folded into the DSpark swap already does what step 3 asks: the
draft-carrying card sits 2.08 GiB above the other (5.49 / 3.41 GiB free) instead
of the ~9 GiB a 1:1 split would leave.
Chasing an exact balance is a mistake worth recording: 39:26 measured tighter
(3.86 / 4.69 GiB free) but that server died on its eighth request with SDMA0
UTCL2 page faults in the amdgpu driver -- "ROCm error: an illegal memory access
was encountered" -> ggml_abort, in the cross-card boundary copy that is the
layer split's hot path. 38:27 has since served the whole e2e gate and 12 varied
requests with no fault, so the residual 2.08 GiB gap is the safe side of this
trade. Comment-only: no restart needed.
Step 2 of the operator's six. --ctx-checkpoints 8 held per-slot KV snapshots;
0 is what llama.cpp ships, and the fleet's clients open conversations fresh.
DSpark declares no sliding window, so its draft context -- built with
n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8
kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where
DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at
33.9 GB of 34.2 GB and the first draft graph capture died inside
common_speculative_impl_draft_dflash::process with "ROCm error: out of memory"
-- after the model had loaded and /health had answered, which is why the deploy
looked like a crash-loop rather than a sizing failure.
Handing three target layers to ROCm0 restores headroom on both cards at the full
window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free
(was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam
now models the busier card as a skewed share plus the drafter's full-context KV,
calibrated to the rocm-smi figure.
AGENTS.md gains the durable lesson: read sliding_window out of the drafter's
GGUF header before sizing a spec-decode deployment.
The log mtimes plus each run's own duration pin it: the first 'invalid internal
status' in the user journal is 18:58:30, while the F bench (launched 18:54:31)
was still loading, and the preceding llama-cli placement probe had just spilled
3.2 GB of '> ' prompts. The note had said 'after the experiment'; that was wrong
and is now recorded with the timestamp it can be checked against.
I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the
pause-process death. journalctl -k records no OOM kill for that window and the
kernel says nothing else about it, so the claim was a guess wearing a finding's
clothes — in AGENTS.md, the x4 note, and the log. All three now state the
trigger is not established, and the note's 'co-tenants' patience' bullet no
longer asserts that causality either. AGENTS.md also gains the real lesson:
journalctl -k is the check, and an unknown cause does not need a story.
DSpark (Q8_0, 1.27 GiB) replaces z-lab DFlash2 (1.92 GiB): same 5-layer
block-diffusion shape, Markov head instead of the candidate selector, and
block_size 7, so the draft depth drops 7 -> 6 (llama.cpp asserts
n_max <= block_size - 1).
The serve script's dspark branch never passed --spec-draft-model (only the
dflash branch did), so dspark was not deployable as written; the model file
gate now covers both types. Download script, env and the unit seam track the
new file and its exact byte count.
The ledger number in the note was pre-reboot. Same model, same pool, no clamp —
but the host was measurably more fragmented when it came back, which is the sort
of thing that decides whether the next co-tenant fits.
Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.
Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:
pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
iGPU alone 25.77 tok/s @64k, pp512 231.8
+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.
Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
hermes-webui is Network=host, so 8787 is a bare host-netns bind, and netavark
DNATs only *published* ports. From Caddy's bridge netns 192.168.0.143:8787 and
10.89.1.1:8787 are both refused, while host.containers.internal:8787
(169.254.1.2) answers 200 — measured live 2026-09-16 from inside the Caddy
container. The dashboard's published 9119 answers on that same LAN IP, which is
what made the old upstream look sound.
The vhost comment asserted "192.168.0.143:8787 is open" from that bridge; it is
not, so the claim is replaced with the measurement, and the hermes-webui quadlet
comment carries the same correction. The seam guarding this invariant accepted
any numeric upstream, so the LAN IP passed it; it now pins the measured-working
form and fails on the LAN IP (teeth verified by substitution).
The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not
expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in
the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer
(Vulkan1) that cannot run the operation (NONE)" -- and both per-layer
cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU
is the only fallback side that works, which is exactly why -ot exps=CPU runs.
What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in
page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s
prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s --
but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7
spare, so it is not deployable beside the co-tenants. Throughput parity, memory
bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than
preference.
New page LLM/x4-pair-experiment.md carries the tensor inventory, the results
table, the two scheduler limits, reproduction commands, and the traps found on
the way. Both index pages also stop calling halogen retired -- the ik swap's
stale claims were still sitting in them.
The fleet now runs draft-dflash,ngram-mod, so the rule needs to be written down
where the next agent will look: draftless types (ngram family) are added to the
spec configuration unconditionally, while draft-mtp/dflash/dspark/eagle3 all
require a draft CONTEXT -- therefore one context-backed drafter plus any number
of ngram types is legal, and two context-backed ones never are ('draft-mtp,draft-dflash'
dies at load with 'model doesn't contain MTP layers'). Plus the operational
trap: ngram-mod prints no startup banner, so a working chain looks unengaged in
the journal.
The port gate was written for the cutover, when the vLLM proxy held 7770. Once
frank-medium itself owns the port, a re-deploy waits 90s for a socket that only
its own predecessor is holding and fails: 'FAIL: port 7770 is still held after
90s', with the old llama-server still serving scrapes. Stop the unit first, then
sweep a stray manual llama-server (the shape a benchmark harness leaves behind);
pkill -x, never -f, so the deploying shell cannot match itself.
ngram-mod is draftless (added to the spec config vector unconditionally, while
every model-backed drafter needs ctx_dft), so unlike MTP it can share the
process with DFlash2. Chained at the fleet's 48/64 window, over three shapes:
prose 1k/512 dflash 43.77 | ngram 17.72 (0 drafts) | chain 43.65
echo 1.5k/400 dflash 76.82 | ngram 102.50 | chain 200.58
structured/400 dflash 91.08 | ngram 17.80 (0 drafts) | chain 91.22
identity dflash 70.37 | ngram 17.77 | chain 70.40
On prose the chain is byte-identical to the drafter alone (same 119 drafts/49
accepts) -- that is what makes it a safe default -- and on prompt-echo traffic
it is 2.6x. VRAM is unchanged: the matcher is a 64 MiB host-RAM hash table.
The window sweep says keep the long one: 48/64 beats 16/32 (161.0) and 8/16
(113.8) on echo, and shortening it costs prose (43.9 -> 36.0) because the
matcher then proposes on ordinary fragments and the rejections still have to be
verified. That is the 'ngram hurts' result this fleet hit twice before.
Two documentation traps recorded with it: ngram-mod prints NO startup banner in
this build (its drafts are only visible behaviourally), and the harness's
'structured' shape is a negative control because its lines differ.
Two real defects from patching a rewritten file by line number: the CUT that
removed a duplicate parse_configs also took write_env with it (NameError after
the first configuration, with the unit already stopped), and the restore step
lived after the loop, so a crash left frank-medium DOWN. The endpoint is the
reason the harness exists; it now restores itself in a finally and shouts if the
service does not come back healthy.
ngram-mod's match window (n_min/n_max) decides which traffic it can see at all
-- the fleet's 48/64 preset is tuned for long verbatim echoes and finds nothing
in a repetitive-but-short output. Config entries now take |KEY=VALUE overrides,
so the window can be swept in the same run as the configurations it applies to.
The first run asked for a verbatim repeat with thinking ON, so the model copied
the passage inside reasoning_content, content came back empty for all three
configurations, and the cross-configuration identity check compared '' to '' --
a pass that proves nothing. echo and structured now disable thinking via
chat_template_kwargs, the echo row carries a copy_ratio (share of answer lines
that really are passage lines), and an empty answer in any shape is a loud
warning rather than a silent pass.
'structured' is new and answers the case the fleet actually runs on: repetitive
OUTPUT with no prompt overlap (record lists, tables, config rewrites), which is
where an n-gram matcher should pay and an echo-shaped test cannot tell.
A relative serve-script path plus a cd into its own directory double-counts the
prefix, so every configuration failed to start ('scripts/serve-frank-medium.sh:
No such file or directory'). The safety net did its job -- the unit was restored
-- but the run measured nothing.
Unlike MTP, ngram-mod needs no draft context: common/speculative.cpp adds the
ngram types unconditionally while the model-backed drafters are gated on
ctx_dft != nullptr, so 'draft-dflash,ngram-mod' is structurally allowed (and the
config vector's fixed order -- ngram first, then the drafters -- makes CLI order
irrelevant).
The harness grows the two things this question needs: --configs, and an ECHO
shape. ngram-mod can only ever match text that is present in the prompt, so
measuring it on prose calls it useless and measuring DFlash on echo calls it
wasteful; both shapes now run, and the greedy text of each is compared across
configurations because speculation may not change answers.
Four configurations of the same binary, identical prompts at temperature 0, the
unit stopped between runs, on the deployed payload:
none 17.7 tok/s card1 26.07 GB
draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all
four produced IDENTICAL greedy text, so the 'lossless verification' claim in the
runbook is now a check rather than an assumption. MTP is not the cheaper option
either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper
drafting loses (n=7 accepts a third of its proposals, n=3 more than half).
The two CANNOT be chained, and the failure is structural rather than tunable:
'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested
but model doesn't contain MTP layers', because a chain has ONE draft context and
MTP builds against whatever model that context holds -- here the DFlash2 file.
The serve script therefore passes --spec-draft-model in the dflash branch ONLY,
and the seam now asserts exactly that (the wiring whose violation is this
error), plus that the deployment stays on the measured winner.
scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking
policy wants the runtime's own harness for this, and the numbers above were
otherwise unreproducible one-offs.
The loader prints no per-device ledger at this verbosity, so the claim is
measured rather than read: +131072 tokens of context added 4.83 GB to one card
and 4.84 GB to the other (both carry their own attention layers' KV), and
neither card's total usage could hold the 31.46 GB file. Worth writing down
because the retired shape WAS two full copies — that is what a replica is, and
it is why the 29.8 GB FP8 checkpoint could never run there.
The LLM index still opened with vllm-radiance as frank-medium's runtime and
drew the replica pair in its topology diagram; the cooperation page's thesis
(collectives are impossible, so the answer is hardware) now has to name the
path the fleet actually took: llama.cpp's layer split, which launches no RCCL
kernel on either card. That reframes the M.2 riser as a means to TENSOR
parallelism rather than a means to using both cards at all.
Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.
The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.
Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.
The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.
omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.
The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
'cp -r SRC DST/' with a missing DST creates DST as a copy of SRC and drops a
directory level, so rocblas/library/* landed as lib/rocblas/* and hipblaslt's
arch subdirectories landed one level high. Nothing complains at load time: the
engine reads 29 GiB of weights, answers /health, and dies on the first request
with
rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat:
No such file or directory for GPU arch : gfx1201
Declare the destination and copy contents, both trees.
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.
AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.
Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
The runtime for the next frank-medium shape is mainline llama.cpp at
fb27a525, built for gfx1201 against the ROCm 7.14 nightly the ROCmFPX build
already used on this host, with the runtime bundled next to the binary.
Two things that pin the ref: DFlash2 (ggml-org/llama.cpp#27342) and qwen35
MTP. Neither is in the ROCmFPX fork's tip (c49ebdb, 2026-08-22), and the
DFlash2 drafter cannot run without the first.
The fetch script pulls the target at UD-Q8_K_XL plus the DFlash2 draft at
Q8_0, resumable and byte-verified -- a truncated 31 GB GGUF is otherwise a
load-time crash.
A redeploy of a running frank-large holds 7770 itself, so waiting on the port
before stopping the unit would have made every subsequent deploy fail. Stop the
user unit first (a no-op on a first cutover), then wait, then install and start.
The 2026-09-16 halogen restore failed exactly here. The ik root unit stopped,
systemd logged "Deactivated successfully", pasta tried to bind 7770 about a
second later, reported "Address already in use", and the quadlet exited 126.
Only Restart=on-failure recovered it -- minutes later, which is not a deploy.
The handler now waits for every socket on the port to clear (listener, accepted
and TIME_WAIT) before installing and starting the quadlet, and fails loudly with
the socket table attached if it never does. The retirement also reports when the
unit file was already gone, because this host's `disable` deletes it.
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.
Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.
Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.
The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with
the measured evidence rather than the theory.
**The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise
`AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall
buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix
it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per
ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1
mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16
bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈
1.4 ms at Gen4 x4.
**New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane
card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all
three, and `send`/`recv` fails too — which kills pipeline parallelism as an
escape, since vLLM's PP handoff is exactly that primitive.
**Correction.** I previously said no software path exists. That overshot. One is
open and unverified: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's
device source contains no live printf/assert/__assertfail, so the trigger is
unexplained, and the claim traces to search summaries, not a primary source. The
page carries the two-command test that would settle it. Also corrected: 0.9.3 is
radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm
7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory
path cannot run.
**Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just
benchmarks — including a bare 2-rank probe. Running one against the live service
on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle
required); a hung AQL dispatch wedging the driver is the plausible mechanism.
The policy also now says how to tell off from wedged: `ip neigh` from a peer.
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
The board showed mariadb, valkey and Postgres as "down". None of them is down:
the checker runs on Dankclaw and can reach that host's loopback, its container
names, and published LAN addresses — while all three cards probe
container-internal database ports (192.168.0.235:3306, 127.0.0.1:6379/:5432).
They were red by construction, which is how a status board stops being read.
Publishing a database port on the LAN to make a badge green is not a trade this
fleet makes, so they are wip with the reason in their description, and the apps
that front them carry the status. Gluetun (WireGuard, UDP) was the pre-existing
example of the same call.
The rule is now written down in the Homepage doc: a card must be probeable from
the monitor's host or be wip. That is the same lesson as the Prometheus jobs
that scrape HTML pages and can never report up — an always-red signal teaches
the operator to ignore the surface it lives on.
Two gaps behind the same near-miss.
1. **No spec covered hermes.franksdank.cloud** — the one guarded vhost whose
upstream is a host address (`reverse_proxy 192.168.0.143:8787`) rather than a
container name, because its service is host-networked. That is how a
loopback-only bind could take the vhost down with a 502 and leave every gate
green. tests/e2e/specs/17-hermes-vhost.spec.ts asserts what a dead upstream
violates: no 5xx through the guard chain, no Caddy error page, and no app
bytes to an unauthenticated raw client. It does not assert SSO-silence, since
this app has its own password login.
2. **Specs are matched by number prefix, not globbed.** A spec file whose prefix
is in no project silently never runs — which is exactly what happened to the
new one until it was registered. The prefix is registered now, the comment in
playwright.config.ts says why, and tests/e2e_scaffold.bats fails if any
spec file matches no project.
The hermes-webui seam assertion is corrected too: it asserted the loopback bind
I wrongly imposed, and now asserts the real contract — the unit's bind must serve
the address the Caddyfile fronts it on.
I tightened HERMES_WEBUI_HOST to 127.0.0.1 on the reasoning that the compose
service had published loopback and the unit had widened it. That reasoning
missed the consumer: this unit is host-networked (it shares the CDP daemon's
loopback), so Caddy cannot reach it by container name and fronts the guarded
vhost hermes.franksdank.cloud with `reverse_proxy 192.168.0.143:8787` — recorded
in dankclaw/Caddyfile with the same explanation. With the app on loopback that
upstream is dead, so the vhost broke.
The wildcard bind is what serves BOTH consumers: Caddy on the LAN address and
the phone's SSH tunnel on the host's localhost. Controls are the auth chain in
front of the vhost plus the webui password. The unit comment now states this
instead of the incorrect 'as before' claim, and the near-miss is worth the
paragraph: the e2e gate stayed green because no spec covers that vhost.
Two things, found while checking whether the hermes trio still needed migrating.
1. **The phone-facing webui was exposed to the LAN.** Its unit runs with
Network=host, so there is no Podman port forwarder and the app's own bind is
the exposure. The compose service it replaced published
`127.0.0.1:8787:8787` (loopback); the unit bound `HERMES_WEBUI_HOST=0.0.0.0`,
and the LAN address answered 200 — password-gated, but on every interface,
which is not the documented "SSH tunnel is the sole ingress" posture. It now
binds 127.0.0.1, which serves the tunnel just as well (sshd dials
localhost:8787 from the host). The unit comment claiming "LAN-facing as
before" was wrong about "before", and is corrected.
2. **Dankclaw's compose.yaml is retired.** Auditing ownership showed every
running container on that host reports a PODMAN_SYSTEMD_UNIT label — the
hermes trio's units have owned their containers since 2026-09-12 — so the
three service blocks left in compose.yaml were dead weight that could only
create a duplicate container fighting a unit for a name. deploy_hermes was
already fully unit-based. The file is staged in .AGENTTRASH/; the planner
plans nothing for it, an unknown service on that target now errors instead of
silently no-oping through `podman-compose`, and DANKCLAW_COMPOSE_SERVICES is
empty.
The hermes-webui seam now reads the unit — it read compose, which is exactly how
the bind drifted while the test kept passing against a file nobody ran.
ComfyUI's page described a compose service and a systemd unit that shelled out
to `podman compose up`. It now documents the Quadlet, the keep-groups + udev
mechanism (verified inside the running container: torch sees Radeon 8060S), the
9,146-restart incident that forced the move, and the trap that caused it — a
stale container outlives the compose edit that should have replaced it.
The observability page gains the audit result: four jobs read "down" and three
of them always will, because they scrape HTML pages or a /metrics that does not
exist. The page names each error, says plainly that none is an outage, and
refuses the tempting fix (deleting the job makes an absent scrape look like a
healthy one) in favour of the remedy it already recommended: a blackbox exporter
in probe mode, and Gitea's metrics enabled or the job repointed.
ComfyUI had failed 9,146 times and had been down ~7 hours when this was found:
its container was created 2026-09-15 14:12, BEFORE the GPU declarations were
added to the compose service, and `restart: unless-stopped` kept restarting
that stale container — no /dev/kfd, no /dev/dri, no keep-groups — so it died
every time on `RuntimeError: No HIP GPUs are available`. Its "unit" was a
hand-written ~/.config/systemd/user/comfyui.service that shelled out to
`podman compose up -d`, which is not repo-tracked state and collides by name
with the Quadlet the generator produces.
- dankclaw/comfyui/quadlet/comfyui.container: image comfyui-rocm:gfx1151
(built from Containerfile.comfyui), /dev/kfd + /dev/dri, keep-groups,
the four comfyui-data volumes, loopback-only publish (an unguarded service
is internal-only here; the homepage card probes 127.0.0.1:8188), Restart=
always, WantedBy=default.target.
- the compose service is deleted — one owner per container name.
- deploy_comfyui builds the image, installs the unit, retires the compose-era
unit into ~/.AGENTTRASH, removes the stale container so the unit's own can
take the name, and waits on :8188.
- the healthcheck is a file, because the podman 5.x v1 generator splits
HealthCmd on spaces and an inline `python3 -c "..."` becomes a quoting
puzzle that fails silently.
The comfyui seam now reads the unit instead of compose for devices, command and
groups (and resolves the Containerfile from the deployer's declaration, not
from a directory listing).
The page said "all five are rootless Quadlet units on Dankclaw", which stopped
being true when Dankserver's node-exporter was restored to a unit. It now names
both hosts and both deployments, records why the second one happened (the same
compose-only reboot argument, a day and a half of missing host metrics, and a
wildcard bind), and states the exposure rule for a two-listener fleet with the
`ss` evidence that the wildcard is gone.
Dankserver's node-exporter has been dead since a reboot: `podman ps -a` showed
it `Exited (143)` and Prometheus' `node-dankserver` job has been scraping
nothing. It ran only from `dankserver/compose.yaml`, which is the failure mode
this repository's boot-persistence rule exists for (rootless Podman does not
restart bare containers after the stop that shutdown performs), and the compose
form bound the wildcard address — the unauthenticated-metrics-API-on-every-
interface posture already removed on Dankclaw.
- dankserver/observability/quadlet/node-exporter.container — mirrors the
Dankclaw unit, listening on exactly 127.0.0.1:9100 (host-local probes) and
192.168.0.235:9100 (Prometheus on Dankclaw), with Restart=always and
WantedBy=default.target.
- the compose service is deleted: one owner per container name.
- deploy_observability now installs from "$TARGET/observability/quadlet"; it
hardcoded the Dankclaw path, so a Dankserver unit could not be installed by
the shared dispatcher at all.
- planner: "node-exporter" joins the Dankserver service set, with a path rule
for dankserver/observability/**; the dispatcher's allowlist gains
dankserver:node-exporter.
Seams cover the silent halves: both listen addresses present and the wildcard
absent, the compose service gone, the allowlist entry present, and the planner
mapping the path to the service.
Frank