The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell
grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau:
ubatch 256 384 512 768 1024 1280 1536 1792 2048
tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078
The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out
1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the
shape, not noise. The logical batch is throughput-neutral (2048..8192 all
within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized
by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell.
So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB
more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2):
36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed)
36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone)
35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed
The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1
holds the output layers and the drafter's graphs -- so it shifted the balance
toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is
one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split
can be tuned finer than a layer, which is why 34,31 is not better.
Re-fitted the seam with both ubatch coefficients measured at this shape and
re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to
0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 ->
0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB.
Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the
shape it actually cares about -- two shares that add up to the model's 65
blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer
breaks a test about parallelism. The numeric contract lives in the seam test.
pcie-link-test.cu (CUDA, pinned buffers, cudaEvent timing, both services stopped):
pinned H2D 6.72 GB/s and D2H 6.60 GB/s, i.e. 85 % of the Gen4 x4 ceiling. Async
is no better than sync and pageable sits within 1 % of pinned, so these copies are
link-limited rather than staging-limited — the usual "pageable costs 2x" does not
apply on this host. A 3 MiB slab costs 474 us, essentially 3 MiB / 6.72 GB/s plus
~27 us of launch.
That settles the expert-streaming question quantitatively: 1.70 GiB of active
experts per token (A3B @ Q4_K_M, 75 GiB expert mass) is 272 ms of link time ⇒
3.7 tok/s, 4.6 with a fifth of the experts resident. The same bytes computed in
host RAM at 256 GB/s cost 7.1 ms — the link is 31x slower than the memory behind
it, which is why llama.cpp's -ot exps=CPU path measured 20.46 tok/s while an
engine that streams loses by 3-5x. The port under the card is x8-capable and the
card is x16-capable, so the x4 is the adapter; doubling it still leaves streaming
well behind.
Also records FreeToken's own constraint: it keeps experts in host RAM with an LRU
GPU cache (exactly this streaming shape) and loads only modelopt block-FP8
safetensors for qwen3_5_moe, so Q4_K_M GGUF is unloadable there regardless.
The harness is in the repo now, so the runbook says how to run it and what the
tool will not tell you:
- / splits devices within one configuration and , splits sweep values, so a
legitimate --tensor-split 36,29 would have benched one card at a time.
- there is no -c: n_ctx = n_prompt + n_gen + n_depth, so no bench can decide
whether a configuration's buffers fit beside the deployed 262144 KV.
- a test that cannot allocate exit(1)s the process, hence one invocation per
cell.
- a shell's memlock limit (8 MiB here) is not the unit's (infinity), which
fails the load for the wrong reason.
- no --spec-* and no --parallel: decode figures are drafterless and slots are
never exercised; -d is what puts KV in front of the measurement at all.
Operator instruction: purge the cache so no run can serve a stale payload. The two
live payloads were cleared and then re-fetched rather than left absent — deleting
them without a re-fetch means neither endpoint starts.
halogen's re-fetch proved the point: its sidecars had been stale *in place*
(overlay.hgn 2,477,677,120 -> 2,572,466,560 bytes, overlay-speed.hgn
2,383,306,048 -> 2,478,095,488) while the 115.55 GiB checkpoint returned
byte-identical. frank-small's 27B and mmproj re-fetched to Hub-matching hashes,
unchanged.
Trap recorded: podman refuses a bind mount whose source directory is missing
(statfs ... no such file or directory), so frank-large needs /models to exist
*empty* for its first-start fetch to run.
Q4_K_S against the recorded Q4_K_M numbers: +2.1/+1.6/+2.6 % (iGPU-only pp512 /
pp16384 / tg64), +2.7 % tg32@64k, +4.1 %/+2.0 % for the two layer-split configs,
~0 on both pp512@64k, host RAM bottoming at 10.0 GiB against ~5 GiB. A memory
lever with a small speed dividend. IQ4_XS (97,473,155,360 bytes, sha256
b4787265...d0574f9d) is pulled and verified for the next comparison.
Also repairs an AGENTS.md edit that landed in the wrong section: the peer
session's commits shifted that file between my read and my write, so the
frank-large text briefly split the Model Config section.
`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands every argument of the builtin
before the builtin assigns any of them, so $b was unbound and set -u aborted the run
four seconds in - after the service was already stopped. Two statements instead.
Found by running it under systemd-run: it exited non-zero with no cells run, which is
exactly what the per-cell TSV is for.
Question it answers: within the headroom the deployment has, how large may the logical
batch and the micro-batch be before the activation buffers stop fitting — and what does
each size buy in prefill?
- One llama-bench invocation per cell. llama-bench exit(1)s the moment a test cannot run,
so a single infeasible ubatch would otherwise take every later cell with it, and
infeasibility is one of the things being looked for.
- A VRAM sampler per cell, covering the load. Batch/ubatch sizes the activation buffers,
so a cell that looks fast because its allocation is smaller than intended has to be
distinguishable from one that is genuinely fast.
- The service is stopped for the sweep and started again from a trap, because a bench
beside a live service starves the same GPU.
- Run it from a transient unit: the deploy's unit sets LimitMEMLOCK=infinity for the ROCm
staging buffers and a shell on these hosts only gets 8 MiB.
- llama-bench separates -ts/-dev with SLASHES and has no -c, --parallel or --spec-*: the
context is n_prompt + n_gen + n_depth, and the drafter cannot take part.
Checking whether a download was still running with pgrep -a curl printed the
whole argv, Authorization header included, into the session transcript. The
token is not in the repo and nothing was committed, but the practice is wrong:
secrets belong in a -K config file or the environment, not in argv, and no
pgrep -a should ever target a command carrying an auth header.
The note described DSpark as the deployed drafter and 40:25 as the split; both
were true for a few hours. It now leads with DFlash2 at 36:29 and keeps the
excursion as the record it is:
- the drafter comparison table with DSpark's REAL size (1,455,376,576, not the
1,359,284,737 that went into the fetch script), and why it lost: no sliding
window, so its draft KV spanned the whole context.
- the split's two moves as one story: the batch pushed it to 40:25, the drafter
swap pulled it back to 36:29, with both measurement tables.
- the layer-size law (0.43 GiB DeltaNet, 1.43 GiB attention) as the reason the
balance is measured and not interpolated.
- the verified-behaviour section now carries the LIVE numbers: 4.71 / 4.91 GiB
free (4.43 / 4.61 under load, 0.18 GiB gap), the DFlash2 init lines, and
draft acceptance from both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0).
- the fault note scopes its own evidence: 39:26 was loaded, not exercised.
DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.
The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:
39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48
38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05
36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.
Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
--verify-only fail on a payload that was in fact correct. Repointed at z-lab
DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
point; it now passes 36:29 and fails every other split measured, including
1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
are gone; the build's defaults apply.
The note still described DFlash2, one slot at 262144 and a 1:1 split -- all of
which the day's work replaced. It now carries:
- the DSpark swap with the table that matters (1.27 vs 1.92 GiB, block_size 7 vs
8 so n_max 6 vs 7, a Markov head instead of the candidate selector, and NO
sliding window) and the rule behind the sizing: the draft context is built
with n_ctx = llama_n_ctx(ctx_tgt), so a drafter without a window caches the
whole target context -- 5.0 GiB here, against DFlash2's ~40 MiB.
- the failure that rule produces when ignored: load succeeds, /health answers,
and the first draft decode dies with an OOM at 1:1.
- why the split moved a second time: tripling the batch costs ROCm1 about twice
what it costs ROCm0, with the 38,27 -> 41,24 -> 40,25 measurement table.
- the deployed flags, the 2-slot shape (n_slots = 2, n_ctx_slot = 131072), the
DSpark init lines, and balanced 29.98 / 29.98 GiB cards.
- honest scoping on the older measurements: the throughput, drafter-worth and
ngram tables were taken on DFlash2, and the agent-loop numbers on one slot.
- the 39:26 page-fault signature as a thing to watch, marked as NOT an OOM.
The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card
floor, and the context maximised inside it.
Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric:
ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid
~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78.
One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement:
41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free.
That balance is also what maximises the context -- an unbalanced split would have
to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's
trained window (262144), now split over two slots of 131072. Verified under two
concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2,
n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots
(acceptance 0.38/0.22 and 0.52/0.43).
8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to
allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in
the env's sizing table.
The unit seam now models both cards (share + draft + ubatch-scaled buffers,
calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor
on each, and asserts the two cards stay within 0.5 GiB of each other. Both
assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and
a 1:1 split) rather than pass by construction.
Correcting a claim I made from a bad query: Q4_K_S does exist for this finetune.
`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` ships the large quants as
`.partNofM` fragments — Q4_K_M 37.00+37.00+36.97 = 110.97 GiB (which is how the
deployed file was assembled) and Q4_K_S 35.00+35.00+34.09 = 104.09 GiB — while
the smaller rungs are single files. A tree listing filtered on `.gguf` makes
Q4_K_S invisible, which is exactly how I concluded it did not exist.
Also records the fetch trap that hid it for a moment: `curl -o FILE URL` exits 0
on 401/403, so a gated repo produced three 153-byte "shards" with identical
sha256 values and a cheerful DOWNLOAD_DONE in 20 s for 104 GiB. -f plus the
Hub's declared size plus the LFS sha256 is the standard the fleet's own
download-frank-medium-gguf.sh already meets.
The 38:27 skew folded into the DSpark swap already does what step 3 asks: the
draft-carrying card sits 2.08 GiB above the other (5.49 / 3.41 GiB free) instead
of the ~9 GiB a 1:1 split would leave.
Chasing an exact balance is a mistake worth recording: 39:26 measured tighter
(3.86 / 4.69 GiB free) but that server died on its eighth request with SDMA0
UTCL2 page faults in the amdgpu driver -- "ROCm error: an illegal memory access
was encountered" -> ggml_abort, in the cross-card boundary copy that is the
layer split's hot path. 38:27 has since served the whole e2e gate and 12 varied
requests with no fault, so the residual 2.08 GiB gap is the safe side of this
trade. Comment-only: no restart needed.
Step 2 of the operator's six. --ctx-checkpoints 8 held per-slot KV snapshots;
0 is what llama.cpp ships, and the fleet's clients open conversations fresh.
DSpark declares no sliding window, so its draft context -- built with
n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8
kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where
DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at
33.9 GB of 34.2 GB and the first draft graph capture died inside
common_speculative_impl_draft_dflash::process with "ROCm error: out of memory"
-- after the model had loaded and /health had answered, which is why the deploy
looked like a crash-loop rather than a sizing failure.
Handing three target layers to ROCm0 restores headroom on both cards at the full
window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free
(was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam
now models the busier card as a skewed share plus the drafter's full-context KV,
calibrated to the rocm-smi figure.
AGENTS.md gains the durable lesson: read sliding_window out of the drafter's
GGUF header before sizing a spec-decode deployment.
The log mtimes plus each run's own duration pin it: the first 'invalid internal
status' in the user journal is 18:58:30, while the F bench (launched 18:54:31)
was still loading, and the preceding llama-cli placement probe had just spilled
3.2 GB of '> ' prompts. The note had said 'after the experiment'; that was wrong
and is now recorded with the timestamp it can be checked against.
I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the
pause-process death. journalctl -k records no OOM kill for that window and the
kernel says nothing else about it, so the claim was a guess wearing a finding's
clothes — in AGENTS.md, the x4 note, and the log. All three now state the
trigger is not established, and the note's 'co-tenants' patience' bullet no
longer asserts that causality either. AGENTS.md also gains the real lesson:
journalctl -k is the check, and an unknown cause does not need a story.
DSpark (Q8_0, 1.27 GiB) replaces z-lab DFlash2 (1.92 GiB): same 5-layer
block-diffusion shape, Markov head instead of the candidate selector, and
block_size 7, so the draft depth drops 7 -> 6 (llama.cpp asserts
n_max <= block_size - 1).
The serve script's dspark branch never passed --spec-draft-model (only the
dflash branch did), so dspark was not deployable as written; the model file
gate now covers both types. Download script, env and the unit seam track the
new file and its exact byte count.
The ledger number in the note was pre-reboot. Same model, same pool, no clamp —
but the host was measurably more fragmented when it came back, which is the sort
of thing that decides whether the next co-tenant fits.
Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.
Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:
pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
iGPU alone 25.77 tok/s @64k, pp512 231.8
+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.
Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
hermes-webui is Network=host, so 8787 is a bare host-netns bind, and netavark
DNATs only *published* ports. From Caddy's bridge netns 192.168.0.143:8787 and
10.89.1.1:8787 are both refused, while host.containers.internal:8787
(169.254.1.2) answers 200 — measured live 2026-09-16 from inside the Caddy
container. The dashboard's published 9119 answers on that same LAN IP, which is
what made the old upstream look sound.
The vhost comment asserted "192.168.0.143:8787 is open" from that bridge; it is
not, so the claim is replaced with the measurement, and the hermes-webui quadlet
comment carries the same correction. The seam guarding this invariant accepted
any numeric upstream, so the LAN IP passed it; it now pins the measured-working
form and fails on the LAN IP (teeth verified by substitution).
The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not
expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in
the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer
(Vulkan1) that cannot run the operation (NONE)" -- and both per-layer
cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU
is the only fallback side that works, which is exactly why -ot exps=CPU runs.
What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in
page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s
prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s --
but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7
spare, so it is not deployable beside the co-tenants. Throughput parity, memory
bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than
preference.
New page LLM/x4-pair-experiment.md carries the tensor inventory, the results
table, the two scheduler limits, reproduction commands, and the traps found on
the way. Both index pages also stop calling halogen retired -- the ik swap's
stale claims were still sitting in them.
The fleet now runs draft-dflash,ngram-mod, so the rule needs to be written down
where the next agent will look: draftless types (ngram family) are added to the
spec configuration unconditionally, while draft-mtp/dflash/dspark/eagle3 all
require a draft CONTEXT -- therefore one context-backed drafter plus any number
of ngram types is legal, and two context-backed ones never are ('draft-mtp,draft-dflash'
dies at load with 'model doesn't contain MTP layers'). Plus the operational
trap: ngram-mod prints no startup banner, so a working chain looks unengaged in
the journal.
The port gate was written for the cutover, when the vLLM proxy held 7770. Once
frank-medium itself owns the port, a re-deploy waits 90s for a socket that only
its own predecessor is holding and fails: 'FAIL: port 7770 is still held after
90s', with the old llama-server still serving scrapes. Stop the unit first, then
sweep a stray manual llama-server (the shape a benchmark harness leaves behind);
pkill -x, never -f, so the deploying shell cannot match itself.
ngram-mod is draftless (added to the spec config vector unconditionally, while
every model-backed drafter needs ctx_dft), so unlike MTP it can share the
process with DFlash2. Chained at the fleet's 48/64 window, over three shapes:
prose 1k/512 dflash 43.77 | ngram 17.72 (0 drafts) | chain 43.65
echo 1.5k/400 dflash 76.82 | ngram 102.50 | chain 200.58
structured/400 dflash 91.08 | ngram 17.80 (0 drafts) | chain 91.22
identity dflash 70.37 | ngram 17.77 | chain 70.40
On prose the chain is byte-identical to the drafter alone (same 119 drafts/49
accepts) -- that is what makes it a safe default -- and on prompt-echo traffic
it is 2.6x. VRAM is unchanged: the matcher is a 64 MiB host-RAM hash table.
The window sweep says keep the long one: 48/64 beats 16/32 (161.0) and 8/16
(113.8) on echo, and shortening it costs prose (43.9 -> 36.0) because the
matcher then proposes on ordinary fragments and the rejections still have to be
verified. That is the 'ngram hurts' result this fleet hit twice before.
Two documentation traps recorded with it: ngram-mod prints NO startup banner in
this build (its drafts are only visible behaviourally), and the harness's
'structured' shape is a negative control because its lines differ.
Two real defects from patching a rewritten file by line number: the CUT that
removed a duplicate parse_configs also took write_env with it (NameError after
the first configuration, with the unit already stopped), and the restore step
lived after the loop, so a crash left frank-medium DOWN. The endpoint is the
reason the harness exists; it now restores itself in a finally and shouts if the
service does not come back healthy.
ngram-mod's match window (n_min/n_max) decides which traffic it can see at all
-- the fleet's 48/64 preset is tuned for long verbatim echoes and finds nothing
in a repetitive-but-short output. Config entries now take |KEY=VALUE overrides,
so the window can be swept in the same run as the configurations it applies to.
The first run asked for a verbatim repeat with thinking ON, so the model copied
the passage inside reasoning_content, content came back empty for all three
configurations, and the cross-configuration identity check compared '' to '' --
a pass that proves nothing. echo and structured now disable thinking via
chat_template_kwargs, the echo row carries a copy_ratio (share of answer lines
that really are passage lines), and an empty answer in any shape is a loud
warning rather than a silent pass.
'structured' is new and answers the case the fleet actually runs on: repetitive
OUTPUT with no prompt overlap (record lists, tables, config rewrites), which is
where an n-gram matcher should pay and an echo-shaped test cannot tell.
A relative serve-script path plus a cd into its own directory double-counts the
prefix, so every configuration failed to start ('scripts/serve-frank-medium.sh:
No such file or directory'). The safety net did its job -- the unit was restored
-- but the run measured nothing.
Unlike MTP, ngram-mod needs no draft context: common/speculative.cpp adds the
ngram types unconditionally while the model-backed drafters are gated on
ctx_dft != nullptr, so 'draft-dflash,ngram-mod' is structurally allowed (and the
config vector's fixed order -- ngram first, then the drafters -- makes CLI order
irrelevant).
The harness grows the two things this question needs: --configs, and an ECHO
shape. ngram-mod can only ever match text that is present in the prompt, so
measuring it on prose calls it useless and measuring DFlash on echo calls it
wasteful; both shapes now run, and the greedy text of each is compared across
configurations because speculation may not change answers.
Four configurations of the same binary, identical prompts at temperature 0, the
unit stopped between runs, on the deployed payload:
none 17.7 tok/s card1 26.07 GB
draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all
four produced IDENTICAL greedy text, so the 'lossless verification' claim in the
runbook is now a check rather than an assumption. MTP is not the cheaper option
either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper
drafting loses (n=7 accepts a third of its proposals, n=3 more than half).
The two CANNOT be chained, and the failure is structural rather than tunable:
'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested
but model doesn't contain MTP layers', because a chain has ONE draft context and
MTP builds against whatever model that context holds -- here the DFlash2 file.
The serve script therefore passes --spec-draft-model in the dflash branch ONLY,
and the seam now asserts exactly that (the wiring whose violation is this
error), plus that the deployment stays on the measured winner.
scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking
policy wants the runtime's own harness for this, and the numbers above were
otherwise unreproducible one-offs.
The loader prints no per-device ledger at this verbosity, so the claim is
measured rather than read: +131072 tokens of context added 4.83 GB to one card
and 4.84 GB to the other (both carry their own attention layers' KV), and
neither card's total usage could hold the 31.46 GB file. Worth writing down
because the retired shape WAS two full copies — that is what a replica is, and
it is why the 29.8 GB FP8 checkpoint could never run there.
The LLM index still opened with vllm-radiance as frank-medium's runtime and
drew the replica pair in its topology diagram; the cooperation page's thesis
(collectives are impossible, so the answer is hardware) now has to name the
path the fleet actually took: llama.cpp's layer split, which launches no RCCL
kernel on either card. That reframes the M.2 riser as a means to TENSOR
parallelism rather than a means to using both cards at all.
Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.
The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.
Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.
The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.
omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.
The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
'cp -r SRC DST/' with a missing DST creates DST as a copy of SRC and drops a
directory level, so rocblas/library/* landed as lib/rocblas/* and hipblaslt's
arch subdirectories landed one level high. Nothing complains at load time: the
engine reads 29 GiB of weights, answers /health, and dies on the first request
with
rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat:
No such file or directory for GPU arch : gfx1201
Declare the destination and copy contents, both trees.
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.
AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.
Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
The runtime for the next frank-medium shape is mainline llama.cpp at
fb27a525, built for gfx1201 against the ROCm 7.14 nightly the ROCmFPX build
already used on this host, with the runtime bundled next to the binary.
Two things that pin the ref: DFlash2 (ggml-org/llama.cpp#27342) and qwen35
MTP. Neither is in the ROCmFPX fork's tip (c49ebdb, 2026-08-22), and the
DFlash2 drafter cannot run without the first.
The fetch script pulls the target at UD-Q8_K_XL plus the DFlash2 draft at
Q8_0, resumable and byte-verified -- a truncated 31 GB GGUF is otherwise a
load-time crash.
A redeploy of a running frank-large holds 7770 itself, so waiting on the port
before stopping the unit would have made every subsequent deploy fail. Stop the
user unit first (a no-op on a first cutover), then wait, then install and start.
The 2026-09-16 halogen restore failed exactly here. The ik root unit stopped,
systemd logged "Deactivated successfully", pasta tried to bind 7770 about a
second later, reported "Address already in use", and the quadlet exited 126.
Only Restart=on-failure recovered it -- minutes later, which is not a deploy.
The handler now waits for every socket on the port to clear (listener, accepted
and TIME_WAIT) before installing and starting the quadlet, and fails loudly with
the socket table attached if it never does. The retirement also reports when the
unit file was already gone, because this host's `disable` deletes it.
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.
Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.
Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.
The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with
the measured evidence rather than the theory.
**The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise
`AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall
buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix
it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per
ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1
mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16
bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈
1.4 ms at Gen4 x4.
**New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane
card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all
three, and `send`/`recv` fails too — which kills pipeline parallelism as an
escape, since vLLM's PP handoff is exactly that primitive.
**Correction.** I previously said no software path exists. That overshot. One is
open and unverified: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's
device source contains no live printf/assert/__assertfail, so the trigger is
unexplained, and the claim traces to search summaries, not a primary source. The
page carries the two-command test that would settle it. Also corrected: 0.9.3 is
radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm
7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory
path cannot run.
**Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just
benchmarks — including a bare 2-rank probe. Running one against the live service
on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle
required); a hung AQL dispatch wedging the driver is the plausible mechanism.
The policy also now says how to tell off from wedged: `ip neigh` from a peer.
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
The board showed mariadb, valkey and Postgres as "down". None of them is down:
the checker runs on Dankclaw and can reach that host's loopback, its container
names, and published LAN addresses — while all three cards probe
container-internal database ports (192.168.0.235:3306, 127.0.0.1:6379/:5432).
They were red by construction, which is how a status board stops being read.
Publishing a database port on the LAN to make a badge green is not a trade this
fleet makes, so they are wip with the reason in their description, and the apps
that front them carry the status. Gluetun (WireGuard, UDP) was the pre-existing
example of the same call.
The rule is now written down in the Homepage doc: a card must be probeable from
the monitor's host or be wip. That is the same lesson as the Prometheus jobs
that scrape HTML pages and can never report up — an always-red signal teaches
the operator to ignore the surface it lives on.
Frank