New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per
cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full
batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and
the limitations — including the one that cost me two runs, that amdgpu does not
return ~26 GiB at once, so a reload after a kill fails with "unable to allocate
ROCm0 buffer" and reads exactly like a configuration that does not fit.
AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the
10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead
of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at
the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the
frank-medium section now say 35:30 and batch 6144x512.
Runbook: deployed flags, the split history with the ubatch-512 table, the fault
note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent
requests).
Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1
ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser
premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not
imply Routing+ on this board (00:08.3 is CPU-side and Routing-).
log.md: the round trip recorded, newest first.
Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.
Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:
pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
iGPU alone 25.77 tok/s @64k, pp512 231.8
+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.
Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not
expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in
the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer
(Vulkan1) that cannot run the operation (NONE)" -- and both per-layer
cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU
is the only fallback side that works, which is exactly why -ot exps=CPU runs.
What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in
page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s
prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s --
but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7
spare, so it is not deployable beside the co-tenants. Throughput parity, memory
bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than
preference.
New page LLM/x4-pair-experiment.md carries the tensor inventory, the results
table, the two scheduler limits, reproduction commands, and the traps found on
the way. Both index pages also stop calling halogen retired -- the ik swap's
stale claims were still sitting in them.
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.
The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with
the measured evidence rather than the theory.
**The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise
`AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall
buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix
it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per
ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1
mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16
bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈
1.4 ms at Gen4 x4.
**New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane
card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all
three, and `send`/`recv` fails too — which kills pipeline parallelism as an
escape, since vLLM's PP handoff is exactly that primitive.
**Correction.** I previously said no software path exists. That overshot. One is
open and unverified: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's
device source contains no live printf/assert/__assertfail, so the trigger is
unexplained, and the claim traces to search summaries, not a primary source. The
page carries the two-command test that would settle it. Also corrected: 0.9.3 is
radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm
7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory
path cannot run.
**Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just
benchmarks — including a bare 2-rank probe. Running one against the live service
on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle
required); a hung AQL dispatch wedging the driver is the plausible mechanism.
The policy also now says how to tell off from wedged: `ip neigh` from a peer.
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling;
decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the
root-cause evidence chain, tuning levers, and co-tenancy traps
- AGENTS.md: fleet topology row + Active Services section for the ik
deployment; halogen demoted to rollback; bench policy unit type fixed
- LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE,
halogen-flash-server retired)
New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records
the hardware swap, the self-contained gfx1201 build, the measured layer-vs-
tensor decision (llama-bench multi-device syntax gotcha included), the
tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm
posture. AGENTS.md, README, docs indexes, Dankserver System Specs,
deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename;
freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED.
Work log entry added.
Raised /swap.img from 8 GiB to 64 GiB on Dankclaw. Host-level state with no
repo file, so it was invisible to the repo until now: fstab line 12 drives the
generated swap.img.swap unit, ext4, fallocated hole-free, swappiness 60,
zswap off. 748 GiB still free on /.
The reusable part is the ordering. The old file held 4 GiB of live pages and
MemAvailable read 73 GiB while only ~5 GiB was genuinely free -- the pinned
model weights are counted as reclaimable page cache, so the headline lies by
the size of the model. Swapping off before the replacement is online would have
pushed those pages into RAM that does not exist. Creating and enabling the new
swap first let them migrate instead: swapoff took 65 s and the host never
wobbled.
New page serverAppConfigs-docs/Dankclaw/Swap Configuration.md, linked from
the index, plus a pointer in the AGENTS.md memory-model section so nobody
reasons about headroom without checking swapon.
Old 8 GiB file staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/
pending approval to purge.
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.
Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.
AGENTS.md's boot-persistence gap note is replaced by the resolution.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.
Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
/home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
daemon itself is a host user unit (signal-cli-daemon.service) bound to
0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
GET /health from another LAN machine returned 200) — the SSH tunnel is an access
path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
tunnel via the shared named network searxng.network, not service:gluetun.
Docs only — deploy is a no-op for this change.
Document the Mac *arr/qBittorrent stack (VM placement, gluetun credential
gate, socat LaunchAgents, PBKDF2 qB seed) and log the day: dankserver
recovery + full quadlet cutover of the six media services, the frankbook
skip-propagation/PBKDF2 handler fix, the 7-service frankbook deploy, and
the jellyfin GPU-parity finding.
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, port 52626) is live on
Dankclaw. Documents the first-deploy crash-loop root cause: iommu=off on the
kernel cmdline breaks the XDNA2 SVA bind (amdxdna_drm_open ret -19), so every
/dev/accel/accel0 open fails with ENODEV; fixed with iommu=pt + reboot.
Whole-registry single-port serving with per-request routing by exact FLM tag;
unknown tags fall back to llama3.2:1b (no 404). Health = GET /v1/version.
Vision + audio towers proven live (image_url color probe; audio control-pair
token-count probe). Updates AGENTS.md fleet table + runtime section, both
index pages, and the change log.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
Live-verified 2026-08-29: plexmediaserver.service is an enabled native
systemd unit on Dankserver (not a quadlet; updates via apt, not deploy.sh),
and tautulli runs as a snap with no auth wall on 0.0.0.0:8181. The hunt list
named both; neither had a page. Index linked per the docs-completeness seam.
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
reference parser rejects before any network I/O (repository name must be
lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
5.7 quadlet path fails container create with 'image not known' for
registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
shell; HealthCmd is now a JSON exec-form array, which passes through the
quadlet generator's argv assembly verbatim.
Tests updated to assert the alias-tag reality plus the pull guard.
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Frank