Source reading in the qwen-fp8 container: glm5next is registered with an
amd/ backend dir + DSA indexer; r4d.so has native moe_dsfp4/moe_dsfp4_mt1
entries (has_dsfp4()==True), so the old 'MXFP4 = emulation, missing
kernels' verdict is dead for this fork. DSFP4 = MXFP4 weights + DeepSeek
activation law. Offload recipes exist for mxfp4 and W4A16 (R4dW4a16MoEExperts
in offloader/expert.py:114). GLM gate is now the payload format
(compressed-tensors MXFP4/W4A16, none public) + a HIP boot probe.
The KV trim is inert: arena identical (5,648 slots) at 4.5 and 4.03 GiB
(planner buckets; the trim went to margin). Final 256k config
MAXLEN=262144 KV_BYTES=4328521728 prefetch off; full-speed alternative
MAXLEN=184224 at 3 GiB.
Asked for iterative decode/prefill improvement at a 32k working point
without touching the model or the quantization. There isn't one: gufo's
served settings are already at this engine's tuned point, so no config
change was made and none is proposed.
Measured on the repo's own prose and shell sources at 32,762 prompt
tokens, salted per run against a warm prefix cache, 3-5 repetitions:
* --prefill-chunk 512 -> 8192 moves prefill 410.7-412.4 t/s. A 16x change
in chunk size, 0.4% out. The lever is dead at this depth.
* --draft-tokens 5 -> 10 gives 13.79-15.86 t/s decode, all inside the
run-to-run noise. The 5-rep confirmation REVERSED the 3-rep leaders:
dt9+chunk4096 looked best and came last (12.86), the deployed config
came first (14.71).
* --draft-policy fixed vs adaptive: no effect. --min-draft-tokens:
rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for
Q8 targets (token_embd.weight is Q8_0).
* --context 262144 -> 40960, tested as a diagnostic: no gain, and it
would break models.yml's declared contextWindow. Not shipped.
What does move the number is speculation itself — 6.65 t/s off vs ~15 t/s
on, 5.3x — and it is already enabled.
The two traps that cost most of the wall time are now in AGENTS.md's
benchmarking policy, because both produced FALSE results:
1. gufo bench cannot rank speculative settings and inverted the ranking.
Its corpus reports acceptance=0.933 +/- 0.000 with an identical trace
sha256 across draft widths — the same highly-predictable text every
run. It reported 29-37 t/s where real traffic gives ~15, and made
dt16 look 6% faster than dt7 when real text made it 14% slower.
2. A live A/B needs >=128 generated tokens. At 8, inter_token is
first-token noise; the first pass produced a nonsense ordering that
the 128-token pass reversed.
Plus the Server-Timing header is `ttft;dur=402.485` — the unit rides in
the NAME half, so splitting on "=" keys the dict "ttft;dur" and every
lookup silently returns nothing. Four configs reported "no Server-Timing"
while the header was present on all of them.
The disk was at 93% with 137G free. Nine retired payloads are now moved
to ~/.AGENTTRASH/20260928-retired-models/ with a MANIFEST.md giving a
restore command per item — moved, not deleted, and nothing is freed until
the purge is approved. Every one maps to a runtime already retired in the
runbook index.
Only the .gguf was taken from Qwen3.8-27B-UD-Q8_K_XL/: the live
frank-large DFlash2 drafter lives in that directory and stays put. The
manifest warns against restoring the directory wholesale.
Kept: checkpoints/ (23G) — unreferenced by the repo, but ComfyUI is live
and its workflows were not searched, so it is not confirmed dead. Also
logs/, because x4e-run.log is a SPARSE file (3.3G apparent, 4K real) and
frees nothing.
The reference check caught a live bug: serve-frank-small.sh's fallback
MODEL_DIR still pointed at the retired GSQ-RCO payload frank-small served
on 2026-09-16/17. frank-small.env has always overridden it, so the drift
was invisible — but a missing or unreadable env file would have turned a
config problem into a "missing asset" naming a model that is no longer
there. Repointed at Gemma4-12B-QAT/, which the env file already sets.
Not taken, by request: podman images (279G reclaimable, 95%) and the
uv+pip caches (73G). Together 352G of regenerable garbage, larger than
what was reclaimed here.
The 29.3G payload rollback target went with the rest, so the payload-only
rollback is now a re-fetch first; the runbook and AGENTS.md say so.
The target moves from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL to
ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL —
an abliterated fine-tune of the same base, under the non-OSI SWIFT OPEN
LICENCE 1.0. Port, alias, context, drafter and client contract are
unchanged.
Compatibility was settled from the GGUF headers before fetching anything:
both files parse to the same architecture (qwen35), the same 866 tensors
with identical names in identical order, zero per-tensor ggml_type or dim
mismatches, and the same MTP head. Two metadata differences that look
alarming are no-ops, confirmed against gufo's source — general.file_type is
deliberately not consulted (the file gufo already served records Q4_K_M
while containing no Q4_K tensor), and Swift's extra
qwen35.attention.recurrent_layers key is never read; layer kinds come from
full_attention_interval, which both files carry as 4 and which agrees.
The load test agreed: 9.2 s to listening, 66.5 GiB of the 116.7 GiB iGPU,
no loader warnings, and a green contract battery including a two-turn tool
loop.
The DFlash2 drafter did not move, so the payload is now two files in two
directories: DRAFTER_DIR is its own tunable. That broke the deployer's
stray-run sweep, which keyed on the target filename and would have stopped
matching — a stray run would then have held 7770 past the port gate. It
now keys on the drafter, which every frank-large command line carries.
The fetch script sends no auth header: the repo is public, and a token on
the curl command line is world-readable via /proc/<pid>/cmdline. The
anonymous transfer ran at 1.8 MiB/s; the host token through a mode-600
curl -K config reached 7 MiB/s.
Test seam: the mutation pass found four real gaps. Two fetch assertions
were satisfied by a header comment rather than the URL assignment, so
repointing a repository left them green, and nothing bound the env's
MODEL_FILE/DRAFTER_FILE to the filenames the fetch scripts download. Both
are now explicit. A harness bug worth recording: a bats --filter that
matches no test reports zero "not ok", indistinguishable from a mutation
that survived — the harness now checks the 1..N count. 16/16 caught.
The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so
grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.
dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.
Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.
Tests: 346/346 pass (no test surface touched).
Public, ungated repo so no auth header is sent — a token on the curl
command line is world-readable via /proc/<pid>/cmdline. Size and sha256
are the Hub LFS values (31457990272 / 9076f99a…), verified after fetch.
Not wired into frank-large.env: the gufo load test has to pass first.
Live numbers from the 2026-09-28 deploy: 13 s load to listening, RSS
30.6 GiB, ttft 549.8 ms / inter-token 24.2 ms on a short prompt, and a
green contract battery (health, ready, /v1/models at 262,144, content +
reasoning_content, tool_calls, SSE, and a two-turn tool loop).
Also recorded: gufo accepts every request field the harness sends in
extraBody and the full reasoning_effort set, both verified 200 — which
is why models.yml's reasoningEffortMap fold is now optional rather than
required, and why it is being kept anyway.
The .AGENTTRASH archive is ignored and exists only in a primary checkout,
so a filesystem grep fails there while passing in a worktree. git grep
searches the tracked tree, which is what the assertion is about.
gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.
Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.
Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.
Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.
The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.
Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.
Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
prefill is token-by-token; 1000 t/s is ~20x away.
Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.
Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
New ACTIVE runbook (setup/doctor PASS=34/FAIL=0, UVA transport verified
in serving workers, wire-verified alias/reasoning/metrics), index +
vault entries flipped: llama.cpp b11030 endpoint RETIRED 2026-09-27 to
rollback target. Topology diagram updated.
download-frank-primary-fp8.sh + the fp8 manifest fetched the vLLM 0.30
payload that never deployed; the R9V cutover's payload is the R9V setup
flow's (packages + PLE, state-gated). Scripts move to old/ per policy.
Replace the vLLM 0.30 UVA-offload wiring (never deployed; llama.cpp
b11030 endpoint replaced by Frank's order, downtime window open) with
the R9V radiance-vLLM shape on the same root unit:
- serve-frank-primary.sh: layer the on-host qualified setup state
(104 tuned R9V_* keys, /home/dank/r9v-state/uncensored/setup.json)
under repo production overrides, then exec the fork's own launcher
through the docker->podman shim and block on docker wait so a crashed
engine is a nonzero ExecStart (launch.sh --detach alone would fake
green). Gates: pinned image by OCI manifest digest in RepoDigests,
PLE at pinned byte count (stat -c %s), all GGUF shards + mmproj.
- frank-primary.env: thin production overrides only (LAN bind 0.0.0.0,
:7770, alias frank-primary, container frank-primary-r9v, GPU BDFs);
tuned engine shape stays in the setup state by design.
- frank-primary.service: same root-unit shape (User=dank,
LimitMEMLOCK=infinity for the tiered-expert pin), plus ExecStop
podman stop — docker wait cannot stop the container child.
- deploy-remote.sh deploy_llm_primary: state-driven gates (state ready,
image digest present in rootless storage, PLE bytes, shard files)
replace the fp030 shard-index gate; false-green MainPID check kept.
- tests/frank_primary_unit.bats: seam rewritten for the R9V shape.
Unit restart SIGKILLs conmon before --rm cleanup runs, leaving the container
name registered: every restart then dies on name-in-use while the orphan
holds the port and GPUs (observed live 2026-09-27, NRestarts=6 with the
container Up and the unit auto-restarting). The serve script now clears its
own name before exec podman run; the deployer stops the unit and removes
the container BEFORE waiting on the port.
HOME inside the container pointed at the root-owned WORKDIR /app, so
vLLM's registry cache write hit PermissionError; HOME now points at the
cache volume.
The vLLM cutover itself is blocked on a structural wall (AMD Qwen4Exp impl
creates the 51.2B PLE table as a bf16 on-GPU param: 47.69 GiB transient on a
31.86 GiB card; the pinned-host UVA path that fits this hardware is
CUDA-gated). llama.cpp is restored live at :7770; host HEAD unchanged.
The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the
FP8 cutover on :7770 — same port, same alias, no client moves.
Serve shape: locally built ROCm gfx1201 image
(localhost/frank-primary:vllm030) execs via podman as dank from the
root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload:
--offload-backend uva pins the PLE family (47.75 GiB, 138 tensors:
131 n-gram table shards + 7 per-layer conv/proj) + routed experts
(114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined,
~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card
strands 2.6 GiB of experts on the cards). Parsers per the host's
own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM
0.30 emits the channel as `reasoning` — reasoning_content is
renamed server-side) + enable_thinking/preserve_thinking template
kwargs. Spec decode stays env-wired and OFF at bring-up.
Deployer deploy_llm_primary rewritten: image gate, index-driven
shard-count gate (131), stale-container cleanup, port wait, and the
false-green hardening — health is only green when the unit's own
MainPID is live behind the answer (2026-09-26 lesson).
Seam tests rewritten to the vLLM contract (15 gates): offload
arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite
349/349 green.
scripts/manifests/qwen3.8-flash-next-fp8.manifest: 144 entries (131
safetensors shards + aux) with Hub LFS sha256s for
Qwen/Qwen3.8-Flash-Next-FP8. download-frank-primary-fp8.sh wraps
hf-fetch.sh: idempotent fetch, full sha256 re-verify of an on-disk
payload (verified live on dankserver: all 144 files match, ~10 min).
Part of the frank-primary vLLM cutover prep; the serve trio itself
lands with the image deploy.
The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).
- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
/opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
the no-credentials policy (paid engine acceptable when bounded + ordered).
Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
The first no-redirect HEAD fix read X-Linked-Size on every 3xx, but
non-LFS files on a Xet-backed repo answer 307 to HF's own
/api/resolve-cache WITHOUT any X-Linked-Size header. The fallback to the
stub's Content-Length recorded the redirect page size (254-264 bytes)
instead of the file size, so hf-fetch.sh failed the size check on every
aux file it verified against the regenerated manifest (README.md
"65400 != 254", config.json "72514 != 258", 7 files on the FP8 repo).
Distinguish the two redirect shapes: 3xx WITH X-Linked-Size = LFS/Xet,
whose X-Linked-* headers are the truth and whose redirect must NOT be
followed (the Xet bridge ETag is the CAS hash, not the sha256); 3xx
WITHOUT = non-LFS, where the Location (followed once, via urljoin)
answers 200 with the real Content-Length and a git-blob-sha ETag that
the 64-hex gate correctly rejects (size-only check for those files).
Verified against Qwen/Qwen3.8-Flash-Next-FP8: all 7 previously-failing
aux files now record their on-disk byte counts exactly (README.md 65400,
config.json 72514, merges.txt 3353259, ...), shard shas unchanged
(model-00002 6841fe21...), total 172.82 GiB across 144 files.
On Xet-backed repos (Qwen/Qwen3.8-Flash-Next-FP8), the resolve endpoint 302s
to a CDN bridge whose final ETag is the Xet CAS chunk-dag hash, NOT the
content sha256. urllib follows redirects, so the manifest recorded the Xet
hash and every downloaded shard failed sha256 verification against a hash no
download could ever match (8/8 shards FAIL, all sizes correct, on-disk bytes
matching the Tree API's lfs.oid). Classic LFS CDNs serve ETag=sha256, which
is why this was invisible on prior fetches.
Fix: HEAD via http.client, which does not follow redirects. 200 = direct
answer (Content-Length + ETag, non-LFS); 302 = LFS/Xet redirect
(X-Linked-Size + X-Linked-ETag = content sha256). Verified against the FP8
repo: shard model-00002-of-00131 now records 6841fe21..., the sha256 of the
bytes already on disk (cross-checked against the Tree API lfs.oid).
Instrument the live server to answer "can -ot target the most active
layers": AMD IBS op sampling on llama-server during 4 mixed concurrent
generations, sampled load addresses mapped through /proc/<pid>/maps to
backing tensors with extents derived from consecutive GGUF header
offsets (no quant type-size tables - a wrong per-element size silently
mis-buckets tensor tails, first run lost 59% of expert samples to a
junk bucket).
Measured (75 s window, 2.7M expert-tensor samples): per-block read
volume is FLAT, 0.93x-1.08x of mean across all 40 CPU-resident blocks.
MoE top-k routing fires every block every token, so "most active
layers" do not exist at -ot granularity - any 8 GPU-resident blocks
are equivalent, the lever is COUNT, already at the 5/3 VRAM ceiling.
Real skew is per-EXPERT (top 10% of 48x512 cells carry 40.5% of
samples), below tensor-override granularity. No config change.
scripts/frank-primary-expert-activity.py: dump + sample + analyze in
one entry point; --analyze-only re-analyzes saved samples. Instruments
production serving in place; not a benchmark.
Blocks 0-4 on ROCm0 + 45-47 on ROCm1. This exact shape was probed and
battery-verified earlier the same session: card0 2.54 GiB free at boot,
2.11 GiB after the vision graph (steady under load), card1 unchanged
at ~3.3 GiB. Card1 cannot take another block (0.82 GiB would OOM the
vision graph + boot drift), so 5/3 is the physical limit at the
4x262144 / V-q8_0 shape. OT assertion pins the new value.
- CONTEXT_SIZE=1048576 / PARALLEL_SLOTS=4: four conversations, each the
full native 262144 window (per-slot window unchanged; slots divide
the pool, they never multiply it).
- CACHE_TYPE_V=q8_0 / CACHE_TYPE_K=f16: main KV drops 24,576 -> 18,816
B/token (-23.4%); the shared MTP drafter's KV ignores the V type
(measured: 2,048 MiB draft pool = the f16 law at 1,048,576 cells).
- OT_OVERRIDES: blocks 0-3 on ROCm0 and 45-47 on ROCm1 stay
GPU-resident (2.42/2.49 GiB per block, measured); the rest stay
CPU-computed. Measured free under load: ~4.9 GB card0 / ~3.4 GB
card1; a 5th block on card0 lands at 2.27 GB, under the 3 GB
checkpoint floor, so blocks are atomic and 4 is the safe fill.
- serve script: --cache-type-k/v env-wired tunables; ONE joined
--override-tensor flag (b11030 deprecates repeated flags, and the
loader is first-match-wins, so the specific patterns must precede
the broad exps=CPU sweep).
- seams: the window test asserts per-slot = 262144 at 4 slots; new
KV-type test; OT exact-value assertion pins the ordering law.
29,696 B/token ctx-scaled (12 of 48 layers carry KV - full_attention_interval=4;
indexer and draft KV included); weights/RS/compute are ctx-independent.
262k = 18.75 GiB total / 9.37 per card; 100k = 14.3 / 7.1. Parallel
slots split the pool rather than multiplying it.
Both b11030 serve scripts never passed --metrics, so /metrics answered
501 and Prometheus could not scrape frank-large (silently down since
its deploy) or frank-primary. frank-primary also gets -lv 4 (VERBOSITY
env tunable, default 4) for the full per-buffer load breakdown.
Battery 6/6: alias, reasoning_content, tool_calls, vision (probe image
described exactly), reasoning_effort high -> 200. Decode 11.8 t/s at
depth-0 prose (draft 109/accepted 72), 0.864 mean len 2.73 on tool
traffic. VRAM 29%/36% (output-head card carries the drafter). Also
records the ROCm1 drafter law in the runbook body (borrowed output.weight
+ n_copies==1 abort) and the two cutover traps: foreign-process health
false green, and llama-server SIGTERM cleanup hang.
The shared MTP head borrows the main model output head, which a -ts 1,1
layer split places on ROCm1. With tensor overrides set, n_copies==1 in the
draft's scheduler, and a borrowed output.weight in a ROCm1 buffer aborts the
draft graph (ggml-backend.cpp:941) when the draft context is on ROCm0 —
the unit crash-looped 4x at ~12 s per attempt during the 2026-09-26 cutover.
Probed live on the host: ROCm0 aborts identically, ROCm1 loads and serves.
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts
(scripts/dump-gguf.py, added — a dependency-free reader that never
touches tensor data, so a 50 GiB part scans in milliseconds) before
the first load.
What the dump says (parts 2-3, 2026-09-26):
- The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd.
weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of
part 3. It is the same tensor the shared MTP head borrows, so the
=CPU override is required for placement AND for the draft load
(GPU-resident borrow: "invalid vector subscript").
- The model is qwen4exp, 512 experts, top-10 routing
(qwen4exp.ple.ngram_size=3, heads_per_ngram=8).
- Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors.
The placeholder pattern ngram_embeddings matched nothing — dropped from
OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook,
LLM index) before it could masquerade as a working override at load
time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU.
Suite: 346 green.
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
Fetches into /home/dank/models/Qwen3.8-Flash-Next-GGUF/ keeping the Hub
layout: the six Q8_0 split parts (175.30 GiB — never concatenated, part
1's header advertises file_count=6), the shared Q8_0 MTP drafter head
(2.6 GiB — "shared" borrows the target's embeddings; unsloth's MTP guide
names it the recommended head), and the BF16 vision projector. Total
191,919,144,448 bytes.
Sizes and sha256s captured from the resolve endpoint's x-linked-etag/
x-linked-size on 2026-09-26; every artifact is checked for exact byte
count AND sha256, curl -f so an auth error fails instead of writing the
error body, resumable via -C -. The repo is public: no token machinery.
The frank-medium retirement tail the 2026-09-26 lucebox cutover left
uncommitted: the radiance quadlet tree, its swap/fetch tooling and its
seam test are deleted (archive copies live in dankserver/.AGENTTRASH/
20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926-
frank-medium-radiance/), the planner/allowlist/test-changed mappings
drop the service, and frank-large's chat template moves to scripts/
(its old home was the retired frank-medium dir).
Finished here:
- tests/frank_primary_unit.bats: the lucebox quadlet seam (boot
persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved
env-name guard, checkout-mounted entrypoint/healthcheck, first-start
patience, deployer payload/port gates).
- test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move
files to trash, and the wrapper refused those additions with "no
focused-test mapping".
- observability: the vllm: namespace left with the radiance runtime.
Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode,
queueing, endpoint-down); the dashboard drops the vLLM KV/latency
panels; prometheus.yml documents that frank-primary (lucebox) has no
/metrics and therefore no scrape job. llm-fleet.json was left
corrupted mid-edit (missing opening brace) and is rewritten from HEAD.
Suite: 339 green (was 7 red).
The runtime's auto budget is gpu_free - kv - warm - safety (22.35 GiB), which
leaves only the 0.50 GiB safety margin free while the fit check demands the
1.50 GiB discrete headroom, so auto-sizing refuses on any 32 GiB target:
"device 0 memory: 30.97 of 31.86 GiB used, 0.89 GiB free (headroom 1.50
GiB): does not fit". 18000 MB is the value DS41.md measured its R9700 rows
at: ~1500 experts resident, ~5 GiB free on device 0.
--ds4-prefill dense made requires_monolithic_model() true, and a monolithic
load asks ggml for the entire 187 GiB expert set as one device-0 buffer:
"allocating 191932.25 MiB on device 0: cudaMalloc failed: out of memory",
followed by "monolithic HIP load required for dense prefill". Approximate
prefill (dense, sparse) is expressible only where the experts are resident
(in-process owners). exact feeds the prompt token by token, which is the
path the hybrid tier streams experts for. Chunk follows DS41.md's
cold-expert recipe (1024).
wc -c falls back to reading the whole stream when its fstat shortcut does
not apply, and this gate runs over a 377 GiB GGUF: the 2026-09-26 cutover
deploy sat in wc for minutes (30 s of CPU, NVMe-bound) before it could
reach the cutover at all. stat -c %s is metadata-only and equally exact.
The llama.cpp hunk (ggml_cuda_node_launches_nothing) was needed on main's
tree; the branch restructured that file and compiles clean, so only the
apt-list and cmake-3.31 hunks remain. Regenerated from the host clone.
The deepseek41 architecture exists only on Luce-Org/lucebox's
feat/ds41-engine branch; main rejects the checkpoint with
"unsupported model architecture 'deepseek41'". Re-aim the image build at
that branch and fix the launch shape, which had been copied from the V4
DS4.md recipe:
- drop --ds4-fused-decode and --ds4-fused-verify-f16-kv, and use
--ds4-prefill dense: V4.1's validate_model_features refuses the V4-only
prefill/fused paths (layer split and --mmproj too).
- --chunk 4096 (the V4.1 dense-prefill measurement point), --max-ctx
131072 (the V4.1 default window), --prefix/--prefill-cache-slots 0 as
the DS41.md bring-up recipe.
- drop --cache-type-k/v: the DeepSeek4 backend ignores them; the MLA
compressed caches are F16 by design.
- expose BOTH R9700s and set LUCE_EXPERT_STREAM_DEVICE=1, so hip:0 owns
the dense weights plus the hot expert stack while hip:1 holds the
streamed-expert slot cache instead of competing with it.
- quadlet: 384 routed experts per layer, Engram tables, 40 layers.
Replaces frank-medium (vllm-radiance TP=2) on :7770. The 377.2 GiB
43-layer MoE (256 routed experts, top-6) is mmapped: dense work and the
MLA KV cache on the R9700, routed experts computed on CPU from the
page cache (246 GiB RAM). The unit bypasses the image entrypoint
(Qwen/DFlash-only flag set) and execs luce_server with the DS4 launch
shape; --ds4-fused-decode's monolithic-load fallback IS the serving
mode here. deploy_llm_primary gates on image + model byte count,
stops/masks the radiance unit, waits for :7770, installs and starts.
Runbook: frank-primary-lucebox-ds4.md.
With a missing file, -C - sends Range: bytes=0-, which HF's xethub
CDN rejects with 400. Xet-backed repos (e.g. Lucebox/DeepSeek-V4.1-
Flash-ROCMFP23-GGUF) fail on the first fetch; the old LFS backend
tolerated the zero-offset range.
Dim the spec line (--dim) so the card reads pale role -> dim spec -> bright
spring-green inputs -> dim use. The inputs line now pops as the bright middle
tier and the spec/use are de-emphasized as reference detail.
Add --input: #00ff9d to the palette and point .desc-inputs at it, so the
Inputs line is visually distinct from the bright-green spec (--fg), the
pale-green role (--head), and the dim-green use (--dim).
Each LLM card now lists its accepted inputs as a line (Inputs: text · image ·
audio), sourced from the harness models.yml:
- frank-large: text (no vision projector)
- frank-medium: text, image (Gemma4 vision path)
- frank-small: text, image, audio (Gemma4 projector audio encoder; audio is
not in models.yml's input list only because OMP's schema rejects it, but the
endpoint accepts input_audio parts)
New .desc-inputs CSS + bare-mode reset; /docs model guide gains an Inputs line
per model. bats 30/30 + live render verified.
Each LLM card now carries a details object (role/spec/use) rendered as a
bold pale-green role, a bright-green spec line, and a dim 'Use:' line.
app.js descHTML() falls back to the plain description for non-LLM cards;
new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset.
Roles made neutral (no nicknames): frank-large = deep reasoning,
frank-medium = primary model, frank-small = fast utility model.
The /docs model guide was fixed — it had drifted to retired model info
(frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8
~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8
19.4 GiB, reordered large->medium->small. bats 30/30 + live render.
frank-large: cybersecurity + debugging (requires Frank's permission)
frank-medium: primary intelligent model (surpasses Claude Sonnet 5 per
Artificial Analysis) — software engineering, sysadmin, instruction tasks
frank-small: fast/cheap utility model — agentic search, RAG, research
subagents, translation, roleplay, summarization, small utility tasks
Updated both the status cards (services.json) and the /docs model guide.
The auth works via the modern /Users/AuthenticateByName endpoint (what the
web UI uses). The 401 was a test artifact from probing the dead legacy route.
No config change needed; LDAP-Auth plugin + authentik 2026.8.3 outpost healthy.
The UserWriteStage writes prompt_data['password'] via user.set_password().
A PasswordStage only authenticates the current password. Document the
correct mechanism (password field on the default-user-settings prompt +
empty-password guard policy) and the throwaway-user verification.
The default-user-settings-flow was missing a PasswordStage, so users could
edit name/email/locale but not their password. Added a PasswordStage
(InbuiltBackend) bound at order 101. Documented the flow's stage table and
the two ORM gotchas (MTI through model, required backends array).
The local username/password path is retired (ENABLE_LOGIN_FORM +
ENABLE_PASSWORD_AUTH false, ui.enable_login_form DB override false) and
qa-e2e-admin is out of the authentik Admins group. Documented in the SSO
section and History.
The authentik OIDC round-trip is verified, so retire the local
username/password path. ENABLE_LOGIN_FORM hides the form and
ENABLE_PASSWORD_AUTH refuses password logins; the only way in is now
the OIDC flow, which folds into the local account by email. The
ui.enable_login_form config-DB override is flipped to false in the
same change (the DB wins over env, so both must agree).
Footer link opens the authentik default-user-settings-flow in a new tab,
giving users a self-service entry point to change their password and
manage MFA/credentials without admin intervention.
systemd splits unquoted Environment= values on whitespace, so the
quadlet quotes OAUTH_SCOPES to keep all four scopes; update the bats
assertion to expect the quoted form (it was checking the old
unquoted line and failing).
Records the LDAP-Auth plugin wiring: the Jellyfin LDAP provider +
outpost on Dankclaw, the svc-jellyfin-ldap service account, the
jellyfin_users/jellyfin_admins access groups, and the key gotcha that
the bind credential is a PASSWORD (set via Recovery), not an API/app
token — the default auth flow's standard-password backend rejects
tokens with 'flow error password: Invalid'. Verified end-to-end:
alex (in jellyfin_users) logs in with their authentik password and the
plugin creates the matching Jellyfin user.
systemd splits unquoted Environment= values on whitespace, so
'OAUTH_SCOPES=openid email profile roles' arrived as 'openid' and the
authorize request carried a single scope — the id_token then lacked the
email claim and the callback failed. Quoted, all four scopes flow.
Also: assign the default openid/email/profile ScopeMappings to the
provider (2026.x no longer auto-creates them; without them the token
grant is the intersection with the provider's mappings = roles only,
and userinfo 403s).
fill-verify-Enter with a single inputValue read per stage (identification
and password); the toHaveValue assertion hard-failed every attempt when the
re-render wipe landed inside its 5s window (00-auth.setup failure 2026-09-24)
Boot 2026-09-22 at 0.98 fill: KV pool auto-sized to 425,047 tokens (1.62x
the 262,144 window) with 14.42 GiB per card available for KV. The DFlash2
drafter on the Swift finetune measures mean acceptance length 7.00 / 85.7%
(easy) and 5.08 / 58.3% (varied) — it survives the base-model change.
Replaces the estimated ~420k / ~17.6 GiB figures with the measured values.
0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.
Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
The Qwen-Image 2.1 VAE is a video VAE (5D convs, time_conv, depth-3).
ComfyUI detects it as WanVAE (latent_dim=3), but EmptyLatentImage produces
a 4D latent. The decode function's memory_used_decode lambda accesses
shape[4] which is out of range for 4D.
Fix: expand 4D latents to 5D in the decode function when latent_dim==3.
Added to the patch script for persistence across container recreations.
The serve script now passes --chat-template-file pointing at the
Qwen-Sharp v22.4.0 template. Without it the binary falls back to the
built-in peg-native template, which produces a ~73,600-token prompt for
a ~250-token request. With the sharp template wired in, the same request
produces a 62-175-token prompt.
The serve script only passed --chat-template-kwargs, so the binary fell
back to the built-in peg-native template. A 250-token request produced a
73,600-token prompt (280x the expected ~260). Wiring in --chat-template-file
drops the prompt to 62 tokens. Draft acceptance 0.79 (mean 4.14), model
answers correctly, reasoning lands in reasoning_content.
The qwen_image21 detection check and GGUF node arch list patch are
applied by a startup wrapper (comfyui-start.sh) that runs before
ComfyUI starts. The patch script and wrapper are baked into the image
so they survive container recreations. The quadlet's Exec now points
at the wrapper instead of main.py directly.
The hermesadmin.franksdank.cloud dashboard (9119 — config, API keys,
sessions) is a distinct admin surface from the web UI client. Add it as a
third AI-section card; probe 9119 (302 login redirect counts as up, verified
on Dankclaw).
The admin Infrastructure section carried the retired 'dnsmasq' (probe-only,
no link) — Pi-hole FTL (a dnsmasq fork) replaced it on 2026-08-24. Replace it
with a linked Pi-hole card (pihole.franksdank.cloud, probe 8083 — the 403
basicauth counts as up). The Hermes card was probe-only; add the web UI
client link (hermes.franksdank.cloud). Both probes verified up on Dankclaw.
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
Swap the serving checkpoint back to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (quark/W4A8) and serve it with
KV_DTYPE=auto (unquantized bf16 KV) instead of the hardcoded fp8. fp8 KV
stacked on MXFP4's already-quantized W4A8 weights compounds attention-path
precision loss; clean KV tests whether that was hurting output quality. The
freed MXFP4 weights (19.37 vs 30.9 GB) buy the KV pool back, so the full
262,144 native window fits unquantized - MAXLEN returns to 262144.
The entrypoint gains a KV_DTYPE tunable (default fp8, the historical behavior),
shared across both quantization paths; the quadlet sets auto. Rollback: SNAP
back to the FP8 checkpoint with KV_DTYPE=fp8 and MAXLEN=192000.
The LLM backends section said two ROCmFPX endpoints; the fleet now serves
three (frank-large/medium/small) with different runtimes. Rewrite it, fix the
gotcha that claimed host.containers.internal does not resolve on a bridge
(it does — verified end-to-end for both on-host endpoints), and add a History
entry for the 2026-09-21 model-picker rebuild.
The per-key config DB already pointed web search at SearXNG (engine=searxng,
query URL http://192.168.0.143:8081/search), but only inside the data volume.
The quadlet now carries the same values as seeding env vars, so a fresh data
dir comes up wired, and the doc records the access path: the internal
host-published port, not the search. vhost — the Anubis->Authentik guard
chain is browser-only, and the bridge egress sits in SearXNG's pass_ip, so
the 4-req/IP/hour JSON cap never applies.
The user page now orders Web Apps, Media Library Apps, AI — the models sit
at the bottom, below the apps people click. The AI card and /docs#models
descriptions drop the infrastructure (runtimes, GPUs, tensor parallelism,
draft heads) in favor of what users care about: the hosted model, its
size, the context/parallel capacity, and what it's good at.
The build default is 8192 MiB. dankclaw is memory-tight (frank-large's
70 GiB GTT plus the rest of the stack) and Gemma4 runs fully on the 3090
(-ngl 999), so the CPU-side cache is not the hot path. Cap it at 2048 to
give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet model, so the
>12.8 GiB CACHE_RAM floor does not apply.
Wired through frank-small.env (CACHE_RAM) -> serve-frank-small.sh
(--cache-ram), with a wiring seam so the cap cannot be silently unwired.
Each section's panels fill an auto-fill row (as many per row as the
viewport allows) and the sections stack vertically — the side-by-side
column layout was the wrong read of 'render the sections horizontally'.
The board rendered vertically for a user whose browser had cached the
pre-sectioning style.css: deploys overwrite /srv/homepage in place, and
nothing told the browser to revalidate. The @home handler on the post-auth
listener now sends Cache-Control: no-cache, so every load is an ETag
round-trip (a cheap 304 when unchanged) and a deploy is always visible on
the next navigation.
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.
- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
~/.AGENTTRASH/20260921-frank-large-assembled/
Both pages now group their panels under section headings rendered by shared
renderSection()/renderPanel() components in app.js (SECTION_ORDER fixes each
page's order: user = AI, Web Apps, Media Library Apps; admin = Media Library
Apps, Observability, AI, Infrastructure). The WIP badge splits into WIP: OPEN
and WIP: CLOSED via the wip value ("open"/"closed"); the checker passes the
value through as wip-open/wip-closed. The frank-large card follows the
unsloth llama.cpp cutover, frank-medium follows the FP8 checkpoint, and the
/docs model guide matches. Tautulli (Plex statistics, 192.168.0.235:8181)
joins the admin page's Media Library Apps section with an open link.
Only the 8084 forward is exposed; the desktop GUI runs inside the
container netns and is internal-only. Drop the now-unneeded
CUSTOM_HTTPS_PORT (no host-port conflict exists inside the netns).
lscr.io/linuxserver/calibre:9.15.0 (desktop build) running calibre-server
as the s6 main service on internal 8090 -> host 8084, auth-gated with a
plaintext userdb (mode 600) bootstrapped once by the deploy handler.
The desktop GUI is kept on 8080/8182, basic-authed with the same
credential pair via the machine-local env file.
The verification looped over every LFS file under UD-IQ4_XS/ and MTP/ in
the live Hub tree — the five MTP head variants we do not fetch (BF16,
Q4_K_M, Q8_0, shared-BF16, shared-Q4_K_M) printed MISSING, set ok=False,
and set -e killed the script silently between verification and assembly
after a 3-hour download. Verify exactly the FILES set instead (passed to
python as argv), and report NOT IN HUB TREE when an expected file has
vanished upstream. Also fix the layout comment: the transport parts live
in UD-IQ4_XS/, not parts/.
New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload,
deployed config, the explicit -md, the 114 GiB GTT memory model, client
contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback
target, payload and runtime intact on disk). AGENTS.md topology row,
frank-large section, harness note, runbook index, and the GPU-budget
lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure —
the CIRU Orca held 92.6 GiB on the same device). LLM index header,
mermaid, active list, and the vault root index updated. Work-log entry.
Replace the CIRU v4.4.1 Orca runtime with the unsloth llama.cpp prebuilt
(b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB, three HF split parts
concatenated) on the Strix Halo iGPU. One conversation, 200,000 ctx,
PARALLEL_SLOTS=1, MTP depth 2 via the explicit -md shared-Q8_0 head
(sidecar auto-discovery does not search the mtp/ subfolder), text-only.
Fleet contract unchanged: port 7770, alias frank-large.
- serve-frank-large.sh: unsloth binary, -m/-md, --spec-type draft-mtp
--spec-draft-n-max 2, --jinja, preserve_thinking; gates on binary,
GGUF, MTP head; conditional --mmproj on ENABLE_VISION=1
- download-frank-large-unsloth.sh: resumable fetch of 3 parts + MTP head
+ mmproj, each verified against its LFS sha256 fetched live from the
Hub tree API; concatenates parts, records/verifies SHA256SUMS
- deploy-remote.sh: LLM_LARGE_MODEL/LLM_LARGE_DRAFT paths, pkill pattern
on the model path (not the binary name), wait_port_free gate
- tests: seams for 200k/1-slot, explicit -md, text-only, handler gates,
sweep safety
Runbook deployed-flags table and AGENTS.md now name the template
(froggeric v22.4.0, repo-tracked, terse by default) and the rollback
target; the stale qwen-fixed-v22.3.jinja row and the long-gone
MM_MERGE_ENGINE note are gone.
The template was staged in the working tree (then stashed by a peer for the
prometheus deploy) but never committed, so the live container kept serving
/patches/qwen-fixed-v22.3.jinja. Commit the repo-tracked template and flip
the quadlet: CHAT_TEMPLATE now points at it, mounted read-only at
/templates/qwen-sharp-v22.4.0.jinja. The entrypoint already passes
--chat-template on the shared exec path, so this applies to the FP8 shape.
Rollback: one env flip back to /patches/qwen-fixed-v22.3.jinja.
node-exporter is net=host, so its 192.168.0.143:9100 bind is a host bind,
not a published port — the bridge-networked Prometheus container cannot
reach its own host's LAN IP there (netavark DNATs published ports only).
The node-dankclaw target has been red by construction since the
observability cutover while the container reported healthy. Point it at
host.containers.internal:9100 (netavark delivers that to the host LAN IP,
where node-exporter listens) — the same rule frank-large and frank-small
already use. node-dankserver is a different machine and keeps its LAN
address. Update the cross-file contract test and the Targets table.
DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
The entrypoint now reads the model's weight quantization from config.json
and serves the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP
self-speculation, ROCM_AITER unified attention with the R4D all-reduce)
or the MXFP4 patch-and-serve path. A model swap is now: change SNAP
(+ SPEC_METHOD/SPEC), restart.
- radiance-entrypoint.sh: quantization detection, gated MXFP4 env vars
and patch block, --quantization fp8 flag
- quadlet: SNAP -> Qwen3.8-27B-Uncensored-FP8, SPEC_METHOD=mtp, SPEC=8,
KV_MEM=0 (profile the shape, the documented plain-FP8 default)
- deploy-remote.sh: model-aware gates (read SNAP/SPEC_METHOD from the
quadlet; drafter gate only for dflash, r4d.so gate only for mxfp4)
- tests: quantization-aware drafter/spec and KV-pool tests
- hf-fetch.sh: generic, resumable, verified, idempotent HF fetch tool
- frank-medium-fp8.manifest: 17 files (10 config + 7 shards) with sizes
and LFS sha256
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
cached tokens (its own HELP line), so the ratio was unbounded — measured
77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
fire on that side. Now cached / (cached + prompt), with the traffic floor
counting both terms too; the header documents the unit trap. Also fixed the
LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
the llamacpp:* family (verified live), so the alert covers all three
endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
namespaces. Every panel reads both (the endpoint picker selects); the
KV-pool-fill panel retired 2026-09-16 returned with
vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
Llm.* rules; the stale halogen-era description is gone. All 21 panel
expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
reuse happens at 1,648-token block granularity (attention block size padded
to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.
./tests/run.sh: 325 pass.
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.
- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
had), the job:llm_vllm_prefix_cache_reuse:rate5m and
job:llm_spec_accept_length:rate5m recording rules, and re-cut
LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
per-family traffic floors so an idle endpoint cannot fire them. Every
expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
and frank-large scraped at host.containers.internal:7770. Both are HOST
processes on Dankclaw and netavark DNATs published container ports only, so
the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
(per-endpoint reachability by process type; every named series must be on the
measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
capabilities, rollback, replica-era sections labelled); llama.cpp runbook
carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
verdict; Observability records the namespace split and the DNAT lesson;
AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
index flipped; both indexes updated; work-log entry added.
Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
The warm-up request was a warm prefix-cache hit on the next run, so it
hid exactly what the arms were meant to differ on (prefill). Token
counts now come from the streamed usage of each job itself, and --salt
defeats cross-arm inheritance.
bench-conversation.py answers 'what does this endpoint feel like': ten real
jobs streamed once, TTFT split from decode tok/s, salted so no arm inherits a
warm cache, optional concurrent burst for capacity. Needed to compare the two
tensor-parallel runtimes the 2026-09-20 board swap unblocked.
dankserver-route-primary.sh pins eno1 (the fleet address, .235) as the
preferred default route after the board swap brought up eno2 with DHCP and a
lower metric; eno2 stays up as failover. nmcli device reapply, not connection
up, so the link this session rides is not torn down.
librccl DT_NEEDEDs librocm-core.so.1, so a GGML_HIP_RCCL=ON binary would not
exec from its bundled lib dir (measured: 'error while loading shared
libraries: librocm-core.so.1'). The layer-split build never pulled it in.
Layer split needs no collectives, so RCCL stayed OFF. The 2026-09-20 board
swap put both R9700s on CPU root ports with atomic routing, so a
tensor-parallel (row-split) runtime is now buildable: pass
GGML_HIP_RCCL=ON with a distinct BUILD_DIR beside the production one.
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.
- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
topology table, rebuild-hypothesis door marked moot, probe plan is now
standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
eno2 (192.168.0.247) second 5GbE port holding the low-metric default
route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.
Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
The frank-medium runbook's deployed-flags table still carried the
predecessor's values (35,30 / 393216 / 196608 / draft-mtp / BF16 mmproj);
updated to the deployed shape (32,33 / 327680 / 163840 / draft-dflash / f16
mmproj) from scripts/frank-medium.env. The halogen runbook still described
itself as 'the active frank-large runbook'; marked RETIRED 2026-09-19 with a
pointer to [[frank-large-ciru-orca]].
The CIRU launcher gates on hip/lib/libamdhip64.so + rocr/lib/libhsa-runtime64.so
under its runtime root and exits 2 without them; the build links the private
venv through its RUNPATH, so point those entries at the venv's unversioned
devel symlinks (idempotent ln -sfn). Without this the unit crash-looped for
the full 2400s health gate on first deploy.
Operator call 2026-09-19: switch the iGPU endpoint from
halogen-flash-server to jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the CIRU
v4.4.1 runtime with the QSA window hotfix (host build against the private
ROCm 10 SDK), one stream of the full 262144 trained window. The service
leaves the quadlet shape for a root unit (the same shape as frank-small and
frank-medium: a private-SDK host binary has no image to install and no
generated unit to dodge); the halogen quadlet is archived, the deploy
handler retires it on the host, gates on 7770, and its stray-run sweep
matches the model path so it cannot take frank-small down.
Operator call 2026-09-19: switch the target to
orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 (gated repo; the fetch script now
requires an HF token and verifies sha256, not just size), take the drafter to
the Q4_K_M rung of z-lab/Qwen3.8-27B-DFlash2-GGUF, cut the pool to two
163840-token conversations (327680, entirely inside the trained 262144
window), and balance the tensors across the cards (32:33) in place of the
35:30 x16-lean. The in-file MTP head stays one SPEC_TYPE change away.
The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
int8_per_token_head: page-size unification fails (scale tensor breaks
the 256/512 head-dim ratio). turboquant_k8v4: backend validation fails
(no mm-prefix attention, which Gemma4's vision/audio path requires).
fp8: hardware-dead on sm_86. bf16 it is; history documented in
frank-small.env, seam pins auto + no backend pin.
The VLLM_ATTENTION_BACKEND env var does not exist in this vLLM dev
build (warns 'Unknown vLLM environment variable' and is ignored), so
the pin landed nowhere and Gemma4's hook force-selected TRITON_ATTN
again. Pass --attention-backend TURBOQUANT as a CLI flag instead.
Gemma4's config hook force-selects TRITON_ATTN for the mixed 256/512
head dims whenever no backend is selected, and TRITON_ATTN rejects the
turboquant KV dtype (measured: 'kv_cache_dtype not supported', crash
loop). The TURBOQUANT backend accepts any head size and the
turboquant_* dtypes, and the Gemma4 MTP speculator carries the
target's selected backend through to the draft layers.
int8_per_token_head keeps a separate fp32 scale tensor per (token,
head); Gemma4's hybrid head dims (SWA 256 / full 512) make the
per-token page bytes 4160 vs 8256, not an integer ratio, so
unify_kv_cache_spec_page_size raises NotImplementedError — nine
crash-loop restarts measured. TurboQuant k8v4 packs its block scales
inside the tensor, keeps the 2:1 page ratio, and has a dedicated
backend with an explicit Ampere path. 289,647 -> ~772k-token pool.
fp8 KV is hardware-dead on the 3090 (sm_86, no cvt.e4m3x2), so int8 is
the cheapest quant the TRITON_ATTN backend can run: pure ALU dequant,
same backend, no fp8 tensor cores needed. 289,647 -> ~579k-token pool,
two -> four full-context sessions. Quality verified against the bf16
baseline (needle-in-haystack at 213k + short battery) before the swap.
podman 5.7.0's v1 quadlet generator rejects both ContainerEntrypoint
and Entry; the key is Entrypoint= (pkg/systemd/quadlet/quadlet.go
KeyEntrypoint, mapped to --entrypoint).
The podman 5.x v1 quadlet generator on dankclaw rejects
ContainerEntrypoint with 'unsupported key in group Container', so the
unit was never generated and the deploy died at restart with 'Unit
frank-small.service not found'. Entry= is the v1 spelling.
Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
top docs index updated; old runbook moved to retired
--cache-ram caps the total bytes of saved prompt states in HOST RAM, and
alloc() refuses any single state larger than the whole cap. That refusal was
firing in production: 137 times in the two days from 2026-09-16, at 11.0-11.4
GiB per state, so every conversation past ~170k tokens was uncacheable and
re-prefilled from scratch on its next turn.
The state law at this shape is 68 KiB/token (16 full-attention layers x 4 kv
heads x 256 dim x 2 B x 2 for K/V, plus the MTP draft context's 4096 B), so
8192 MiB held ~120k tokens and 16384 holds ~240k: a FULL 196608-token slot
state (~12.8 GiB) now fits with ~3 GiB spare.
Host RAM only, allocated on demand and never pinned, so the VRAM fit is
untouched. New seam: the cap must clear one full slot's state and stay under
half the host's RAM (it fails on the old 8192 and passes on 16384). Also
corrects the adjacent claim that 0 is the build's default for
--ctx-checkpoints -- this binary's --help says 32.
Isolated-port (:7790) 2k-token decode with AVG rocm-smi util over the
request (peak util lies: a layer split touches both cards every token):
35:30 draft ROCm1 31.52 t/s util0 41% util1 98% <- deployed
36:29 draft ROCm1 31.49 t/s util0 43% util1 98%
35:30 draft ROCm0 31.52 t/s util0 42% util1 98%
36:29 draft ROCm0 31.48 t/s util0 43% util1 98%
The x1 card (ROCm1) stays the 98% bottleneck in every configuration.
Moving the MTP drafter to the x16 card does not flip the balance: the
per-step cost is the verify phase (both cards run the full target) plus
the logits, which are pinned to the card holding the last layers. A
target-layer move (36:29) buys 2 points on the x16 card and erodes its
VRAM margin (fit free 0.45 GiB) for no throughput change. The x1 card is
compute-bound, not PCIe-bound (~5.2 MB activation per ubatch on the link),
so saturating it costs no tokens. 35:30 is the balance; no split change.
A pending entry for frank-default-sightsound (FLM NPU endpoint retired
2026-09-11) survived in the dankclaw state store and made every --pending
run exit 2 at the dispatcher before the Playwright e2e gate could
execute. The pending pass now drops entries the target no longer allows
into ~/.AGENTTRASH and keeps going. Behavioral seam test added.
- services.json: frank-large/medium/small move from admin-only to the user
page with current descriptions (halogen w4b iGPU; 35:30 layer-split MTP
27B on the R9700s; Gemma4-12B QAT vision path on the 3090) and docs
links to /docs#models
- docs.html: static user documentation (access chain, model guide, media
how-tos, troubleshooting); served at /docs via the post-auth homepage
listener; shipped by deploy_homepage; footer + index link to it
- tests: pages seam asserts the docs contract (page, link, Caddy route,
deployer copy)
- vault: Homepage.md synced (docs landing shipped, cards reclassified)
The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s). With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card. Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.
signal-cli 0.14.7 parses an all-digit -a value as a phone number (the
numeric account name never resolves), and the message is a -m option,
not a positional (a positional is parsed as a second recipient).
Verified live: corrected invocation returns type SUCCESS.
Every webhook died on 'cannot access local variable _last_sent' because
the prune comprehension rebound the module global inside the function.
Prune in place instead.
The bridge-subnet-only pair dropped the host's own health probe
(source 192.168.0.143/127.0.0.1) and failed a healthy deploy. The
deployer now installs the full set: ACCEPT 127.0.0.1, ACCEPT the host
IP, ACCEPT 10.89.0.0/16, then DROP.
169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
- runbook: identity now 4x262144, audio verified, throughput table with
the OOM and negative-result rows, deployed flags updated to reality.
- AGENTS.md frank-small paragraph matches the deployed shape.
- log.md: the sweep entry.
- scripts/frank-small.env: my earlier line-range edit had clobbered
NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The
live service was unaffected because the script's defaults match the
declared values exactly, but the file contract is restored.
Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved.
Swept on the 3090 at the deployed 4x262144 geometry, salted API probes:
- ubatch 512: pp@254k ~918 t/s, tg@250k 90-204 (noisy)
- ubatch 1024: pp@254k 962 t/s (+4.8%), tg@250k 257-302
- ubatch 2048: OOM — the MTP draft context needs ~3.3 GiB of pp compute
buffer against 2.2 GiB headroom; do not try.
- draft depth 4 + poll 100 decoded WORSE at 250k (230-257) than 3/50:
the longer verification round re-reads more long-ctx attention than the
extra accepted token earns. Kept 3/50, now with the evidence attached.
Idle VRAM 23,300 MiB; all ubatch-1024 variants idled identically
(compute buffers reserved at max graph size), so 8192 buys the deepest
prefill window for free over 4096.
The 393216/34:31/MTP deploy measured 30.78 GiB on ROCm1 against the
fit's 28.32 — the MTP draft buffers (one nextn layer's KV, 4096 B/pos,
plus draft buffers) came in 2.46 GiB above the estimate. Recalibrate
the seam's MTP constant to the measured 4.20 GiB total charge on card
1 and drop the floor 1.5 -> 1.0 GiB for this shape (ROCm1's 1.08 GiB
headroom is real; 35:30/36:29 fail card 0's floor at this pool, so
34:31 is the only split that fits 2x196k).
AGENTS.md and the runbook move to the MTP drafter: 34:31, 393216 pool
(196608 per conversation), the DFlash2 sidecar documented as a
one-line revert, and a dated part-2 section with the fit/measured
table and the recalibration.
Both were implicit build defaults (3 and 50); the throughput sweep needs
to move them without editing the exec line each time. Defaults preserve
current behavior exactly. Additive only.
DFlash2's sidecar (1.92 GiB + buffers on ROCm1) kept the 393216 pool off the table; the MTP head lives in the target file, so its cost is one nextn layer's KV (4096 B/pos) plus small buffers. Fit: ROCm0 29.77 / ROCm1 28.32 GiB, over the floor, 34:31 the balance point at this pool. n_max 3 is the measured MTP winner. The seam's fit is now drafter-aware, the slack is the deployed integer gap (1483 MiB), the ctx cap 393216 with the rope caveat (top 131k extrapolates on the base rope).
CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained
window. Measured fit arithmetic on the 3090: 1 full-context slot idles
at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed.
A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool
allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit
on purpose: fit maximises the TOTAL pool and would divide a smaller
window across slots at parallel>1. The slot-grid seam now pins both the
full-window division and the budget inequality so a future capacity
bump fails pre-deploy, not in the boot allocation.
Audio verified while measuring: the mmproj's audio encoder transcribes
input_audio parts verbatim (JFK WAV sample). Header comments updated.
AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no
longer why), and the runbook index. New frank-small-gemma4-12b runbook:
payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the
MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and
the fit-probe error that looks like a fault at boot. The Qwen runbook and
both indexes carry the retirement; work log updated.
AGENTS.md topology row and section, and the runbook: the pool is 327680
positions (163840 per conversation over two slots), the split is 36:29 -- one
layer past the 34:31 balance at this context, on purpose, for headroom on the
drafter card (fit 29.19/26.04, measured 28.40/26.66 GiB at load, 3.46/5.20
headroom) -- and vision is on via the explicit --mmproj flag, because this
build has no autoload (the first deploy came up text-only and 500'd image
requests). The rope caveat is written down where the cap lives: the GGUF has
no scaling keys, so the top 65k of the pool extrapolates on the base rope.
The Qwen3.8-27B IQ3_S file carried its own MTP head; Gemma4's ships as a
separate assistant file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant,
4 nextn layers) that loads via --model-draft and shares the target's last
layers' KV. The draftless ngram-mod matcher chains behind it in the same
--spec-type value — two context-backed drafters cannot share one process,
so this chain is the only shape that carries both.
KV is q8_0 (16 KiB/token/layer vs 32 at f16) and CTX_SIZE is unset so
--fit sizes the pool against what the 7.4 GiB of payloads leave on the
24 GiB card. build-cuda was rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON,
which quantized KV on CUDA requires. The three seams that pinned the Qwen
contract (MTP-in-file, the Qwen mmproj name, the f16 KV arithmetic) now
pin the Gemma4 contract instead, and the deploy handler's presence check
points at the new payload.
The 36:29/327680/vision deploy came up healthy and text-only: image requests
answered 500 'image input is not supported - hint: ... provide the mmproj'
with no mmproj line in the load log, so llama-server fb27a525 does not
autoload mmproj*.gguf from the model directory. Name the file, the way
frank-small's serve script already does, and pin the flag in the seam.
The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV,
and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point,
computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82
GiB free, both above the 1.5 GiB floor.
36:29 is one layer over the 35:30 balance on purpose: at this context the
balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0
to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so
the seam's slack is that gap rounded up (3225 MiB) with the trade written down,
and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries
no scaling; positions past the trained 262144 extrapolate on the base rope).
Vision: NO_MMPROJ=0, the serve script now refuses to start without the
projector when vision is on (a missing file is a deploy error, not a silent
text-only server), the download script fetches and byte-verifies it, and the
deploy handler's preflight checks for it. The projector is already on the host,
byte- and sha256-verified against the Hub.
6.44 / 6.90 is the state straight after load; after the service has served
traffic it sits at 6.23 / 6.64, with and without requests in flight. Both are
~5 GiB above the floor, but the runbook should not imply a load-dependent
difference that the two readings do not actually show.
New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per
cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full
batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and
the limitations — including the one that cost me two runs, that amdgpu does not
return ~26 GiB at once, so a reload after a kill fails with "unable to allocate
ROCm0 buffer" and reads exactly like a configuration that does not fit.
AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the
10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead
of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at
the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the
frank-medium section now say 35:30 and batch 6144x512.
Runbook: deployed flags, the split history with the ubatch-512 table, the fault
note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent
requests).
Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1
ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser
premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not
imply Routing+ on this board (00:08.3 is CPU-side and Routing-).
log.md: the round trip recorded, newest first.
The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell
grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau:
ubatch 256 384 512 768 1024 1280 1536 1792 2048
tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078
The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out
1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the
shape, not noise. The logical batch is throughput-neutral (2048..8192 all
within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized
by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell.
So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB
more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2):
36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed)
36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone)
35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed
The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1
holds the output layers and the drafter's graphs -- so it shifted the balance
toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is
one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split
can be tuned finer than a layer, which is why 34,31 is not better.
Re-fitted the seam with both ubatch coefficients measured at this shape and
re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to
0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 ->
0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB.
Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the
shape it actually cares about -- two shares that add up to the model's 65
blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer
breaks a test about parallelism. The numeric contract lives in the seam test.
pcie-link-test.cu (CUDA, pinned buffers, cudaEvent timing, both services stopped):
pinned H2D 6.72 GB/s and D2H 6.60 GB/s, i.e. 85 % of the Gen4 x4 ceiling. Async
is no better than sync and pageable sits within 1 % of pinned, so these copies are
link-limited rather than staging-limited — the usual "pageable costs 2x" does not
apply on this host. A 3 MiB slab costs 474 us, essentially 3 MiB / 6.72 GB/s plus
~27 us of launch.
That settles the expert-streaming question quantitatively: 1.70 GiB of active
experts per token (A3B @ Q4_K_M, 75 GiB expert mass) is 272 ms of link time ⇒
3.7 tok/s, 4.6 with a fifth of the experts resident. The same bytes computed in
host RAM at 256 GB/s cost 7.1 ms — the link is 31x slower than the memory behind
it, which is why llama.cpp's -ot exps=CPU path measured 20.46 tok/s while an
engine that streams loses by 3-5x. The port under the card is x8-capable and the
card is x16-capable, so the x4 is the adapter; doubling it still leaves streaming
well behind.
Also records FreeToken's own constraint: it keeps experts in host RAM with an LRU
GPU cache (exactly this streaming shape) and loads only modelopt block-FP8
safetensors for qwen3_5_moe, so Q4_K_M GGUF is unloadable there regardless.
The harness is in the repo now, so the runbook says how to run it and what the
tool will not tell you:
- / splits devices within one configuration and , splits sweep values, so a
legitimate --tensor-split 36,29 would have benched one card at a time.
- there is no -c: n_ctx = n_prompt + n_gen + n_depth, so no bench can decide
whether a configuration's buffers fit beside the deployed 262144 KV.
- a test that cannot allocate exit(1)s the process, hence one invocation per
cell.
- a shell's memlock limit (8 MiB here) is not the unit's (infinity), which
fails the load for the wrong reason.
- no --spec-* and no --parallel: decode figures are drafterless and slots are
never exercised; -d is what puts KV in front of the measurement at all.
Operator instruction: purge the cache so no run can serve a stale payload. The two
live payloads were cleared and then re-fetched rather than left absent — deleting
them without a re-fetch means neither endpoint starts.
halogen's re-fetch proved the point: its sidecars had been stale *in place*
(overlay.hgn 2,477,677,120 -> 2,572,466,560 bytes, overlay-speed.hgn
2,383,306,048 -> 2,478,095,488) while the 115.55 GiB checkpoint returned
byte-identical. frank-small's 27B and mmproj re-fetched to Hub-matching hashes,
unchanged.
Trap recorded: podman refuses a bind mount whose source directory is missing
(statfs ... no such file or directory), so frank-large needs /models to exist
*empty* for its first-start fetch to run.
Q4_K_S against the recorded Q4_K_M numbers: +2.1/+1.6/+2.6 % (iGPU-only pp512 /
pp16384 / tg64), +2.7 % tg32@64k, +4.1 %/+2.0 % for the two layer-split configs,
~0 on both pp512@64k, host RAM bottoming at 10.0 GiB against ~5 GiB. A memory
lever with a small speed dividend. IQ4_XS (97,473,155,360 bytes, sha256
b4787265...d0574f9d) is pulled and verified for the next comparison.
Also repairs an AGENTS.md edit that landed in the wrong section: the peer
session's commits shifted that file between my read and my write, so the
frank-large text briefly split the Model Config section.
`local b=$1 ub=$2 dir="$OUT/b${b}_ub${ub}"` expands every argument of the builtin
before the builtin assigns any of them, so $b was unbound and set -u aborted the run
four seconds in - after the service was already stopped. Two statements instead.
Found by running it under systemd-run: it exited non-zero with no cells run, which is
exactly what the per-cell TSV is for.
Question it answers: within the headroom the deployment has, how large may the logical
batch and the micro-batch be before the activation buffers stop fitting — and what does
each size buy in prefill?
- One llama-bench invocation per cell. llama-bench exit(1)s the moment a test cannot run,
so a single infeasible ubatch would otherwise take every later cell with it, and
infeasibility is one of the things being looked for.
- A VRAM sampler per cell, covering the load. Batch/ubatch sizes the activation buffers,
so a cell that looks fast because its allocation is smaller than intended has to be
distinguishable from one that is genuinely fast.
- The service is stopped for the sweep and started again from a trap, because a bench
beside a live service starves the same GPU.
- Run it from a transient unit: the deploy's unit sets LimitMEMLOCK=infinity for the ROCm
staging buffers and a shell on these hosts only gets 8 MiB.
- llama-bench separates -ts/-dev with SLASHES and has no -c, --parallel or --spec-*: the
context is n_prompt + n_gen + n_depth, and the drafter cannot take part.
Checking whether a download was still running with pgrep -a curl printed the
whole argv, Authorization header included, into the session transcript. The
token is not in the repo and nothing was committed, but the practice is wrong:
secrets belong in a -K config file or the environment, not in argv, and no
pgrep -a should ever target a command carrying an auth header.
The note described DSpark as the deployed drafter and 40:25 as the split; both
were true for a few hours. It now leads with DFlash2 at 36:29 and keeps the
excursion as the record it is:
- the drafter comparison table with DSpark's REAL size (1,455,376,576, not the
1,359,284,737 that went into the fetch script), and why it lost: no sliding
window, so its draft KV spanned the whole context.
- the split's two moves as one story: the batch pushed it to 40:25, the drafter
swap pulled it back to 36:29, with both measurement tables.
- the layer-size law (0.43 GiB DeltaNet, 1.43 GiB attention) as the reason the
balance is measured and not interpolated.
- the verified-behaviour section now carries the LIVE numbers: 4.71 / 4.91 GiB
free (4.43 / 4.61 under load, 0.18 GiB gap), the DFlash2 init lines, and
draft acceptance from both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0).
- the fault note scopes its own evidence: 39:26 was loaded, not exercised.
DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.
The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:
39,26 2.42 / 7.20 37,28 4.28 / 5.34 35,30 5.14 / 4.48
38,27 3.85 / 5.77 36,29 4.71 / 4.91 34,31 6.57 / 3.05
36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.
Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
--verify-only fail on a payload that was in fact correct. Repointed at z-lab
DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
point; it now passes 36:29 and fails every other split measured, including
1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
are gone; the build's defaults apply.
The note still described DFlash2, one slot at 262144 and a 1:1 split -- all of
which the day's work replaced. It now carries:
- the DSpark swap with the table that matters (1.27 vs 1.92 GiB, block_size 7 vs
8 so n_max 6 vs 7, a Markov head instead of the candidate selector, and NO
sliding window) and the rule behind the sizing: the draft context is built
with n_ctx = llama_n_ctx(ctx_tgt), so a drafter without a window caches the
whole target context -- 5.0 GiB here, against DFlash2's ~40 MiB.
- the failure that rule produces when ignored: load succeeds, /health answers,
and the first draft decode dies with an OOM at 1:1.
- why the split moved a second time: tripling the batch costs ROCm1 about twice
what it costs ROCm0, with the 38,27 -> 41,24 -> 40,25 measurement table.
- the deployed flags, the 2-slot shape (n_slots = 2, n_ctx_slot = 131072), the
DSpark init lines, and balanced 29.98 / 29.98 GiB cards.
- honest scoping on the older measurements: the throughput, drafter-worth and
ngram tables were taken on DFlash2, and the agent-loop numbers on one slot.
- the 39:26 page-fault signature as a thing to watch, marked as NOT an OOM.
The operator's revised spec: batch x3 (6144/1536), parallel 2, a 1.5 GiB per-card
floor, and the context maximised inside it.
Measured at 262144 with the old 38:27 split, 3x batch was brutally asymmetric:
ROCm1 (output layers + the drafter's graphs) paid ~3.3 GiB where ROCm0 paid
~1.7, leaving the draft-carrying card at 0.08 GiB free against ROCm0's 3.78.
One -ts step is worth ~1.5 GiB at this batch, so the balance is a measurement:
41:24 overshot (1.47 / 2.38), 40:25 landed at 1.92 / 1.96 GiB free.
That balance is also what maximises the context -- an unbalanced split would have
to buy its headroom out of CTX_SIZE, and CTX_SIZE is already at the model's
trained window (262144), now split over two slots of 131072. Verified under two
concurrent 1832-token prompts: 200/200, 1.87 / 1.86 GiB free, n_slots = 2,
n_ctx_slot = 131072, tool calls intact, and DSpark drafting on BOTH slots
(acceptance 0.38/0.22 and 0.52/0.43).
8192/2048 does not fit at this context at all: its 2020 MiB pp buffer fails to
allocate on ROCm1, and 4096/1024 fits but leaves 1.11 GiB -- both recorded in
the env's sizing table.
The unit seam now models both cards (share + draft + ubatch-scaled buffers,
calibrated to the rocm-smi figures to within 0.002%), asserts the 1.5 GiB floor
on each, and asserts the two cards stay within 0.5 GiB of each other. Both
assertions were confirmed to FAIL on a deliberately bad config (ubatch 8192, and
a 1:1 split) rather than pass by construction.
Correcting a claim I made from a bad query: Q4_K_S does exist for this finetune.
`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` ships the large quants as
`.partNofM` fragments — Q4_K_M 37.00+37.00+36.97 = 110.97 GiB (which is how the
deployed file was assembled) and Q4_K_S 35.00+35.00+34.09 = 104.09 GiB — while
the smaller rungs are single files. A tree listing filtered on `.gguf` makes
Q4_K_S invisible, which is exactly how I concluded it did not exist.
Also records the fetch trap that hid it for a moment: `curl -o FILE URL` exits 0
on 401/403, so a gated repo produced three 153-byte "shards" with identical
sha256 values and a cheerful DOWNLOAD_DONE in 20 s for 104 GiB. -f plus the
Hub's declared size plus the LFS sha256 is the standard the fleet's own
download-frank-medium-gguf.sh already meets.
The 38:27 skew folded into the DSpark swap already does what step 3 asks: the
draft-carrying card sits 2.08 GiB above the other (5.49 / 3.41 GiB free) instead
of the ~9 GiB a 1:1 split would leave.
Chasing an exact balance is a mistake worth recording: 39:26 measured tighter
(3.86 / 4.69 GiB free) but that server died on its eighth request with SDMA0
UTCL2 page faults in the amdgpu driver -- "ROCm error: an illegal memory access
was encountered" -> ggml_abort, in the cross-card boundary copy that is the
layer split's hot path. 38:27 has since served the whole e2e gate and 12 varied
requests with no fault, so the residual 2.08 GiB gap is the safe side of this
trade. Comment-only: no restart needed.
Step 2 of the operator's six. --ctx-checkpoints 8 held per-slot KV snapshots;
0 is what llama.cpp ships, and the fleet's clients open conversations fresh.
DSpark declares no sliding window, so its draft context -- built with
n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8
kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where
DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at
33.9 GB of 34.2 GB and the first draft graph capture died inside
common_speculative_impl_draft_dflash::process with "ROCm error: out of memory"
-- after the model had loaded and /health had answered, which is why the deploy
looked like a crash-loop rather than a sizing failure.
Handing three target layers to ROCm0 restores headroom on both cards at the full
window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free
(was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam
now models the busier card as a skewed share plus the drafter's full-context KV,
calibrated to the rocm-smi figure.
AGENTS.md gains the durable lesson: read sliding_window out of the drafter's
GGUF header before sizing a spec-decode deployment.
The log mtimes plus each run's own duration pin it: the first 'invalid internal
status' in the user journal is 18:58:30, while the F bench (launched 18:54:31)
was still loading, and the preceding llama-cli placement probe had just spilled
3.2 GB of '> ' prompts. The note had said 'after the experiment'; that was wrong
and is now recorded with the timestamp it can be checked against.
I had written 'memory pressure from the 116 GiB iGPU runs' as the cause of the
pause-process death. journalctl -k records no OOM kill for that window and the
kernel says nothing else about it, so the claim was a guess wearing a finding's
clothes — in AGENTS.md, the x4 note, and the log. All three now state the
trigger is not established, and the note's 'co-tenants' patience' bullet no
longer asserts that causality either. AGENTS.md also gains the real lesson:
journalctl -k is the check, and an unknown cause does not need a story.
DSpark (Q8_0, 1.27 GiB) replaces z-lab DFlash2 (1.92 GiB): same 5-layer
block-diffusion shape, Markov head instead of the candidate selector, and
block_size 7, so the draft depth drops 7 -> 6 (llama.cpp asserts
n_max <= block_size - 1).
The serve script's dspark branch never passed --spec-draft-model (only the
dflash branch did), so dspark was not deployable as written; the model file
gate now covers both types. Download script, env and the unit seam track the
new file and its exact byte count.
The ledger number in the note was pre-reboot. Same model, same pool, no clamp —
but the host was measurably more fragmented when it came back, which is the sort
of thing that decides whether the next co-tenant fits.
Asked whether Vulkan on both devices unlocks the split. It does not: the
per-tensor design aborts byte-identically, because the gate is not the driver
mix but `--override-tensor` — `has_tensor_overrides()` is a required term of
the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place
in the tree that reads it), and that flag sets `n_copies` in
`ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts.
Chasing it exposed a harness artifact that invalidated the earlier conclusion:
llama-bench splits a device LIST with `/` and SWEEP values with `,`, so
`-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15`
two sweep values, i.e. device 0 takes everything — which is where the bogus OOM
came from). With the correct syntax two-device splits work in both flavours:
pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090)
cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB)
iGPU alone 25.77 tok/s @64k, pp512 231.8
+7 % decode, +16 % prefill — real but not a regime change, and it costs the
3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's
premise still fails: whole layers carry their experts (1.62 GiB each), the 12
attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the
main device, so KV cannot be consolidated on the 3090.
Also records the evening's incident (a dead rootless podman pause process took
all container management down; `podman system migrate` panics in 5.7.0; reboot
repaired it) and the netavark host-address lesson from the hermes 502.
hermes-webui is Network=host, so 8787 is a bare host-netns bind, and netavark
DNATs only *published* ports. From Caddy's bridge netns 192.168.0.143:8787 and
10.89.1.1:8787 are both refused, while host.containers.internal:8787
(169.254.1.2) answers 200 — measured live 2026-09-16 from inside the Caddy
container. The dashboard's published 9119 answers on that same LAN IP, which is
what made the old upstream look sound.
The vhost comment asserted "192.168.0.143:8787 is open" from that bridge; it is
not, so the claim is replaced with the measurement, and the hermes-webui quadlet
comment carries the same correction. The seam guarding this invariant accepted
any numeric upstream, so the LAN IP passed it; it now pins the measured-working
form and fails on the LAN IP (teeth verified by substitution).
The design we sketched (trunk + KV on the 3090, MoE experts on the iGPU) is not
expressible in mainline llama.cpp. Per-tensor cross-backend placement aborts in
the scheduler -- "pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer
(Vulkan1) that cannot run the operation (NONE)" -- and both per-layer
cross-backend forms fail at load. CUDA + Vulkan cannot serve one model; the CPU
is the only fallback side that works, which is exactly why -ot exps=CPU runs.
What the same build does instead is the surprise: iGPU-only (Vulkan/RADV, PLE in
page cache) gives 32.64 tok/s shallow, 25.77 tok/s at 64k depth and 456.7 tok/s
prefill, with the 3090 idle at 0 %. That matches halogen's deployed ~25 tok/s --
but commits 116 of the host's 121 GiB against halogen's 95.4 held with 14.7
spare, so it is not deployable beside the co-tenants. Throughput parity, memory
bankruptcy: the fleet's two-endpoint split stands, now on evidence rather than
preference.
New page LLM/x4-pair-experiment.md carries the tensor inventory, the results
table, the two scheduler limits, reproduction commands, and the traps found on
the way. Both index pages also stop calling halogen retired -- the ik swap's
stale claims were still sitting in them.
The fleet now runs draft-dflash,ngram-mod, so the rule needs to be written down
where the next agent will look: draftless types (ngram family) are added to the
spec configuration unconditionally, while draft-mtp/dflash/dspark/eagle3 all
require a draft CONTEXT -- therefore one context-backed drafter plus any number
of ngram types is legal, and two context-backed ones never are ('draft-mtp,draft-dflash'
dies at load with 'model doesn't contain MTP layers'). Plus the operational
trap: ngram-mod prints no startup banner, so a working chain looks unengaged in
the journal.
The port gate was written for the cutover, when the vLLM proxy held 7770. Once
frank-medium itself owns the port, a re-deploy waits 90s for a socket that only
its own predecessor is holding and fails: 'FAIL: port 7770 is still held after
90s', with the old llama-server still serving scrapes. Stop the unit first, then
sweep a stray manual llama-server (the shape a benchmark harness leaves behind);
pkill -x, never -f, so the deploying shell cannot match itself.
ngram-mod is draftless (added to the spec config vector unconditionally, while
every model-backed drafter needs ctx_dft), so unlike MTP it can share the
process with DFlash2. Chained at the fleet's 48/64 window, over three shapes:
prose 1k/512 dflash 43.77 | ngram 17.72 (0 drafts) | chain 43.65
echo 1.5k/400 dflash 76.82 | ngram 102.50 | chain 200.58
structured/400 dflash 91.08 | ngram 17.80 (0 drafts) | chain 91.22
identity dflash 70.37 | ngram 17.77 | chain 70.40
On prose the chain is byte-identical to the drafter alone (same 119 drafts/49
accepts) -- that is what makes it a safe default -- and on prompt-echo traffic
it is 2.6x. VRAM is unchanged: the matcher is a 64 MiB host-RAM hash table.
The window sweep says keep the long one: 48/64 beats 16/32 (161.0) and 8/16
(113.8) on echo, and shortening it costs prose (43.9 -> 36.0) because the
matcher then proposes on ordinary fragments and the rejections still have to be
verified. That is the 'ngram hurts' result this fleet hit twice before.
Two documentation traps recorded with it: ngram-mod prints NO startup banner in
this build (its drafts are only visible behaviourally), and the harness's
'structured' shape is a negative control because its lines differ.
Two real defects from patching a rewritten file by line number: the CUT that
removed a duplicate parse_configs also took write_env with it (NameError after
the first configuration, with the unit already stopped), and the restore step
lived after the loop, so a crash left frank-medium DOWN. The endpoint is the
reason the harness exists; it now restores itself in a finally and shouts if the
service does not come back healthy.
ngram-mod's match window (n_min/n_max) decides which traffic it can see at all
-- the fleet's 48/64 preset is tuned for long verbatim echoes and finds nothing
in a repetitive-but-short output. Config entries now take |KEY=VALUE overrides,
so the window can be swept in the same run as the configurations it applies to.
The first run asked for a verbatim repeat with thinking ON, so the model copied
the passage inside reasoning_content, content came back empty for all three
configurations, and the cross-configuration identity check compared '' to '' --
a pass that proves nothing. echo and structured now disable thinking via
chat_template_kwargs, the echo row carries a copy_ratio (share of answer lines
that really are passage lines), and an empty answer in any shape is a loud
warning rather than a silent pass.
'structured' is new and answers the case the fleet actually runs on: repetitive
OUTPUT with no prompt overlap (record lists, tables, config rewrites), which is
where an n-gram matcher should pay and an echo-shaped test cannot tell.
A relative serve-script path plus a cd into its own directory double-counts the
prefix, so every configuration failed to start ('scripts/serve-frank-medium.sh:
No such file or directory'). The safety net did its job -- the unit was restored
-- but the run measured nothing.
Unlike MTP, ngram-mod needs no draft context: common/speculative.cpp adds the
ngram types unconditionally while the model-backed drafters are gated on
ctx_dft != nullptr, so 'draft-dflash,ngram-mod' is structurally allowed (and the
config vector's fixed order -- ngram first, then the drafters -- makes CLI order
irrelevant).
The harness grows the two things this question needs: --configs, and an ECHO
shape. ngram-mod can only ever match text that is present in the prompt, so
measuring it on prose calls it useless and measuring DFlash on echo calls it
wasteful; both shapes now run, and the greedy text of each is compared across
configurations because speculation may not change answers.
Four configurations of the same binary, identical prompts at temperature 0, the
unit stopped between runs, on the deployed payload:
none 17.7 tok/s card1 26.07 GB
draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all
four produced IDENTICAL greedy text, so the 'lossless verification' claim in the
runbook is now a check rather than an assumption. MTP is not the cheaper option
either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper
drafting loses (n=7 accepts a third of its proposals, n=3 more than half).
The two CANNOT be chained, and the failure is structural rather than tunable:
'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested
but model doesn't contain MTP layers', because a chain has ONE draft context and
MTP builds against whatever model that context holds -- here the DFlash2 file.
The serve script therefore passes --spec-draft-model in the dflash branch ONLY,
and the seam now asserts exactly that (the wiring whose violation is this
error), plus that the deployment stays on the measured winner.
scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking
policy wants the runtime's own harness for this, and the numbers above were
otherwise unreproducible one-offs.
The loader prints no per-device ledger at this verbosity, so the claim is
measured rather than read: +131072 tokens of context added 4.83 GB to one card
and 4.84 GB to the other (both carry their own attention layers' KV), and
neither card's total usage could hold the 31.46 GB file. Worth writing down
because the retired shape WAS two full copies — that is what a replica is, and
it is why the 29.8 GB FP8 checkpoint could never run there.
The LLM index still opened with vllm-radiance as frank-medium's runtime and
drew the replica pair in its topology diagram; the cooperation page's thesis
(collectives are impossible, so the answer is hardware) now has to name the
path the fleet actually took: llama.cpp's layer split, which launches no RCCL
kernel on either card. That reframes the M.2 riser as a means to TENSOR
parallelism rather than a means to using both cards at all.
Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.
The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.
Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.
The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.
omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.
The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
'cp -r SRC DST/' with a missing DST creates DST as a copy of SRC and drops a
directory level, so rocblas/library/* landed as lib/rocblas/* and hipblaslt's
arch subdirectories landed one level high. Nothing complains at load time: the
engine reads 29 GiB of weights, answers /health, and dies on the first request
with
rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat:
No such file or directory for GPU arch : gfx1201
Declare the destination and copy contents, both trees.
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.
AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.
Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
The runtime for the next frank-medium shape is mainline llama.cpp at
fb27a525, built for gfx1201 against the ROCm 7.14 nightly the ROCmFPX build
already used on this host, with the runtime bundled next to the binary.
Two things that pin the ref: DFlash2 (ggml-org/llama.cpp#27342) and qwen35
MTP. Neither is in the ROCmFPX fork's tip (c49ebdb, 2026-08-22), and the
DFlash2 drafter cannot run without the first.
The fetch script pulls the target at UD-Q8_K_XL plus the DFlash2 draft at
Q8_0, resumable and byte-verified -- a truncated 31 GB GGUF is otherwise a
load-time crash.
A redeploy of a running frank-large holds 7770 itself, so waiting on the port
before stopping the unit would have made every subsequent deploy fail. Stop the
user unit first (a no-op on a first cutover), then wait, then install and start.
The 2026-09-16 halogen restore failed exactly here. The ik root unit stopped,
systemd logged "Deactivated successfully", pasta tried to bind 7770 about a
second later, reported "Address already in use", and the quadlet exited 126.
Only Restart=on-failure recovered it -- minutes later, which is not a deploy.
The handler now waits for every socket on the port to clear (listener, accepted
and TIME_WAIT) before installing and starting the quadlet, and fails loudly with
the socket table attached if it never does. The retirement also reports when the
unit file was already gone, because this host's `disable` deletes it.
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.
Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.
Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.
The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
New page: LLM/Dual-R9700-Cooperation.md — registered in the vault index, with
the measured evidence rather than the theory.
**The answer.** Card #2 sits behind the B650 chipset, whose bridges advertise
`AtomicOpsCap: Routing-` (read-only), so ROCr refuses to fill the hostcall
buffer RCCL's kernels declare and no RCCL op can dispatch on it. Two paths fix
it: an M.2→PCIe x4 riser into the free, CPU-attached M.2_2 slot (~$30; per
ASUS's spec page M.2_1 and M.2_2 are CPU-attached, PCIEX16_2 is chipset in x1
mode, and there is no x8/x8 bifurcation on this board), or a board whose CPU x16
bifurcates to x8/x8. Bandwidth is not the constraint: ≈10 MB per token-step ≈
1.4 ms at Gen4 x4.
**New measured results.** A bare 2-rank torch.distributed run: rank 0 (CPU-lane
card) passes broadcast/all_reduce/barrier; rank 1 (chipset card) fails all
three, and `send`/`recv` fails too — which kills pipeline parallelism as an
escape, since vLLM's PP handoff is exactly that primitive.
**Correction.** I previously said no software path exists. That overshot. One is
open and unverified: recompiling RCCL so its kernels stop declaring a hostcall
buffer (`-mprintf-kind=buffered` is the claim). Counter-evidence recorded: RCCL's
device source contains no live printf/assert/__assertfail, so the trigger is
unexplained, and the claim traces to search summaries, not a primary source. The
page carries the two-command test that would settle it. Also corrected: 0.9.3 is
radiance's version, not vLLM's — the base is vLLM 0.27.1 / torch 2.11.0 / ROCm
7.14, and its shipped RCCL exposes no `ncclDevComm*`, so torch's symmetric-memory
path cannot run.
**Rule added to the benchmarking policy**: it covers ANY GPU experiment, not just
benchmarks — including a bare 2-rank probe. Running one against the live service
on 2026-09-16 preceded the host going hard down (no ARP, no ICMP, power cycle
required); a hung AQL dispatch wedging the driver is the plausible mechanism.
The policy also now says how to tell off from wedged: `ip neigh` from a peer.
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
The board showed mariadb, valkey and Postgres as "down". None of them is down:
the checker runs on Dankclaw and can reach that host's loopback, its container
names, and published LAN addresses — while all three cards probe
container-internal database ports (192.168.0.235:3306, 127.0.0.1:6379/:5432).
They were red by construction, which is how a status board stops being read.
Publishing a database port on the LAN to make a badge green is not a trade this
fleet makes, so they are wip with the reason in their description, and the apps
that front them carry the status. Gluetun (WireGuard, UDP) was the pre-existing
example of the same call.
The rule is now written down in the Homepage doc: a card must be probeable from
the monitor's host or be wip. That is the same lesson as the Prometheus jobs
that scrape HTML pages and can never report up — an always-red signal teaches
the operator to ignore the surface it lives on.
Two gaps behind the same near-miss.
1. **No spec covered hermes.franksdank.cloud** — the one guarded vhost whose
upstream is a host address (`reverse_proxy 192.168.0.143:8787`) rather than a
container name, because its service is host-networked. That is how a
loopback-only bind could take the vhost down with a 502 and leave every gate
green. tests/e2e/specs/17-hermes-vhost.spec.ts asserts what a dead upstream
violates: no 5xx through the guard chain, no Caddy error page, and no app
bytes to an unauthenticated raw client. It does not assert SSO-silence, since
this app has its own password login.
2. **Specs are matched by number prefix, not globbed.** A spec file whose prefix
is in no project silently never runs — which is exactly what happened to the
new one until it was registered. The prefix is registered now, the comment in
playwright.config.ts says why, and tests/e2e_scaffold.bats fails if any
spec file matches no project.
The hermes-webui seam assertion is corrected too: it asserted the loopback bind
I wrongly imposed, and now asserts the real contract — the unit's bind must serve
the address the Caddyfile fronts it on.
I tightened HERMES_WEBUI_HOST to 127.0.0.1 on the reasoning that the compose
service had published loopback and the unit had widened it. That reasoning
missed the consumer: this unit is host-networked (it shares the CDP daemon's
loopback), so Caddy cannot reach it by container name and fronts the guarded
vhost hermes.franksdank.cloud with `reverse_proxy 192.168.0.143:8787` — recorded
in dankclaw/Caddyfile with the same explanation. With the app on loopback that
upstream is dead, so the vhost broke.
The wildcard bind is what serves BOTH consumers: Caddy on the LAN address and
the phone's SSH tunnel on the host's localhost. Controls are the auth chain in
front of the vhost plus the webui password. The unit comment now states this
instead of the incorrect 'as before' claim, and the near-miss is worth the
paragraph: the e2e gate stayed green because no spec covers that vhost.
Two things, found while checking whether the hermes trio still needed migrating.
1. **The phone-facing webui was exposed to the LAN.** Its unit runs with
Network=host, so there is no Podman port forwarder and the app's own bind is
the exposure. The compose service it replaced published
`127.0.0.1:8787:8787` (loopback); the unit bound `HERMES_WEBUI_HOST=0.0.0.0`,
and the LAN address answered 200 — password-gated, but on every interface,
which is not the documented "SSH tunnel is the sole ingress" posture. It now
binds 127.0.0.1, which serves the tunnel just as well (sshd dials
localhost:8787 from the host). The unit comment claiming "LAN-facing as
before" was wrong about "before", and is corrected.
2. **Dankclaw's compose.yaml is retired.** Auditing ownership showed every
running container on that host reports a PODMAN_SYSTEMD_UNIT label — the
hermes trio's units have owned their containers since 2026-09-12 — so the
three service blocks left in compose.yaml were dead weight that could only
create a duplicate container fighting a unit for a name. deploy_hermes was
already fully unit-based. The file is staged in .AGENTTRASH/; the planner
plans nothing for it, an unknown service on that target now errors instead of
silently no-oping through `podman-compose`, and DANKCLAW_COMPOSE_SERVICES is
empty.
The hermes-webui seam now reads the unit — it read compose, which is exactly how
the bind drifted while the test kept passing against a file nobody ran.
ComfyUI's page described a compose service and a systemd unit that shelled out
to `podman compose up`. It now documents the Quadlet, the keep-groups + udev
mechanism (verified inside the running container: torch sees Radeon 8060S), the
9,146-restart incident that forced the move, and the trap that caused it — a
stale container outlives the compose edit that should have replaced it.
The observability page gains the audit result: four jobs read "down" and three
of them always will, because they scrape HTML pages or a /metrics that does not
exist. The page names each error, says plainly that none is an outage, and
refuses the tempting fix (deleting the job makes an absent scrape look like a
healthy one) in favour of the remedy it already recommended: a blackbox exporter
in probe mode, and Gitea's metrics enabled or the job repointed.
ComfyUI had failed 9,146 times and had been down ~7 hours when this was found:
its container was created 2026-09-15 14:12, BEFORE the GPU declarations were
added to the compose service, and `restart: unless-stopped` kept restarting
that stale container — no /dev/kfd, no /dev/dri, no keep-groups — so it died
every time on `RuntimeError: No HIP GPUs are available`. Its "unit" was a
hand-written ~/.config/systemd/user/comfyui.service that shelled out to
`podman compose up -d`, which is not repo-tracked state and collides by name
with the Quadlet the generator produces.
- dankclaw/comfyui/quadlet/comfyui.container: image comfyui-rocm:gfx1151
(built from Containerfile.comfyui), /dev/kfd + /dev/dri, keep-groups,
the four comfyui-data volumes, loopback-only publish (an unguarded service
is internal-only here; the homepage card probes 127.0.0.1:8188), Restart=
always, WantedBy=default.target.
- the compose service is deleted — one owner per container name.
- deploy_comfyui builds the image, installs the unit, retires the compose-era
unit into ~/.AGENTTRASH, removes the stale container so the unit's own can
take the name, and waits on :8188.
- the healthcheck is a file, because the podman 5.x v1 generator splits
HealthCmd on spaces and an inline `python3 -c "..."` becomes a quoting
puzzle that fails silently.
The comfyui seam now reads the unit instead of compose for devices, command and
groups (and resolves the Containerfile from the deployer's declaration, not
from a directory listing).
The page said "all five are rootless Quadlet units on Dankclaw", which stopped
being true when Dankserver's node-exporter was restored to a unit. It now names
both hosts and both deployments, records why the second one happened (the same
compose-only reboot argument, a day and a half of missing host metrics, and a
wildcard bind), and states the exposure rule for a two-listener fleet with the
`ss` evidence that the wildcard is gone.
Dankserver's node-exporter has been dead since a reboot: `podman ps -a` showed
it `Exited (143)` and Prometheus' `node-dankserver` job has been scraping
nothing. It ran only from `dankserver/compose.yaml`, which is the failure mode
this repository's boot-persistence rule exists for (rootless Podman does not
restart bare containers after the stop that shutdown performs), and the compose
form bound the wildcard address — the unauthenticated-metrics-API-on-every-
interface posture already removed on Dankclaw.
- dankserver/observability/quadlet/node-exporter.container — mirrors the
Dankclaw unit, listening on exactly 127.0.0.1:9100 (host-local probes) and
192.168.0.235:9100 (Prometheus on Dankclaw), with Restart=always and
WantedBy=default.target.
- the compose service is deleted: one owner per container name.
- deploy_observability now installs from "$TARGET/observability/quadlet"; it
hardcoded the Dankclaw path, so a Dankserver unit could not be installed by
the shared dispatcher at all.
- planner: "node-exporter" joins the Dankserver service set, with a path rule
for dankserver/observability/**; the dispatcher's allowlist gains
dankserver:node-exporter.
Seams cover the silent halves: both listen addresses present and the wildcard
absent, the compose service gone, the allowlist entry present, and the planner
mapping the path to the service.
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
dankclaw/comfyui/Containerfile is referenced by nothing — compose builds from
dankclaw/Containerfile.comfyui, deploy-remote.sh copies that one, and the
ComfyUI doc names it — and its content is an older iteration (torch
2.9.1+rocm7.12.0rc1 against the deployed gfx1151 base). It also caused real
damage: the comfyui test's directory-listing selector picked non-comfyui files
and reported a red suite for sessions.
Staged in .AGENTTRASH/ rather than deleted, so it stays recoverable and visible
until Frank decides it can go for good. This is also why the earlier test fix
had to resolve the Containerfile from compose rather than from the filesystem:
the directory is not the contract.
This test failed on a clean tree for many sessions, and the working belief
became "test 73 is a known failure, ignore it" — which is exactly what a
permanently red suite teaches. The cause was a selector bug: `find
dankclaw -name 'Containerfile*' | head -1` returned whichever Containerfile the
directory listing happened to put first (dankclaw/alloy/Containerfile), so the
assertion read a file that never claimed to be ComfyUI's. It would equally have
passed on a wrong file.
The test now resolves the Containerfile the way the build does — build.context
joined with build.dockerfile, both read from dankclaw/compose.yaml — so it
checks the file the deploy actually builds (dankclaw/Containerfile.comfyui, the
one compose, deploy-remote.sh and the ComfyUI doc all name) and follows a future
rename instead of guessing.
Suite: ALL TESTS PASSED (310), no exceptions to explain away.
The card probing 192.168.0.235:7770 — frank-medium's endpoint since the
radiance cutover — was still called "frank-small", a service retired on
2026-09-15. The probe target was right; only the identity was wrong, which is
the worse half: a green card named after a service that no longer exists teaches
the wrong thing about the fleet. It is now frank-medium, described as what it
actually is (Qwen3.8-27B Uncensored MXFP4, vllm-radiance, two R9700s behind one
endpoint), and frank-large's description stops crediting the retired halogen
runtime for work ik_llama.cpp does.
The seam added to homepage_config.bats pins the cross-file contract that let
this drift: a card's probe target and its name must agree with the fleet, and a
retired service must not keep a card.
Both pages claimed there was no alerting; there are six rules now, so they say
which rules exist, that promtool and /api/v1/rules both confirm them, and — in
the same breath — that no rule reaches a person, because this stack has no
Alertmanager and no contact point. The dashboard's ALERTS panel is named as the
only place an alert becomes visible without opening Prometheus.
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):
LlmKvPoolSaturated kv pool > 90% for 10m — the 122,636-token wall
LlmRequestsQueueing waiting > 0 for 10m — MAXSEQS=4 reached
LlmPrefixCacheCollapse reuse < 20% for 15m with traffic — the thrash signature
LlmEndpointDown scrape target down 2m — both replicas gone
LlmLatencyHigh p95 end-to-end > 60s for 10m — what an operator feels
job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
dashboard, so the two cannot drift
Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.
Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
The page said Grafana has zero datasources and zero dashboards; that is now
false, so it says what exists (Prometheus datasource, LLM fleet dashboard), how
it was verified (the API — a provisioned dashboard does not show up in the
legacy dashboard table on this Grafana version, which is a trap worth recording)
and what is still missing (nothing alerted on, Loki unread, correlation manual).
The section heading drops "UNCONFIGURED" because it no longer is, entirely.
Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):
- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
container name, not the loopback publish, with a fixed uid so dashboards
reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
rate, e2e latency p95, and decode/prefill throughput, with an endpoint
variable covering both frank-medium and frank-large. Every series name was
checked against the live /metrics endpoint: this fleet already shipped one
comment naming a metric that does not exist, and a dashboard full of those
would be worse.
Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.
Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
The capacity work in this runbook was done with ad-hoc probes; the same signals
now have history in Prometheus, so the page says which series to watch and
admits the honest gap — this stack has no alerting, so a saturated pool shows up
on a graph after the fact rather than paging anyone.
vllm:prefix_cache_hit_rate does not exist in this build; the exported pair is
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give
the hit rate. A comment naming a series nobody can query is worse than no
comment.
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.
frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.
The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
Two more hypotheses, both measured on the same three-family workload (one
~80k-token family plus two ~50k-token families, each turn re-sending its
document):
1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt
sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too
(22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%"
is an average across every prompt that replica ever saw, not a verdict on
any one request, and the pinning policy is safe.
2. Serialising large requests does not fix the overload. Admission control
(at most one >16k-token request per card, small requests untouched) gave no
clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums
34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was
reverted like the placement change before it.
The arithmetic now closes: with three families on two cards, one card holds two
families and must carry 2 x 50k of cached prefix plus the 50k of the sequence
generating — 150k against 122,636 tokens. Those co-tenants evict each other
regardless of concurrency limits. The wall is capacity, and the runbook says so
with the numbers.
The obvious next lever for cache thrash was placement: send a new conversation
to the card carrying less context instead of alternating. Implemented it
(decayed prompt-token estimate per replica, continuations still pinned) and
A/B'd it against the counter on the workload it targets — one ~80k-token
family plus two ~50k-token families, each turn re-sending the same document:
counter heavy 13 turns median 25.4 s (23% cached)
medium 34.7 s / 45.2 s (14% / 15% cached)
balanced by load heavy 5 turns median 46.7 s (20% cached)
medium 23.4 s / 34.3 s (0% cached, small sample)
No improvement, so the change is reverted and routing.py keeps the counter.
The reason is arithmetic: a running request's live sequence and its cached
prefix draw on the same pool, so a card running two 50k-token generations has
already spent its 122,636 tokens before any cache entry is considered.
Placement chooses which card is over capacity, not whether one is.
Third hypothesis measured and rejected tonight (with SPEC depth kept and CHUNK
disproved); the runbook records all three so the next operator starts from
evidence rather than from these ideas looking obvious.
Rebooted dankserver unattended and verified without any manual step: all three
units active 59 s after boot, both replicas healthy ~3.5 min after boot, real
completion through :7770 — the failure mode this repo has been bitten by before
(a service that only ever ran from compose and stayed dark after a reboot).
Records why it works: [Install] WantedBy=default.target wired into the
generator's default.target wants, plus Linger=yes, plus a disk-backed compile
cache so a cold boot needs no warm-up beyond the model load.
A 25-minute soak with three concurrent large-context agent loops (each turn
re-sending a ~30k-token document, growing history) came back clean — 155
requests, 155 answers, no errors, no failover, both replicas steady — but only
25% of turns hit the prefix cache, the rest paying a 13.9 s median re-prefill.
That is the capacity budget showing: 122,636 KV tokens per card is about two
30k-token conversations with their cached prefixes, and a third family on the
same card evicts.
Chasing the obvious lever disproved it. Available KV per card is 5.44 GiB at
CHUNK=8192, 4.22 GiB at 16384 and 4.16 GiB at 4096 — where the engine refuses
to start. The budget is not monotonic in the chunk size, so 8192 is a sweet
spot rather than a compromise and freeing KV needs another card, not a knob.
The proxy trusted its health table, which the watcher refreshes every 10 s. A
replica that restarts — and a deploy restarts both — therefore produced a
window in which requests routed to the dead card got a 502 back, even though
the other card was serving the whole time.
A failed connection now demotes that replica immediately and retries the
request on another live one, before anything is written to the client.
vLLM's OpenAI endpoints are stateless, so re-sending a request that never
reached an engine is safe; a failure after streaming has begun is still not
retried. The failover is logged, because a silent one hides a card that is
down.
Measured with card 1 stopped mid-flight: six requests, six 200s, one failover
line in the log, no client-visible error. Before this commit that burst
returned 502s.
Upstream ships dflash depth 7 as a content-dependent default measured for TP=2
on their corpus. Re-measured on one card here, two runs per setting, 16
repo-derived prompts at 512 output tokens:
SPEC=7 conc1 46.2 tok/s, TPOT 20.8 ms, 3.86 tok/update
conc4 118.7 / 127.0 tok/s
SPEC=5 conc1 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update
conc4 136.1 / 129.4 tok/s
Step time is nearly identical at both depths (~79 ms vs ~80 ms single stream):
7 wins by landing more tokens per step, 5 wins at conc 4 by ~8% because more
narrow verify steps pack better. Depth 7 stays — this fleet is interactive
agent loops where per-turn latency is what is felt — and the runbook now
records the numbers and the one-line switch for a batch-first workload.
Raising the prefill chunk to 16384 grows the arena enough that the KV cache
falls to 4.22 GiB against the 5.21 GiB a 114688-token context needs, so the
engine refuses to start. Measured single-card prefill at 8192 is ~2.5k tok/s
(~15k-token prompt, 5992 ms TTFT); the trade would mean cutting context for
unmeasured prefill gain, so the default stands.
The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.
Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.
So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.
The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.
ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.
The single-GPU replicas died with torch.OutOfMemoryError during KV
allocation (2.16 GiB requested, 116 MiB free, 29.34 GiB already held by
PyTorch) because the entrypoint blanked KV_MEM=0 into an omitted flag. The
image's patched KV sizing reads an explicit 0 as 'profile this shape'; with
the flag absent vLLM sizes the cache from --gpu-memory-utilization alone and
overshoots. Passing 0 literally is the shape the bench numbers were measured
on, and MAXSEQS gets its default back under set -u.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling;
decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the
root-cause evidence chain, tuning levers, and co-tenancy traps
- AGENTS.md: fleet topology row + Active Services section for the ik
deployment; halogen demoted to rollback; bench policy unit type fixed
- LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE,
halogen-flash-server retired)
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).
- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records
the hardware swap, the self-contained gfx1201 build, the measured layer-vs-
tensor decision (llama-bench multi-device syntax gotcha included), the
tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm
posture. AGENTS.md, README, docs indexes, Dankserver System Specs,
deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename;
freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED.
Work log entry added.
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
Follow-up (same session, measured): llama-bench on the host with the correct
multi-device syntax (-dev ROCm0/ROCm1) overturned the tensor choice:
single R9700 pp4096 1160.9 / tg128 19.42 (d32768: 512 / 18.4)
layer 2x pp4096 1991.2 / tg128 18.76 (d32768: 859 / 17.6)
tensor 2x pp4096 271.8 / tg128 26.26 (d32768: 229 / 25.1)
This build has no RCCL, so tensor mode falls back to a "meta-backend
butterfly" allreduce and prefill collapses 7x. Served path with MTP:
layer 35.7-46.2 tok/s vs tensor 28.7. SPLIT_MODE=layer, draft pinned to
ROCm1 (the card holding the last target layers). Served prefix cache with
the draft head verified live: turn 2 reprocessed 22 tokens instead of 949.
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger
printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous
2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also
documents the ~45-60 s first-request PING silence after a restart (the
healthcheck's HealthStartPeriod covers it; do not restart the unit).
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).
Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.
Answering "does the FreeToken runtime use sampling arguments?" -- yes, and
nothing in our config revealed it. serve-frank-small.sh passes no sampling
flags; the behaviour is the runtime default:
freetoken/server/args.py:31 sampling_defaults: str = "model"
which fills temperature/top_k/top_p from the checkpoint's
generation_config.json -- this one ships do_sample:true, temp 1.0, top_k 20,
top_p 0.95. Proved live: three identical requests with no sampling params
produced three different reasoning traces (487/540/577 chars).
The asymmetry with frank-large is the part that will bite someone: a silent
client samples on frank-small and decodes greedy on frank-large, so the same
request is reproducible on one endpoint and not the other. Documented with the
determinism options (explicit temperature:0, or --sampling-defaults none).
Also recorded two request-level traps found during the probe: thinking eats the
whole max_tokens budget and returns empty content with finish_reason length,
and the API accepts unknown fields with a 200, so a 200 proves nothing about
whether a key was understood.
Ledger confirms the change landed: "3 slot(s) over one 524288-position pool",
"slots: 1 -> 3 POOLED, 111.0 MiB each", held 103.5 -> 103.4 GiB.
Recorded the oversubscription nuance that the old row missed: the slots were
never the binding constraint on context. 4 x 262144 exceeds the 524288 pool,
so admission was pool-bound all along -- two full-length conversations, or
four at 131k. Lowering the cap trims simultaneous requests, not capacity.
Concurrency cap trimmed to the real fleet size. Each slot is 111.0 MiB of
O(1) state, so this hands ~111 MiB back on a host sitting at 7.1 GiB left
for everything else.
Note the slots already oversubscribed the pool (4 x 262144 > 524288), so
admission was pool-bound regardless: at most two full-length conversations,
or four at 131k. This does not change how much context fits -- it caps
simultaneous requests at three, and a fourth now queues.
Raised /swap.img from 8 GiB to 64 GiB on Dankclaw. Host-level state with no
repo file, so it was invisible to the repo until now: fstab line 12 drives the
generated swap.img.swap unit, ext4, fallocated hole-free, swappiness 60,
zswap off. 748 GiB still free on /.
The reusable part is the ordering. The old file held 4 GiB of live pages and
MemAvailable read 73 GiB while only ~5 GiB was genuinely free -- the pinned
model weights are counted as reclaimable page cache, so the headline lies by
the size of the model. Swapping off before the replacement is online would have
pushed those pages into RAM that does not exist. Creating and enabling the new
swap first let them migrate instead: swapoff took 65 s and the host never
wobbled.
New page serverAppConfigs-docs/Dankclaw/Swap Configuration.md, linked from
the index, plus a pointer in the AGENTS.md memory-model section so nobody
reasons about headroom without checking swapon.
Old 8 GiB file staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/
pending approval to purge.
Seven restarts measured. New section "The reserve cannot buy air": reserve 16
was deployed and held exactly what 15 holds — 103.5 GiB, 7.1 GiB left to the
host. The flag is a term in the fit-check inequality, not a reservation, so it
has only two regimes: below the pool's requirement it changes nothing
observable, above it the pool silently disappears.
Sizing rule now defers to the engine's printed requirement (budget >= 36.5 for
524288 at max_tok 32768, from "need ~35.0 GiB plus 1.5 of margin") over my
modelled 37.0, and the ledger table carries the reserve-16 row.
Tested 16 live. Pool stayed 524288 (no clamp), server held 103.5 GiB and
reported "7.1 GiB left for everything else" at BOTH 15 and 16 — identical
footprint. HALOGEN_HOST_RESERVE_GIB feeds the fit-check budget only; it is not
an allocation, so raising it cannot give RAM back to the co-tenants. Its sole
observable effect above 16 is to take the KV pool away.
16 therefore cost half the fit margin (1.0 vs 2.0 against the engine's 36.5 GiB
requirement) and bought nothing. 15 is strictly better at the same cost.
Recorded the ceiling (16) and the real air levers in the unit comment: pool
262144 frees 7.2 GiB, max_tok 16384 frees 8.4.
Operator asked for 18. 18 is not available: the engine's own fit-check line,
read from a clamped boot rather than modelled, requires budget >= 36.5 GiB to
keep 524288 at max_tok 32768 ("524288 positions need ~35.0 GiB (plus 1.5 of
margin)"). budget(18) = 35.5, so 18 silently un-doubles the pool we just paid
the reserve for. 17 fits by 0.02, inside the noise of the 67.7 resident-weights
term. 16 is the largest reserve that survives the check with real margin:
37.5 budget vs 36.5 required.
Recorded the ceiling and both remedies in the unit comment. The reserve is now
spent, so a future stall is answered by the pool or the arena, not here.
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.
Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.
Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
Neither lever alone could give both features. At reserve 24 the device budget
is 29.5 GiB and max_tok 32768's 37.0 GiB need always lost the doubling to the
fit-check clamp; at max_tok 16384 the doubling survived but prefill paid 8-9%.
Dropping the reserve to 15 moves the budget to 38.5:
121.2 host - 67.7 resident weights - 15 reserve = 38.5
need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB spare
So the payment is page cache for the n-gram lookup table rather than either
feature. 15 GiB, against a table of 47.7 GiB read through the file cache.
Two reasons this is a survivable risk rather than a cliff:
- The engine's own ledger reported only 12-17 GiB left at the previous 24
setting, so the declared reserve was never held literally; 15 may cost
less real cache than it reads as.
- The failure mode is legible: long prompts fault heavily against a busy
disk, and raising the reserve is the documented remedy. Not the silent
clamp, and not the 65536 no-boot mode.
1.5 GiB of headroom is thin and deliberate. Documented in the unit as a
matched pair with the pool: raise the reserve back to 24 and the doubling
quietly goes away again.
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:
kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all
so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.
Two corrections to what I wrote earlier today:
1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
spare it" line, print ONLY when the fit check had to clamp something. The
09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
of a warning in old logs was never evidence of headroom.
2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
stated budget -- and has been the whole time. It boots and serves with
12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
floor, not a wall.
The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
Operator call. Prediction from the measured model: 524288 positions cost
35.0 GiB against a 29.5 budget, so HALOGEN_KV_POOL_FIT clamps the pool to
262144 (27.8). Rollback is a single revert; 16384 was the config that made
the doubling real.
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
Operator call, measuring what the prefill arena actually costs against
upstream's do-not-raise warning. Rollback is one revert: the settled
config was 16384, which made the 524288 pool fit at 26.6 GiB.
The first draft of these notes estimated the doubled pool at 103.5 GiB
held / ~5 GiB host remainder. Both numbers were wrong — they costed the
KV alone and ignored that the arena shrinks when MAX_TOK drops. Measured:
pool 262144 / max_tok 32768 -> 96.3 GiB held, 12.1 GiB host left
pool 524288 / max_tok 16384 -> 94.6 GiB held, 13.8 GiB host left
The doubled pool is 1.7 GiB cheaper overall. Also records the failed
first attempt verbatim, since the silent-revert failure mode is the thing
a future operator most needs to recognise, and adds the HALOGEN_MAX_TOK
row to the deployed-flags table (the "baked 32768" line it overrode was
stale).
The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT
measured the device budget at startup and lowered the pool straight back
to 262144, while the deploy still reported healthy in 12s.
The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0
reserved for the n-gram page cache and the OS = 29.5 GiB of device
budget, and that single budget carries the KV pool AND the prefill arena.
At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions
need 35.0 GiB against 29.5 available.
HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the
524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is
about 8-9% of prefill throughput, per upstream. This is the engine's own
prescribed fix, quoted verbatim in its refusal message.
Lowering HALOGEN_CTX was the other option the engine offered; rejected
because both clients declare 262144 as the context window, so it would
break the client contract rather than pay for the pool.
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Documents the root-cause chain for the marreta.franksdank.cloud login wall:
null invalidation_flow (outpost serde crash) -> missing flows -> and the real
blocker, ProxyProvider.application=None so the embedded outpost's
filter(application__isnull=False) never loaded the provider. Notes the
providers are live-only (no repo blueprint) as a provenance gap.
16-marreta.spec.ts (standard project) asserts the user homepage's Marreta
open link actually reaches the Marreta app through the full guard chain
(Anubis PoW -> Authentik SSO -> app), and explicitly NOT the authentik
login flow. Guards the 2026-09-12 incident where marreta-proxy was created
without a linked Application, so the embedded outpost never loaded it and
the homepage link parked on a login wall.
Wires spec 16 into the standard project testMatch. Verified passing live.
Adds the Marreta content-reader tile (http://marreta.franksdank.cloud,
probe 127.0.0.1:8082 via gluetun's publish) to the user grid so regular
authenticated users get a link to it. Registry-driven e2e picks it up
automatically.
The 2026-09-12 homepage outage slipped through because the live-fleet
Playwright suite was a manual 'MUST invoke' that deploys skipped. Wire it
into deploy.sh as the [3/3] gate: after a behavior-affecting deploy,
run tests/e2e/run.sh against the live fleet; a failure exits DEPLOY
INCOMPLETE. Add --skip-e2e as an emergency escape hatch. Update the
AGENTS.md deployment-completeness contract and flow diagram to match.
Verified: bash -n clean, --skip-e2e parsed, unknown flags rejected,
seam suite green (only the pre-existing comfyui test 73 fails).
Deployed live at 91e9410 (gluetun+marreta+caddy), authentik provider pk 13,
guard chain verified end-to-end. Documented that Laravel APP_KEY must carry the
base64: prefix — a bare base64 key 500s every app route with 'Unsupported
cipher or incorrect key length' while /up still passes.
The scoped runner (deploy-remote.sh) already knew marreta; the top-level
deploy.sh valid_service() gate did not, so --service marreta was rejected
before reaching the planner.
Verbose engine logging is the vendor's first-line evidence for load,
allocation and admission questions; shipping it on means an incident
report already carries the full startup account instead of needing a
restart-into-debug. HALOGEN_DMALLOC_LOG stays off (every alloc >64MB is
steady-state noise on a shared host). Doc updated with the rootless
podman-logs-vs-journalctl retrieval note.
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.
Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.
AGENTS.md's boot-persistence gap note is replaced by the resolution.
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.
The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
Hermes hardcodes response_format=json_schema on its title call. No local
back-end implements constrained decoding — frank-small and frank-large both
answer 400 'response_format json_object/json_schema is not supported' — so
every session logged a title failure and fell back to a message-derived
title. The NPU endpoint that used to absorb this call was retired on
2026-09-11.
The call now goes out over the Responses transport (the only one that does
not forward response_format) with thinking off. A 64-token cap cannot hold a
thinking pass plus a title: with the default reasoning effort the model spent
all 64 tokens thinking and returned empty content, which is why disabling
reasoning is part of the fix and not decoration.
Verified against the live endpoint with the real prompt and the real
extraction path: raw {"title": "Friendly greeting"} -> title
'Friendly greeting' in 3.4s, where the previous configuration returned 400.
Rebooting Dankclaw for the amd_iommu=off change exposed five services that
exist only as compose.yaml services with no Quadlet unit (prometheus, grafana,
loki, alloy, node-exporter): they did not come back and had to be restored by
hand. The boot-persistence rule already requires units for them; this records
the concrete gap so the next reboot is not a surprise.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
The unit comment always claimed the webui drives the same agent state as
the gateway, but the image bakes HERMES_HOME=/opt/data and no unit
overrode it. Podman auto-creates a fresh anonymous volume at every
image-VOLUME point on container recreation, so each restart reset the
webui's config home to the image-default template (provider: auto ->
openrouter.ai with no API key), while the gateway read the corrected
/data/config.yaml. Pin HERMES_HOME=/data so both processes share one
config/state root and config edits propagate without a recreate.
Live config was still routing on pre-rename aliases (frank-default-large,
frank-default-small) after the 2026-09-03 frank-large/frank-small/frank-npu
rename cutover; hermes-webui resolved those ids to nonexistent models and
requests failed. Live config fixed in place (backup kept on-host as
config.yaml.bak-20260905); this backup now mirrors the live shape: named
providers for aux routes, title_generation/triage/kanban routed to their
actual models, moa + mcp sections, signal disabled.
Wget-style (no-JS) clients must never receive application bytes from guarded
services: copyparty serves authenticated traffic with anonymous read perms, so
the guard chain is the only boundary between an unauthenticated client and the
file bytes. Assertions target leakage markers in app code (Copyparty UI title,
homepage panel grid, services.json JSON shape), not gate internals, so any
guard-route regression fails the suite while any gate implementation passes.
Also route 15-*.spec.ts into the guest project (empty storage state).
Flip the homepage Sonarr card from the user category to admin and repoint
its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded
public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its
post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A
record with it. No public Porkbun record ever existed (NXDOMAIN at the zone),
so after this there is no WAN path to Sonarr: users request TV via Seerr
(guarded), admins reach Sonarr's own UI from the LAN only.
Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
Rootless podman on dankserver already maps container root (PUID=0) to
the dank user (uid 1000) by default — /proc/<pid>/uid_map reads
0->1000, 1..->100000+. Verified across plex, tautulli, jellyfin,
calibre-web, copyparty, romm. userns_mode: host is unnecessary, and
podman-compose 1.5.0 cannot apply it anyway (compose services run in
pods; --userns and --pod cannot be set together). The real fix for the
never-booted arr stack was chowning the root-owned /opt/<svc>/config
dirs to 1000:1000.
The arr services were committed (b9de886) with PUID=0 but no userns_mode.
Under rootless podman's default subuid mapping, container-root cannot write
host-root-owned bind mounts, so every app died in logger init
(UnauthorizedAccessException on /config/Sentry) before binding its port —
the stack 502'd since it was first brought up (2026-09-04 00:33).
Per the fleet convention (AGENTS.md, plex/tautulli migration): PUID=0 +
UserNS=host maps container-root to dank (1000), which owns /opt data roots
and /mnt/entropy. Config dirs chowned 1000:1000 at bring-up.
Also drops a duplicate 'networks: [appnet]' key on prowlarr.
The arr stack (prowlarr, radarr, sonarr, lidarr, readarr, qbittorrent)
had A records in addn-hosts but no Caddyfile blocks. This adds:
- Six public vhost blocks in the dankserver section, each proxying to
192.168.0.235:<port> (exempt by construction: own auth, no Anubis)
- Six @arr named handlers inside :8017 for authenticated traffic routing
All 233 bats tests pass.
Boot tag flips from qwen3.8-distilled-heretic:9b to nuextract3:4b (verified
live: per-request routing already serves every registered tag; the boot tag
only decides which model is resident at start, so the first request after a
swap pays the load cost). FLM_CTX_LEN=-1 drops the global --ctx-len override
so each pack loads with its own trained context (nuextract3 32768, gemma4
131072).
Live run against the fleet showed the outpost redirects direct navigation to
auth.franksdank.cloud with the guarded host preserved as redirect_uri, rather
than rendering the flow UI in place. The security property under test is that
an unauthenticated visitor never sees app content; assert that shape (uidField
visible + no leak markers + gate is either in-place or a redirect carrying the
return path) instead of hostname preservation.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
14-guard.spec.ts: every authentik-guarded vhost (www, search, romm, files,
seerr, grafana, prometheus, loki) must challenge an unauthenticated direct
navigation; plex is asserted exempt (PMS answers directly, no IdP form).
11-admin.spec.ts: admin direct-nav to grafana/prometheus/loki passes through
with host preserved and no uidField challenge.
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
Post-change doc reconciliation for the hermes vhosts + DNS deletion batch:
- Hermes-Dashboard.md: access is now the hermesadmin.franksdank.cloud vhost
(direct Caddy proxy to 192.168.0.143:9119, no guard chain — exempt because the
dashboard enforces its own basic-auth login), plus the raw LAN port. The old
text claimed an Authentik-guarded public vhost, which was decommissioned.
- Hermes-WebUI.md: description updated — the webui is now also reachable via the
unguarded hermes.franksdank.cloud vhost (its own password auth is the gate).
- LAN-DNS.md: "add a name in two places" corrected to a two-or-three-step
decision; internal-only names (prometheus/loki/grafana/seerr) deliberately get
no public A record and resolve on the LAN only, and DNS-01 issuance is
unaffected because the challenge is a TXT record in the same zone.
- Seerr.md: DNS row corrected — addn-hosts only since 2026-09-02; NXDOMAIN on WAN.
- hermes-dashboard.container: unit comment no longer claims Authentik termination.
No behavior change in this commit; tests 233/233 green (docs-completeness seam included).
Both Hermes services get direct (unguarded) vhosts: each authenticates itself
(webui password; dashboard basic_auth in /data/config.yaml), so no Anubis
challenge and no Authentik proxy provider is required. Upstreams are the host
LAN IP because Caddy itself sits on dankclaw_default, where the gateway
address 10.89.1.1 has nothing serving 8787. Internal-only by DNS absence:
hermes.franksdank.cloud already resolves via Pi-hole addn-hosts; hermesadmin
is added in this same commit so the caddyfile/addn-hosts cross-check (test 178)
stays green.
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.
Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
/home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
daemon itself is a host user unit (signal-cli-daemon.service) bound to
0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
GET /health from another LAN machine returned 200) — the SSH tunnel is an access
path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
tunnel via the shared named network searxng.network, not service:gluetun.
Docs only — deploy is a no-op for this change.
PMS 1.43 serves TLS-only on 32400: a plain-HTTP GET gets a TLS alert and
curl reports "empty reply from server" (exit 52, code 000), so the card read
down while Plex answered 200 at /web/index.html over HTTPS. TCP connect is the
honest liveness signal for this port; the public URL stays HTTP because Caddy
terminates TLS and re-keys the path.
The Hermes card probed 127.0.0.1:8090, a port retired when the hermes
stack moved from compose (8090 web dashboard) to quadlets: the dashboard
now publishes 9119 and the webui listens on 0.0.0.0:8787 in the host netns.
Nothing has listened on 8090 since, so the card reported "down" while the
gateway, dashboard, and webui were all running.
Probe 127.0.0.1:8787 instead — the webui runs HERMES_HOME=/data, the same
agent state as the gateway, and /health answers only when that server is
alive (uptime, sessions, active runs). The prober runs on Dankclaw, so the
loopback probe hits the host netns where the webui binds.
The C5 probe failed for the wrong reason: plex.container lost its Network=
key in the same edit churn that ate tautulli's ContainerName. Plex ran on
the default podman network — published 32400 still worked (clients fine)
but tautulli could not resolve the plex alias, breaking the seeded
PMS_Web_Host=plex. Restore appnet membership; structure now parallel to
tautulli.container. Lesson recorded: after multi-hunk line edits on these
unit files, diff the whole file — key-order churn has eaten three keys
already.
Without ContainerName= the v1 generator names the container
systemd-tautulli; the fleet references containers by their plain names
(podman inspect/exec, deployer health checks). Container was healthy but
renamed — caught by post-cutover verification, not by the seam; the seam
only asserts file keys, so pin the name here and keep plex/tautulli
structures parallel.
- tautulli.container: HealthCmd switches to the repo's proven curl pattern
(jellyfin unit) — quadlet's systemd-style value parsing mangles embedded
double quotes; PublishPort=8181:8181 restored (a mis-aimed line edit
replaced it; the PublishPort seam test caught the regression)
- deploy_plex readiness gate: 36x5s -> 120x5s; the 2026-09 cutover showed
the first container boot of a years-old library runs ~10 min while PMS
pre-binds 32400 and empty-replies; 180s failed the gate twice with a
healthy, still-booting service
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
- runtime rows: podman quadlet under the systemd user generator, pinned
image digests, in-place bind mounts, identity model, appnet aliases
- cutover/rollback section on Plex.md (mask-native-unit lever, decommission
gate); drop the false 'updates are apt-only' and 'snap refresh' gotchas
- update path is now deploy.sh dankserver --service plex|tautulli
- plex: lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322 (verified ==
ghcr latest digest; plan's ls9 pin superseded by the re-verify clause),
/var/lib/plexmediaserver bind-mounted in place as /config, /mnt/entropy
mounted ro at the same path, PUID=997/PGID=984 + UsernsMode=host so the
container's file identity is byte-compatible with the native unit
- tautulli: lscr.io/linuxserver/tautulli:v2.18.1-ls242 (verified == ghcr
latest digest), fresh /home/dank/Containers/tautulli config root,
PUID=1000/PGID=1000, python3 /home healthcheck
- both on dankserver_appnet with aliases; 32400/8181 published as before
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
test-changed cutover case, quadlet seam enumeration + new
userns/PUID=997/PGID=984 ownership-contract test
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
- qB 5.x's 204 login breaks Readarr's retired client at runtime; 4.6.7 answers
legacy 200+Ok. Pin recorded in the unit and documented.
- Prowlarr download client created via schema template; all five clients test
200 through the podman gateway.
- PBKDF2 gotcha: qB 5.x conf rewrite left a hash 4.6.7 rejects; fresh rewrite
of the same line restored login.
- E2E: torrent through the Panama tunnel at ~17 MB/s, completed into Finished
Torrents, deleted with data.
podman 5.7.0 refuses ghcr.io/goauthentik/ldap:2026.8.0's nested manifest
list ('Unexpectedly received a manifest list instead of a manifest for a
single image'); pin the linux/amd64 single-platform digest instead.
Document the Mac *arr/qBittorrent stack (VM placement, gluetun credential
gate, socat LaunchAgents, PBKDF2 qB seed) and log the day: dankserver
recovery + full quadlet cutover of the six media services, the frankbook
skip-propagation/PBKDF2 handler fix, the 7-service frankbook deploy, and
the jellyfin GPU-parity finding.
The frankbook handler ran port verification even for services the
credential gate skipped, so a parked qbittorrent failed the deploy and
aborted the five arrs. Gate now lives Mac-side (test -s gluetun/env
inside the VM) before restart+verify.
The qB first-boot seed wrote a plaintext WebUI\Password, which qB >= 4.2
ignores (PBKDF2 @ByteArray only), and omitted HostHeaderValidation=false,
which would 401 the LAN forwarder path. Seed now generates the PBKDF2
hash in the VM (python3 hashlib) and sets ServerDomains=* +
HostHeaderValidation=false. Arr categories (lidarr/radarr/tv-sonarr)
pre-created with save_path=/Volumes/COMPLETE/Finished Torrents so any
grab lands in the review dir regardless of wiring details.
Seven units installed INSIDE the Mac VM (/home/core/.config/containers/
systemd) by a new deploy_frankbook_stack handler: gluetun (ProtonVPN
WireGuard, env-file credentials, publishes qbit's 8080/6881), qbittorrent
(Network=gluetun.container, verbatim /Volumes data roots, first-boot
config seed with fresh machine-local WebUI password and the Finished
Torrents category), and prowlarr/sonarr/radarr/lidarr/readarr (grab-only,
config migrated from dankserver at provisioning, readarr pinned to
amd64-1.0.980). deploy.sh routes the new target through the ssh-config
alias with REMOTE_SHELL=/usr/local/bin/bash (macOS bash 3.2 lacks
associative arrays); the planner learns FRANKBOOK_SERVICES and the
gluetun->qbittorrent netns pairing; new seam test pins the contract.
Two live-verified loader failures on Dankserver (2026-09-01), each a
clean exit at load (no swap-thrash, 37 GB fits RAM):
1. ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors PER-CHANNEL fp8):
ValueError: Missing MoE expert source layers: {'gate_up': [0, 1, ...]}
— nonstandard expert tensor naming, provider finds no expert sources.
2. lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8
(compressed-tensors BLOCK-fp8 128x128, standard per-expert naming):
RuntimeError: The size of tensor a (2048) must match the size of
tensor b (...) in _dequant_fp8_weight — the dequant path multiplies
weight * weight_scale directly and cannot expand compressed-tensors
block scales.
Empirical matrix for FreeToken v0.1.2 on qwen3_5_moe (RTX 3090):
modelopt block-fp8 (quant_method 'fp8' + weight_scale_inv) -> LOADS
compressed-tensors per-channel FP8 -> fails
compressed-tensors block-fp8 128x128 -> fails
batsclamp/Huihui-...-abliterated-FP8 is the proven uncensored checkpoint;
restoring it until a modelopt/NVFP4 requant of a wanted model exists or
FreeToken learns compressed-tensors.
Phase 1/2 of the podman standardization plan: service pages updated to the
quadlet reality (units, Containers data roots, env-file secret convention,
the v1-generator findings each caught live), Storage Policy documents the
two rootless ownership regimes and the merge-move migration, Deployment
Flow's service-manager table no longer claims dankserver media is bare
inventory-only. Bundle change log entry added.
Ornith-1.5-35B-A3B-FP8 failed to load on FreeToken v0.1.2: its
compressed-tensors checkpoint uses a nonstandard expert tensor layout
(FreeToken: 'Missing MoE expert source layers: gate_up'). Retarget to
the HauhauCS Aggressive FP8 finetune of the SAME Qwen3.6-35B-A3B arch
Huihui used, which carries compressed-tensors BLOCK-fp8 (128x128) with
the standard per-expert naming (mlp.experts.N.gate_proj/up_proj/down_proj
+ .weight_scale) that FreeToken's MoE loader reads.
- MODEL_DIR -> /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8
- use the checkpoint's own chat_template.jinja (drop the Sharp override;
forcing a template the finetune wasn't trained on risks format drift)
- pool plan unchanged (same ~35 GB FP8 mass / qwen3_5_moe arch)
The cutover unit joined dankserver_appnet but dropped the compose-era
0.0.0.0:3923 mapping (live inspect: bridge, 3923 published), so
copyparty listened inside the netns while the host refused. New seam
asserts every host-facing port matches the pre-cutover PortBindings:
romm 8085->8080, jellyfin 8096, calibre-web 8083, copyparty 3923.
The podman 5.x v1 quadlet generator rejected the whole jellyfin unit on
the unsupported Device= key (generator log: unsupported key 'Device' in
group 'Container'), so no jellyfin.service was generated and the cutover
restart failed with 'Unit not found' after the data root had already
migrated cleanly. gluetun proves AddDevice= is the house key. New seam
test denies Device= across all six dankserver units and pins the
AddDevice=/dev/dri replacement.
rename() of a directory requires write on the directory itself; mariadb/
redis data roots are owned by the containers' mapped subuids (100998),
so the deploy user cannot move them. Use run_privileged mv (numeric
ownership preserved = rootless mapping contract) and skip the chown pass
for romm via the new keep mode — chowning mapped subuid data to the
deploy uid breaks the in-container mysql/redis users. Collisions with
rescued anonymous-volume stubs now merge one level deep instead of
stranding real data under /opt. Reverse the podman rm order so the
dependent romm container is removed before its backends.
dankserver's ~/.config/containers/systemd did not exist (no quadlet units
had ever been installed there); the dankserver media handlers cp'd into it
without mkdir -p, failing before touching any container.
- MODEL_DIR -> /home/dank/models/ornith-1.5-35b-a3b-fp8 (37 GB,
compressed-tensors per-channel FP8, same qwen3_5_moe arch/mass as
Huihui so the pool plan carries unchanged)
- CHAT_TEMPLATE_FILE -> scripts/chat-template.jinja (Qwen-Sharp, shared
with frank-default-large); the checkpoint's stock template
mis-serializes reasoning per its own model card
- env/serve/unit headers updated; loader-risk + BF16-does-not-fit note
mv of a data root onto a non-empty directory fails (Directory not empty).
The romm/copyparty cutover creates Containers/<svc>/ before migrating:
the seed env file lands there first, and the romm anonymous-volume rescue
extracts into it before the /opt move. Merge the old root's entries in
without overwriting existing state (mv -n), warn and leave residue if a
collision kept the new side, and keep the plain rename when the
destination is absent or empty.
The first deploy after a unit-file edit restarted with the stale generated
unit (generator had not seen the new Volume= paths), crash-looping hermes on
statfs ENOENT for the just-renamed data root. Reload between cp and restart.
Repoint Volume=/EnvironmentFile= in the 8 affected quadlet units to the
caddy-established Containers/<svc> layout: gitea (data, postgres, env),
authentik (data, postgres), hermes (data, webui checkout). Data-root mv on the
target happens at deploy time before each group's unit restart; same-fs rename,
instantly reversible. Running containers keep their inode-bound binds until the
unit restart flips them to the new path.
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).
Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
/romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
ExecStart, gitea precedent). copyparty passes its admin account via
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
0.6.27-ls397), pasta network preserved.
Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
-v /w::A granted anonymous full rwmda on the LAN (verified: anon PUT -> 201,
PROPFIND -> 207). New perms ::r:ra,dank: browse/download for anonymous
(required by the guarded files.franksdank.cloud chain), admin for dank,
no w/m/d for anyone. Runbook corrected with the --help-accounts semantics.
Adds a digest-pinned headless-shell Chrome DevTools Protocol daemon as a
boot-persistent quadlet unit (chrome-devtools-protocol.container). The
entrypoint override drops the image's socat front so the CDP endpoint binds
loopback-only on 127.0.0.1:9222 in the host network namespace; only the two
host-network consumers (Signal gateway, webui server) can drive it.
hermes-webui flips Network=dankclaw_default -> Network=host so its agent
process shares the daemon's namespace (0.0.0.0:8787 bind is LAN-facing as
before; no exposure change). deploy_hermes installs the new unit and starts
it if inactive without restarting in-flight browser sessions.
DNS at Porkbun was renamed (chat->ai, jellyfin->jelly) and the qbittorrent,
*arr-stack, comfyui, hermes, dankclaw, and dankserver public records were
deleted. This brings the Caddy config in lockstep: ai/jelly vhosts now match
the new names; comfyui (guarded), hermes (guarded), and the *arr/qb vhosts
are removed as their DNS is gone. The legacy dankclaw/dankserver redirect
vhosts stay: Pi-hole still answers those names on the LAN.
LAN clients now connect directly to 192.168.0.143:8787 (password-gated via
HERMES_WEBUI_PASSWORD); the SSH tunnel via port 22 remains the sole
off-network ingress. Recreation also re-reads the rotated password from
dankclaw/.env via the [Container] EnvironmentFile.
Verified live 2026-08-31: guarded path serves the Anubis JS challenge at
HTTP 200 to non-browser API clients (status-only consumers silently get
HTML); direct published port 8081 from pass_ip sources returns real JSON.
Decision: no second agent-only instance — same engines and same
rate-sensitive VPN exit IP would see double request volume for zero gain.
Server-side implementation landed on Dankclaw: dedicated 'tunnel' account
(nologin shell, password-locked, key-only auth), sshd drop-in
99-tunnel.conf scoping AllowTcpForwarding to that user with PermitTTY off,
and permitopen="localhost:8787" key-option guidance so a leaked tunnel key
can only forward to the webui port. WebUI stays loopback-only published.
Token exchange chain verified end-to-end in-process on the live provider
(pk 9): authorize -> ORM-minted code -> form-encoded token exchange with
Basic client auth -> userinfo. Key findings documented:
- Application row + grant_types must exist before authorize can complete
- token endpoint consumes form-encoded bodies only (JSON body => no
grant_type parsed -> unsupported_grant_type); client auth is Basic-only
- PropertyMappingEvaluator.evaluate(*args,**kwargs) forwards kwargs to
BaseEvaluator.evaluate(expression_source), so context kwargs become the
expression source and every evaluation raises a swallowed exception;
userinfo therefore answers sub-only claims for every provider in this
fork. Scope granting filter (provider= M2M descriptor) is always empty.
- Grafana identity = sub hash until the evaluator bug is fixed upstream.
The embedded outpost carries no provider token (token=None), so the
auth_proxy identity header was never set and no user could log in. RP
mode is topology-independent: challenge gate stays in front, login runs
inside Grafana via the OAuth2 code flow against auth.franksdank.cloud.
Provider 'grafana' (pk 9, confidential) created via ak shell; secret
written straight into the live .env (never in git). Auth URL uses the
global /application/o/authorize/ path — this fork resolves the provider
by client_id query param; per-slug authorize does not exist.
Caddyfile comment corrected: no identity header is forwarded to Grafana.
Grafana leaves the exempt-by-construction set: grafana.franksdank.cloud
now terminates at the Anubis guard (anubis:8923) and authenticated
traffic returns at the internal :8017 listener, where an @grafana Host
handler proxies through to grafana:3000 with the outpost-proxy trust
token (GRAFANA_PROXY_TOKEN = client_id_client_secret of the new
'grafana-proxy' ProxyProvider on the embedded outpost; created via
ak shell before this route flip — provider-first ordering per the
External Exposure Policy).
The Grafana admin account is intentionally kept: auth.proxy in
auto-upsert mode only creates non-admin users, so the local admin
remains the only way to administer Grafana. The old edge-basicauth
wrapper is retired (shared family password, no per-user audit trail).
- Caddyfile: grafana vhost flipped to the guarded chain; @grafana
handler added at the :8017 return listener.
- compose.yaml: GF_AUTH_PROXY_ENABLED/HEADERS/TOKEN on the grafana
service (auth_proxy trusted-proxy CIDRs left at the image default —
the proxying Caddy container is not statically addressable on
10.89.0.0/24, so Grafana trusts any in-network source and relies on
the token as the capability).
- .env.example: GRAFANA_PROXY_TOKEN placeholder so test env_consistency
keeps guarding the compose<->env contract.
Every Caddy-proxied service is now either behind the auth chain or
explicitly internal-only, with the provider-first flip procedure codified.
The comfyui GPU-access seam test now asserts the deployed contract
(group_add keep-groups + udev MODE 0666 on kfd/renderD*) instead of the
pre-fix 'video' group assumption.
Loki's public vhost had zero auth — any host that can resolve DNS could
query all logs via LogQL. The vhost now terminates at anubis:8923 and
authenticated traffic returns at the internal :8017 listener, where a new
@loki host handler routes to loki:3100. The outpost proxy provider
'loki-proxy' (external_host https://loki.franksdank.cloud/, internal_host
http://caddy:8017) was created before this deploy — provider-first
ordering so authenticated traffic never hits the outpost error page.
Ingestion is unaffected: Alloy pushes to loki:3100 on the compose network.
The flip hunk's first comment line was written with a literal ':' prefix
(edit-tool auto-prefix artifact). Caddyfile parses ':# ...' as a matcher token,
which made the following site block an unrecognized directive and failed
config validation — that is why the earlier deploy validated, failed, and
skipped the reload, leaving the live config pre-flip (comfyui still direct).
The first revision guarded the kfd line with SUBSYSTEM=="misc", which never
matches (misc-class devices carry no subsystem for udev matching), so the rule
was read but never fired and /dev/kfd stayed 0660 root:render. Name-only match
applies 0666 on creation and on trigger.
Also fixes test-changed.sh to exit 0 when changed paths map to zero tests
(udev rules are mapped-but-untestable) instead of invoking bats with an empty
selection.
The comfyui container is spawned by a systemd user unit whose spawn context
carries no supplementary groups, so render-group access to /dev/kfd and
renderD128 was structurally unreachable (keep-groups preserved an empty set;
named group_add 'render' fails name lookup inside the userns). World-mode on
the compute nodes is the only spawn-context-independent grant; display cards
keep their modes. Apply: install to /etc/udev/rules.d, udevadm control
--reload-rules, then trigger by subsystem to re-apply modes to live nodes.
Also maps dankclaw/udev/* in the focused-test selector (not bats-testable).
Rootless podman cannot resolve the literal group name 'render' inside the
container user namespace and fails the start with 'unable to find group
render'. keep-groups preserves the launching user's real supplementary
groups (dank is in render on this host), which is the canonical rootless
ROCm device-access recipe.
/dev/kfd and /dev/dri/renderD128 are root:render 0660 on this host. The
container passed the device nodes but only joined group video(44), so torch's
HIP init failed with 'No HIP GPUs are available' and main.py never bound
:8188 — every proxy hop to the app returned 502. Group membership is a
create-time property, so this deploy recreates the container.
comfyui.franksdank.cloud was exempt (direct reverse_proxy to comfyui:8188).
The comfyui-proxy proxy provider was created on the embedded outpost first
(internal_host http://caddy:8017, flows copied from hermes-proxy), then the
vhost flips onto the guard chain and authenticated traffic routes back via a
@comfy host match. ComfyUI has no auth of its own; this closes the last
unguarded admin surface on the internal hosts.
The quadlet generator space-splits HealthCmd into the healthcheck argv, so the
quoted /bin/sh -c form arrived mangled and failed with exit 1 forever (FailingStreak
569) while the DB itself accepted connections. Bare argv survives both exec forms.
Env file is the source of truth (unit ExecStart runs the serve script which
sources it); serve-script default and the deployer health-gate URL flipped in
lockstep so the post-restart health gate polls the new port. Docs, fleet table,
and both index bullets updated to match.
Prior docs claimed unknown tags silently serve llama3.2:1b. Live probes on the
live endpoint disprove it: /v1/chat/completions answers HTTP 200 with a flat
{"error":"Failed to load <tag> model!"} body and no choices; /v1/completions
answers a nested OpenAI-shaped server_error (code 500) under HTTP 200. Valid
tags route per-request to the named model (response echoes the requested tag);
no restart needed to switch between registry tags. Corrected in the runbook,
the AGENTS.md runtime section, the serve-script header comment, and the
historical FastFlowLM incident note.
frank-default-sightsound (FLM v1.0.1, XDNA2 NPU, port 52626) is live on
Dankclaw. Documents the first-deploy crash-loop root cause: iommu=off on the
kernel cmdline breaks the XDNA2 SVA bind (amdxdna_drm_open ret -19), so every
/dev/accel/accel0 open fails with ENODEV; fixed with iommu=pt + reboot.
Whole-registry single-port serving with per-request routing by exact FLM tag;
unknown tags fall back to llama3.2:1b (no 404). Health = GET /v1/version.
Vision + audio towers proven live (image_url color probe; audio control-pair
token-count probe). Updates AGENTS.md fleet table + runtime section, both
index pages, and the change log.
The v1 quadlet generator skips the whole unit when [Container] holds an
unsupported key; the stale key was a leftover of a draft edit. Network=host is
the only host-networking key this generator version accepts.
The v1 generator on this host skips a unit whose [Container] section holds an
unknown key, so the deploy-time regeneration dropped hermes.service entirely
while the container kept running unmanaged. Network=host generates
--network host verbatim (v1 passes extension-less values through).
The gateway unit ran on dankclaw_default and could not reach the host
daemon: from that network only DNAT'd published ports accept connections to
the LAN address. The compose era ran the gateway with network_mode: host, so
the daemon was reachable; the unit silently changed topology at takeover and
broke the adapter. Host networking + a loopback dial removes the dependency
entirely — the gateway is an outbound-only client and publishes nothing.
The trio ran as bare compose containers and died silently at the Aug 26
reboot — nothing restarts compose containers at boot. Each now has a
boot-persistent unit (WantedBy=default.target; the v1 quadlet generator on
this host cannot enable generated units). The deploy handler retires the
compose-era containers only while the units are not yet active, then
(re)starts each unit.
Units run the image under a localhost/hermes-agent:latest alias tag (the
quadlet create path on this host cannot resolve registry-prefixed refs from
local storage); the deploy handler tags first. The gateway's :8090 publish
is dropped — nothing consumes it; the public vhost now routes to the
dashboard's login-gated UI on 9119.
The gateway's SIGNAL_HTTP_URL pointed at the host LAN IP. From container
space on netavark networks only DNAT'd published ports accept connections
to that address; :8080 has no forward, so every connection was refused
and the Signal adapter stayed disconnected since the Aug 26 reboot.
host.containers.internal is the verified working path (probed live from a
probe container on dankclaw_default).
The public hostname also routed to hermes:8090, which serves nothing —
the gateway has no HTTP listener. It now routes to the dashboard's login-
gated admin UI on :9119, behind the existing Anubis+Authentik guard.
The first deploy attempt pulled c3ed3a6 on the target but the remote runner
rejected the explicit --service selection because service_is_allowed() keeps
its own per-target allowlist; add the new LLM service to it.
New Dankclaw service frank-default-sightsound: FLM 1.0.1 serving the
Gemma4-E4B-IT-NPU2 pack (vision+audio+reasoning, 131072 ctx) on the XDNA2
NPU at port 52626. Model choice rationale: gemma4-it is the only NPU2 model
family with BOTH image and audio towers; the uncensored candidate qwythos:9b
is text-only, so sight+sound wins over uncensored here.
Deployer wiring: planner maps env/unit/serve files to the new service name;
deploy-remote gains deploy_llm_sightsound (relink + daemon-reload + enable +
restart + health gate on /v1/version — FLM has no /health route). Unlike the
retired frank-default-npu relink-only path, this unit is boot-enabled.
- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook:
runtime build recipe (WEBUI=OFF gotcha), deployed serving profile,
fa-on-works-unpatched evidence, MTP heads documented unloadable,
swap procedure + 2026-08-30 validation record
- agentionai-fast-imatrix marked RETIRED (rollback record kept)
- freetoken-frank-default-small: model/lineage/template sections now
Huihui abliterated FP8; new operational quirk: chat API 422 without
explicit model field
- LLM index + AGENTS fleet table/runtime sections current; both
services' history chains fixed (small was still Tiel in AGENTS.md)
- worklog entry added; models.json contextWindow for large corrected
to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
- same qwen3_5_moe arch and native block-fp8 format (e4m3, 128x128
weight_block_size, un-fused per-expert layout) as the official
Qwen FP8 checkpoint: identical FreeToken loader path, pool numbers
unchanged (2048 MoE slots / 453005 KV tokens / 4 requests)
- checkpoint downloaded to /home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8
(26 shards, ~35G); official checkpoint kept for rollback
- ships froggeric fixed Qwen3.6 chat template, read from checkpoint
- also refresh both .service Descriptions (large still said STRIX_LEAN)
- runtime: charlie build-qwen38 -> kingjones30/ROCmFPX (card recipe
build, LLAMA_BUILD_WEBUI=OFF since npm/HF-bucket provisioning fails;
all 35 of our serving flags verified present in the tree)
- model: agentionai FAST imatrix -> kingjones777 STRIX single GGUF
(on-host copy sha256 == HF LFS oid c6770d74...49fa verified)
- args unchanged from our validated profile (401408x2, b4096/ub512,
t16/tb16, f16 KV, fa on, fit off); mmap stays default per card
(PLE streams via page cache; --no-mmap = silent cgroup OOM)
Meta-header routing and forward_auth removed from the described design — both
disproved live. Documents: Host-based provider routing with per-guarded-host
proxy providers (lockstep rule), the null-flows wedge that starves the
embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport
exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next
URL as a dead design, and the public-DNS record requirement per guarded host.
Login/PoW/POST traffic to the outpost URL space cannot solve a fresh PoW
challenge per request; exempt the transport prefix (authentik's own session
gate still protects it) so the guarded-host sign-in journey completes.
The redirect-to-flow design with an absolute cross-host next= was rejected by
the flow's open-redirect guard at completion ("Request has been denied:
Invalid next URL"), and meta-header routing does not exist in this build
(providers route on the ORIGINAL Host). :8016 now proxies every request whole
to authentik-server:9000 preserving Host; the embedded outpost serves login
in place and forwards authenticated traffic back to :8017. Requires one proxy
provider per guarded host (provisioned live: search/hermes/prometheus/romm/
files + www, flows backfilled). Tests flipped from the meta-header claim to
the upheld contract: Host preserved, no absolute next= redirects.
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
Runtime: ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp at tag v1.1, built
with the vendor build-linux-amd.sh flag set plus GGML_CUDA_FA +
GGML_CUDA_FA_ALL_QUANTS (the Q4_1_MOE_FORCE_J path is live in the hipified
mmq kernels). Model: jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 (Q4_1 IU4 experts,
MTP Q8_0 draft, CIRUPLE1 PLE sidecar) with checksums verified at download.
Vendor production profile verbatim on first deployment: 262144 ctx x 1 slot,
b 2048 / ub 512, t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache +
8 GiB cache-ram + idle slots, 32 ctx checkpoints (min step 8192), MTP depth
3 with p-split 0.10, PLE pager env knobs (Q41_MOE_FORCE_J=32,
PLE_WORKERS=16, PLE_STRICT_SHA=0, HIPBLASLT=1). Qwen-Sharp chat template
kept deliberately; reasoning stays in DeepSeek format.
This 2026.x build does not mount /outpost.goauthentik.io/* for proxy-provider
meta-header routing (probed live: ping answers 204, app paths with full meta
headers 404). Guarded vhosts now redirect to /if/flow/default-authentication-flow/
with next= back to the app — unauthenticated sees the login page in place,
authenticated sessions complete instantly and bounce straight to the app.
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
All four arr roots live: /tv, /movies via API; /music, /books created after
v1 validator layers peeled back (quality/metadata profile ids must reference
existing profiles; v1 has no /api/v1/profile endpoint). Auth pitfall: the
literal AuthenticationRequired=Disabled string throws DryIoc.ScopedItemException
in v1 AuthorizationMiddleware on every request — the valid no-auth state is
ABSENCE of the key. Fixed live on four containers, verified boot clean.
Sonarr/Radarr root folders created via API (201). Lidarr/Readarr deferred to
first-run wizard (v1 validators demand quality-profile ids that don't exist
yet). Root cause of the write failures: LSIO PUID=1000 under rootless podman
maps to host uid 101000 — a phantom user owning nothing; PUID=0 maps to the
host user that owns every media dir.
Under rootless podman, LSIO's PUID=1000 maps to host uid 101000 — a phantom
user owning nothing: Sonarr's own validator rejected /tv because the process
could not write it. Container-root under a single-user userns maps to the host
user (dank), which owns every /mnt/entropy media dir and /opt config dir.
Identity is now mapping-independent instead of depending on subuid offsets.
calibre-web untouched (working; churn avoided).
Verified live: indexers->Prowlarr->per-arr QBittorrent download clients with
category routing. Readarr's client parked (qB proxy auth defect in abandoned
upstream). EZTV/1337x parked with reason (CloudFlare-blocked at this egress);
Nyaa/sukebei/RuTor/Byrutor/YTS passed live tests and are enabled. Prowlarr v2
Applications rows now carry real arr baseUrl+apiKeys; before this every sync
attempt threw NullConfig in GetIndexerMappings.
Verified live: AuthenticationMethod=None + AuthenticationRequired=Enabled on
all five arrs; unauthenticated API probes return 401 both locally and through
the vhosts. Also records the as-built divergences (entropy mounts, LA timezone,
readarr tag pin, prowlarr mount fix) that supersede the design compose.
Kestrel rejects the h2c upgrade preface; Caddy surfaced that as a 502 per
request while the backend answered 200 on plain HTTP/1.1 from Caddy's network.
Contrast recorded: Plex PMS needs the TLS transport, Jellyfin needs none.
Jellyfin's Kestrel backend rejects the h2c upgrade preface, so Caddy turned
every request into a 502 while the same port answers 200 over plain HTTP/1.1
from Caddy's network. Default transport (HTTP/1.1) restores the vhost.
The page claimed an env-var interface (COPAY_PASS/COPAY_MODE) and a /data
volume convention that this image does not implement: the ac edition consumes
accounts and volumes as argv flags (-a USER:PASS, -v SRC:DST:PERMS), serves
the LUKS array at /w, and answers HTTP 500 with its own error page when zero
volumes are configured. Image ref corrected to copyparty-ac (bare latest tag
does not exist on ghcr).
Ground truth from the image itself: python3 -m copyparty --help-accounts shows
-a takes USER:PASS and -v takes SRC:DST:PERMS:FLAGS — the COPAY_PASS/COPAY_MODE
envs this compose carried are not consumed by this image at all (zero COPAY_
references in the package). With no accounts and no volumes configured, the ac
edition serves its cwd read/write for everyone: the stack was running an
unauthenticated world-writable share of the 91T array. Now: admin account dank
with the A flag on volume /w; anonymous gets nothing.
The ac edition's entrypoint (cpp.sh -> python3 -m copyparty) consumes config
via /z/initcfg, not argv; -v is the volume flag and ate the next arg,
crash-looping argparse. The image mounts everything under /cfg/copyparty as
volumes by marker-file convention — plant the marker on the host instead.
The image entrypoint is /bin/ash /z/cpp.sh -c /z/initcfg: copyparty flags are
passed through command, and the ac edition mounts volumes with -m, not via a
COPAY_VOL env. Without any volume mounted every request resolved zero volumes
and the app answered HTTP 500 with its own error page. Verbose flag included
for the bring-up; drop it once stable.
The ac edition maps volumes from COPAY_* envs; without a volume root every
request resolves zero volumes and the app answers HTTP 500 with its own error
page. Point it at /w where the LUKS array is mounted.
ghcr.io/9001/copyparty:latest does not exist — ghcr answers 403 on the token
request for nonexistent repos, which is why the bring-up failed at pull. The
upstream publishes ghcr.io/9001/copyparty-ac (ac = full edition) and serves
whatever is mounted at /w, so the compose mounts the LUKS array there; config
persists in /opt/copyparty/cfg per upstream layout.
Jellyfin ran as user 1000:1000 inside a rootless user namespace, where host
uid 1000 maps to container uid 0 — the pinned uid is unmapped, so /config/log
creation EACCES'd and the service SIGSEGV'd on boot. Container root (= host
dank) is correct under rootless podman; group_add 992 (render) still maps via
the userns group map, so GPU access is preserved.
Readarr: lscr.io/linuxserver/readarr:latest does not exist — LSIO ships only
arch-prefixed nightly/version tags for readarr (verified against the ghcr tag
API). Pin amd64-1.0.980, the newest stable tag.
The compose stack mounted /mnt/media/{movies,shows,tv,music,books,downloads}
— a prefix that has never existed on this host. The 91T LUKS array mounts at
/mnt/entropy and its libraries are case-exact (Movies, TVShows, Music,
eBooks). Also restores prowlarr's config mount (eaten by an off-by-one in the
prior edit) and fixes TZ New_York->Los_Angeles on the arr stack.
Live-verified 2026-08-29: plexmediaserver.service is an enabled native
systemd unit on Dankserver (not a quadlet; updates via apt, not deploy.sh),
and tautulli runs as a snap with no auth wall on 0.0.0.0:8181. The hunt list
named both; neither had a page. Index linked per the docs-completeness seam.
The policy must ALLOW Anubis's own endpoints before the challenge-all rule
or the challenge challenges itself: browsers POST-solve via make-challenge
then GET-navigate pass-challenge, and a challenged answer renders as the
browser's own fetch-failure error page. Non-JS clients never reach the
verify step, which is why every curl probe looked green while every human
hit a wall. Also records that v1.13.0's valid action verbs are
ALLOW/CHALLENGE/DENY (ACCEPT crash-looped the unit on first deploy).
The pinned binary's policy parser rejects unknown actions and exits 1 on
startup (verified against the image binary: the valid action literals are
ALLOW, CHALLENGE, DENY). The first cut of the own-endpoints accept rule
used ACCEPT and crash-looped the unit on deploy.
The catch-all path_regex "." policy challenged everything, including
Anubis's own API namespace /.within.website/x/cmd/anubis/*
(make-challenge, pass-challenge, static assets). In whole-request proxy
mode a JS-capable browser POSTs its challenge solution to
pass-challenge and received another challenge page (HTML) instead of the
verification verdict (JSON): the challenge app's fetch failed and the
browser rendered its own error — 'the request filters did not return an
alternative response'. Non-JS clients never POST the solution, so every
curl probe looked green while every human hit a wall.
Accept rule ordered above the catch-all; policy is volume-mounted ro from
this checkout, so the deploy updates it in place.
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
The quadlet-dir allowlist silently dropped every change under
dankclaw/authentik/ — the new outpost unit (and its siblings) now route
by exact unit name, mirroring the searxng/gitea pattern. Adds
authentik-outpost to the service inventories; it is a quadlet unit, not
a compose service, so it stays out of the compose set.
In authentik 2026.8.x the proxy-provider URL space (/outpost.goauthentik.io/*)
is served by the Rust worker binary, not the server — verified live: with a
tenant-matching Host header, /if/flow/... answers 200 on the server while
/auth/caddy still 404s. The outpost instance existed in the DB but no process
ever served it, so Caddy's forward_auth subrequests hit a 404 and browsers got
authentik's own Not-Found page instead of the authentication flow.
The new unit runs the same image as server/worker with Exec=worker and flips
into outpost mode via AUTHENTIK_HOST/AUTHENTIK_INSECURE/AUTHENTIK_TOKEN (the
token is read from the DB into the host .env at deploy time; it never enters
the repo). It dials back to core over websocket and serves forward-auth on its
own 9000. Deploy handler restarts only this unit — the rest of the stack is
untouched.
Replaces the pre-Anubis direct-routing diagram with the live chain:
Caddy TLS -> Anubis :8923 verify -> accepted traffic back to Caddy's
internal listener :8016, where per-vhost routing resumes and Authentik's
embedded outpost gates wrapped apps via forward_auth. Guarded set
(homepage, search, hermes, prometheus, romm, files) and the
exempt-by-construction set (auth, plex, gitea git paths, grafana) are
listed explicitly. Access Policy bullet updated: basic auth was replaced
by the challenge-then-forward_auth layering.
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
reference parser rejects before any network I/O (repository name must be
lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
5.7 quadlet path fails container create with 'image not known' for
registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
shell; HealthCmd is now a JSON exec-form array, which passes through the
quadlet generator's argv assembly verbatim.
Tests updated to assert the alias-tag reality plus the pull guard.
The image has no shell (ko-build distroless; entrypoint /ko-app/anubis),
and this host's quadlet generator joins a bare HealthCmd into one
\x20-escaped argv element that podman then runs shell-form — exec of a
missing shell exits 1 and the container sits unhealthy. A JSON array in
HealthCmd passes through the generator verbatim and executes the binary
directly.
The v1.13.0 image is distroless: no shell, no HEALTHCHECK in the image
config (verified live: Healthcheck config is null on this host's image),
so quadlet inherits nothing. Declare the probe explicitly in exec form;
'anubis --healthcheck' exits 0 when the challenge server answers.
On this host's podman 5.7 quadlet path, a registry-prefixed Image ref
fails to resolve from local storage at container create time even when
the image is present ('image not known'); the identical argv works from
a shell or transient unit. Verified live: an identical unit with
Image=localhost/techarohq/anubis:v1.13.0 starts immediately. The deploy
handler now tags the alias after the guarded pull; upgrades re-tag
after pulling the new tag.
image inspect on an absent reference can exit non-zero for reasons
unrelated to existence in a non-tty deploy context; image exists is
the boolean primitive and cannot misreport. Absent-image behavior
(pull from ghcr) is unchanged.
podman's reference parser rejects mixed-case repository refs before any
network I/O ('repository name must be uppercase'); upstream's own install
docs pull lowercase (ghcr resolves case-insensitively). The first deploy
of this chain exit-2'd on that parse error — the image was not yet
pulled, so the guarded vhosts 502 until this deploy lands.
The first deploy run dispatched caddy (hot reload, no restart) but exit-2'd
on the inventory guard before deploy_anubis could run: deploy.sh's
valid_service was extended but the remote runner's per-target inventory
was not. Guarded vhosts 502 until this lands.
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).
Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.
Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).
Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
Verified live on dankclaw: the podman 5.x v1 quadlet generator splits
HealthCmd values on spaces into the container healthcheck argv array, so
quoted probes (sh -c "pg_isready ...") arrive mangled and every probe
exits 127 — postgres read 'unhealthy' with the database actually up.
Probe forms now use only space-free argv tokens: pg_isready passes its
args directly; the server probe runs manage.py check --database default
(verified exit-0 standalone in the image).
Probed the live generator on dankclaw: podman 5.x v1 hard-fails unit
generation on dash-prefixed options (--shm-size killed authentik-server's
unit) and rejects Command= entirely; Exec= is the accepted role selector.
Drop --shm-size (uwsgi uses unix sockets under /run, not /dev/shm; default
64m shm is fine) and drop systemctl --user enable — generated units are
transient and cannot be enabled; WantedBy=default.target in each unit file
is what makes them boot-persistent on this generator.
The compose-era authentik trio died at every boot (rootless podman does not
auto-start bare containers). Convert the trio to Quadlet units installed by
deploy-remote.sh into ~/.config/containers/systemd/, enabled + WantedBy=
default.target:
- 3 unit files under dankclaw/authentik/quadlet/ (postgres -> server -> worker
ordering via Wants=/After=; secrets stay in the git-ignored host .env,
expanded by systemd at start time exactly like the gitea units)
- deploy_authentik() handler: install units, daemon-reload, retire stale
compose-era containers only while units are inactive, enable once, then
start in dependency order with podman-health gates between stages
- compose.yaml: authentik service definitions retired to a comment pointing at
the quadlet dir (same pattern as searxng/open-webui/gitea/caddy)
- tests/authentik.bats rewritten from compose assertions to unit-file
contract assertions (images, Exec roles, ordering, loopback-only publish,
pg_isready probe, secret hygiene, dispatch routing)
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Both the EasiiX Q8_0 and agentionai ROCmFP4-FAST MTP draft heads fail
with 'missing tensor blk.0.hc_attn_norm.weight' against the STRIX_LEAN
Uncensored model. The qwen4exp binary expects this hyper-connection
attn norm tensor which neither draft GGUF carries. Using ngram-mod
only until a compatible MTP head is found.
The EasiiX Q8_0 draft was incompatible with STRIX_LEAN (missing
blk.0.hc_attn_norm.weight). Switched to agentionai/Qwen3.8-Flash-Next-
MTP-ROCmFP4-FAST-GGUF (2.28 GiB, ROCmFP4-FAST quant matched to target).
n_max 3 per the HF card (n_max 4 measured worse). SPEC_TYPE=draft-mtp,ngram-mod.
The script runs ON Dankclaw but was calling ssh() to SSH back to
dankclaw for service management. Nested SSH fails (no key auth for
localhost). Replaced ssh() with local() using subprocess.run(shell=True)
for systemctl and sed commands.
The env file had the old STRIX paths (directory doesn't exist on disk).
Updated to the actual running STRIX_LEAN (Uncensored) variant:
- MODEL_DIR → kingjones-Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN
- MODEL_FILE → 3-shard Q4_0 (00001-of-00003)
- MMPROJ_PATH → mmproj-Qwen3.8-Flash-Next-Uncensored-BF16.gguf
- serve-frank-large.sh: add --mmproj $MMPROJ_PATH
Reconciles the uncommitted local changes that were on Dankclaw.
serve-frank-large.sh: only pass --model-draft when SPEC_TYPE contains
'draft-mtp'. ngram-only configs (ngram-mod, ngram-simple, etc.) do not
need a draft model file — loading one wastes VRAM and may error.
bench-spec-decode.py: new benchmark script. Sweeps 4 configs (none,
ngram-mod, draft-mtp, draft-mtp+ngram-mod), generates 16k tokens per
config at temp 0 from a minimal prompt. Uses HTTP API (llama-bench
cannot target --spec-type). Follows benchmarking policy: stop+mask
the production service, bench, unmask+restart, health check.
One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) and terminates TLS for every service on the LAN. Pi-hole on Dankclaw answers the service names for LAN clients; Porkbun answers them for the WAN. The G36 router firewall governs external access. SearXNG outbound traffic exits through the Gluetun VPN container.
## Topology
```mermaid
graph TB
subgraph WAN["G36 router 192.168.0.1"]
EXT[Internet]
end
EXT -->|port forwards| C
subgraph Dankclaw[192.168.0.143]
C[Caddy :80 :443]
D[Pi-hole :53]
C -->|comfyui:8188| F[ComfyUI]
C -->|192.168.0.143:3000| G[Gitea quadlet]
C -->|192.168.0.143:8081| SE[SearXNG quadlet]
C -->|192.168.0.143:3005| O[OpenWebUI quadlet]
SE -->|network_mode service| GL[Gluetun VPN]
GL -->|tunnel| EXT
end
subgraph Dankserver[192.168.0.235]
C -->|192.168.0.235:8085| R[RomM]
C -->|192.168.0.235:3923| P[Copyparty]
C -->|192.168.0.235:8083| W[Calibre-Web]
C -->|192.168.0.235:32400| PL[Plex host service]
end
CLIENT[LAN clients] -->|DHCP DNS| D
```
Dankserver runs no proxy of its own. Its services expose published LAN ports, and the Dankclaw Caddy routes the names to those ports.
## Hostnames
| Scope | Convention | Resolved by |
|-------|------------|-------------|
| LAN | `romm.franksdank.cloud` → 192.168.0.143 | Pi-hole (`addn-hosts`) |
| WAN | `romm.franksdank.cloud` → 71.227.227.172 | Porkbun A record |
Users type a name and nothing else. Caddy maps the name to the upstream. The split between the two resolvers removes the need for router NAT loopback. See [[LAN-DNS]] for the flow and the router hand-off.
## TLS
Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through the DNS-01 challenge via the Porkbun module. No inbound port 80/443 is required. Details in [[Caddy]].
## Gluetun Path (SearXNG only)
SearXNG attaches to the Gluetun network namespace (`network_mode: service:gluetun`). Only SearXNG shares the tunnel. Everything else on Dankclaw uses the normal LAN route. This matches KB Pattern 2 in [[Gluetun]].
## Access Policy
- The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state.
- Caddy adds HTTP basic auth on the homepage and on sensitive services.
- LAN subnet: `192.168.0.0/24`.
## See Also
- [[LAN-DNS]] — name resolution, Pi-hole, router hand-off
All persistent data uses **bind mounts**: a host directory mapped directly into the container. No named volumes. This matches the preference of this repo and the existing Gitea stack on Dankclaw.
## Why Bind Mounts
- The exact host location of every byte is visible and controllable.
- Backups are a simple `tar` of known host directories.
- The data survives container recreation and `podman-compose down`.
- Ownership is explicit: everything runs as UID 1000 (`dank`) so host tools can read it.
The last two rows are bare-metal systemd services, not containers. This
repo owns their scripts, units, and knowledge base (see
[docs/LLM/index.md](../LLM/index.md)). Current mapping (2026-08-22):
`frank-default-large` runs the CIRU ROCmFPX llama.cpp on Dankclaw;
`frank-default-small` runs the upstream llama.cpp on Dankserver.
## Boot Persistence
The Gitea database and server, the SearXNG trio (gluetun, searxng, valkey), open-webui, Pi-hole, and Caddy run as Quadlet units, so they start at boot. Caddy left `compose.yaml` on 2026-08-26 after a reboot exposed that bare compose containers never come back; see [[Caddy]] for the unit and the incident. The former standalone Gitea Caddy unit is retired. User linger is on. ComfyUI runs as a systemd unit around a bare process. The remaining services in the table are defined in `compose.yaml` and are not yet boot-persistent — per AGENTS.md (Boot Persistence & Self-Healing) they each need a quadlet before they may be considered reliably deployed.
Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-webui with `scripts/deploy-open-webui-quadlet.sh dankclaw`, and Pi-hole with `scripts/deploy-pihole-quadlet.sh dankclaw`. The Gitea units install to `~/.config/containers/systemd/` (see [[Gitea]]); Caddy is deployed through `scripts/deploy.sh dankclaw --service caddy`.
## Routing Rules
- SearXNG outbound traffic exits through the [[Gluetun]] VPN container.
- Every other service uses the normal LAN route.
- The homepage lists all services with status.
## See Also
- [[index]] — repository index
- [[Network Architecture]] — routing and proxy
- [[Observability]] — Grafana, Prometheus, logs
- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
The Hermes web dashboard is a browser UI for managing the agent. It shows config, API keys, and sessions. It is a management tool. It is not a chat window. Chat happens over Signal or the CLI. See [[Hermes]].
| Data | shares `/home/dank/hermes-data` with the gateway |
## Why a separate container
The gateway container runs `hermes gateway run` (the Signal platform). That entrypoint does not start the dashboard. The dashboard is a separate process. So it runs in its own compose service (`hermes-dashboard`). Both services mount the same data volume. Both restart at boot (`restart: unless-stopped`).
## Authentication
The dashboard refuses to bind to a non-loopback address without an auth provider. There is no unauthenticated public bind. This repo uses basic auth:
-`dashboard.basic_auth.username` and `dashboard.basic_auth.password_hash` live in the live `/home/dank/hermes-data/config.yaml`; the tracked backup at `dankclaw/hermes/config.yaml.example` intentionally omits the password hash.
- The stored value is a one-way scrypt hash. The plaintext password is never written to a file or the repo.
- The hash is generated with `plugins.dashboard_auth.basic.hash_password`.
## Access
Open `http://192.168.0.143:9119` in a browser on the LAN. Log in with the configured username and password.
## Gotchas
- The dashboard and the gateway share one data volume. A config change made in the dashboard takes effect when the gateway restarts.
- If the shared password is reused elsewhere, treat it as exposed. Rotate it and regenerate the hash.
-`--skip-build` uses the prebuilt web assets in the image. Do not remove it unless the assets are rebuilt.
## See Also
- [[Hermes-WebUI]] — the phone-facing API server (Hermex)
Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compatible endpoints (vLLM, llama.cpp, Ollama) and adds RAG, plugins, tool calling, and multi-user support.
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-22). Both services listen on `:7770`, distinguished by host. On 2026-08-22 `frank-default-large` moved from `:52625` to `:7770`.
The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`).
> Both endpoints are bare-metal systemd services managed by this repository (the former `vllmScripts` repo, merged in 2026-08-22). Model swaps, context sizing, and spec tuning happen here. See the [LLM knowledge base](../LLM/index.md).
## Boot persistence (Quadlet)
The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
2.**Network liveness** — a non-loopback interface exists in `/sys/class/net`. The dead-tap zombie state (History) leaves only `lo` behind, and this is the check that catches it.
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
## Retired instances
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
- The bare-container `open-webui` on Dankserver (port 3000) pointed at a dead backend (`:9090`) and was removed. Its data dir is preserved on Dankserver in case it is needed.
- The bare-container `searxng` on Dankserver (port 8081) was removed. SearXNG now runs only on Dankclaw behind Caddy.
## Gotchas
- Use the non-CUDA image on this CPU-only host.
- The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
## History
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
## See Also
- [[Dankclaw Stack]]
- [[Hermes]] — the agent OpenWebUI can talk to
- [[SearXNG]] — the search engine on this host
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
The *arr suite is a collection of automated media management tools for Usenet and BitTorrent. For Dankserver — which runs Copyparty, Calibre-Web, RomM, and Jellyfin — the *arr stack provides automatic downloading and library management that Copyparty alone does not cover: quality profile enforcement, series/movie discovery, metadata enrichment, and automated file organization.
The core chain is: **Prowlarr** (indexer API aggregator) → **Sonarr/Radarr/Lidarr/Readarr** (media-specific managers) → **qBittorrent** (download client) → **Jellyfin** (media server scan → auto-refresh).
Use Copyparty for curated file collections. Use *arr for automatic downloads. Use both together for maximum functionality — point *arr download directories at Copyparty's watched folders.
Calibre is the e-book library manager that backs Calibre-Web. It manages metadata, formats, and conversions for the library that Calibre-Web serves.
**Host:** Dankserver (library directory on host) — **Docs:** https://calibre-ebook.com
## Role in this Stack
Calibre itself is a desktop/GUI application, not a container service. This stack runs **Calibre-Web** in a container and points it at an existing Calibre library directory on the Dankserver host.
Calibre is used to:
- Import books into the library (drag-and-drop or CLI `calibredb`)
The actual library files (`.epub`, `.mobi`, `.pdf`) and the Calibre database (`.db` file) live on the host filesystem. Calibre-Web reads and writes them through the mounted volume.
## Current Version
Calibre 9.2.1 is the version on the live host (9.9.0 is the upstream latest). Major features include:
- Bookshelf view
- Opt-in AI features
- Content server security fix (CVE-2025-7404 equivalent): a bug since v6.16.0 allows any user with server access to read non-Calibre files on the host. Upgrade to 9.9.0+ if serving content to untrusted users.
## Library Directory
| Purpose | Host path (convention) |
|---------|------------------------|
| Calibre library | `/opt/calibre-web/books` (bind-mounted to `/books` in the Calibre-Web container) |
## What the Container Needs
- Read/write access to the library directory. Calibre-Web edits metadata through the database file.
- Ownership matching `PUID=1000` / `PGID=1000` (the `dank` user on Dankserver).
## Optional: Calibre Content Server
Calibre includes a built-in HTTP content server (`calibre-server`) that can serve the library via OPDS feeds and a mobile-friendly web interface. This enables:
- OPDS feeds for mobile reading apps (FBReader, Moon+ Reader)
Copyparty is a lightweight file server and media streamer that runs as a single Python binary. It handles large payloads, media streaming, resumable uploads, directory browsing, and WebDAV from one process.
A simple HTTP check on port 3923 works. The root path returns HTTP 200 with the directory listing.
## WOPI Integration
Copyparty v1.20.9+ supports WOPI (Web Application Open Platform Interface) for editing Office documents in the web UI via Collabora Online. Enable with the `--wopi-accs` flag to limit WOPI access to specific paths.
Jellyfin is a free, open-source media server and suite of applications for organizing, managing, and streaming personal media collections (movies, TV shows, music, photos) to networked devices. It is a community fork of Emby, written in C# (.NET), and uses FFmpeg for transcoding.
The database must be on local storage (SSD/NVMe preferred). Media libraries can be on NFS/mergerfs. If media is on NFS, set `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`.
## Authentication
- **Built-in auth (default):** Per-user libraries, parental controls, max bitrate limits. Manage via Dashboard → Users.
- **Caddy basic auth:** Single login layer. Does not integrate with per-user Jellyfin permissions.
- **LDAP/OIDC:** Available via plugins (LDAP-Auth, Jellyfin SSO). Integrates with Authentik, Keycloak, etc.
For Dankserver, built-in auth is recommended. Use Caddy basic auth on top for external access.
## Metadata Scraping
| Provider | Content | Type |
|----------|---------|------|
| TheMovieDb (TMDb) | Movies, TV shows, posters, fan art, overviews | Default |
| OMDb API | Movies only, English metadata | Default |
| TheTVDB | TV shows, episodes | Plugin |
| fanart.tv | Extra artwork, banners, thumbnails | Plugin |
| AniDB | Anime metadata | Plugin |
| MusicBrainz | Music metadata | Plugin |
| Open Subtitles | Subtitle downloads | Plugin |
| Intro Skipper | TV show intro detection | Plugin |
Run Jellyfin alongside Plex for an incremental migration. Point Jellyfin at the same media folders as Plex. Watch history and metadata will not transfer cleanly.
7. Configure Caddy reverse proxy. Set Known Proxies in Jellyfin.
8. Add media libraries in Dashboard.
9. Install plugins: Open Subtitles, fanart.tv, Intro Skipper.
10. Test playback on Android/Chrome.
## Gotchas
- **GPU passthrough silently failing:** `/dev/dri` must be passed and the correct render node used. Verify with `vainfo` inside the container.
- **Wrong render group GID:** Common GID is 992, but varies per host. Run `getent group render` to find it.
- **Subtitle burn-in triggers full transcode:** PGS (image-based) subtitles force full video transcoding. SRT/ASS (text-based) play fine.
- **Database on NFS:** SQLite on network storage causes slow startup and corruption risk. Keep `/config` on local storage.
- **Client bitrate limit too low:** Many clients default to 20 Mbps. Set Max Streaming Bitrate to Original or Maximum.
- **Naming convention mismatches:** Non-standard folder/file names cause missing metadata. Use `Movie Title (Year)` for movies and `Show Name/Season 01/Show Name S01E01.mkv` for TV.
## See Also
- [[Dankserver Stack]]
- [[ARR-Stack]] — automated download into Jellyfin libraries
- [[Copyparty]] — alternative file viewer for existing collections
Lidarr manages music albums in the *arr stack on Dankserver. It automatically searches, downloads, and organizes albums from indexers into the Jellyfin music library.
Prowlarr is the indexer management layer for the *arr stack. It aggregates indexers (torrent and Usenet) and exposes a unified API that Sonarr, Radarr, Lidarr, and Readarr consume.
qBittorrent is the torrent download client for the *arr stack on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr, and Readarr via category tagging.
Radarr manages movies in the *arr stack on Dankserver. It automatically searches, downloads, and organizes movies from indexers into the Jellyfin media library.
> **RETIRED (May 2026).** The Servarr Team retired Readarr: its metadata became unusable and the community migration to Open Library stalled. There is no upstream development. Alternatives: the Bookshelf community fork, or the Calibre content server. See [Readarr (Retired)](https://wiki.servarr.com/readarr).
Readarr manages ebooks and audiobooks in the *arr stack on Dankserver. It automatically searches, downloads, and organizes books into the Calibre library.
Sonarr manages TV shows in the *arr stack on Dankserver. It automatically searches, downloads, and organizes episodes from indexers into the Jellyfin media library.
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). One Alloy instance runs per host and forwards container and journal logs to Loki, and optionally metrics to Prometheus.
**Hosts:** Dankclaw + Dankserver (one per host) — **Docs:** https://grafana.com/docs/alloy/latest
## Facts (from research)
| Item | Value |
|------|-------|
| Image | `franksdank/alloy:latest` (local build with a static busybox healthcheck) |
| Port | none published (push-only) |
| Config | River syntax, `config.alloy` |
| Successor to | Promtail (EOL 2026-03-02) |
| Inputs | journald socket, container log files |
| Output | Loki push endpoint |
## Why Alloy and not Promtail
Promtail reached end-of-life on 2026-03-02. Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
## Configuration
River config at `shared/alloy/config.alloy`:
- Read journald logs from the host's `/run/systemd/journal/socket`.
- Read container logs from `/var/lib/containers/...` (podman) or the docker path.
- Forward to `http://loki:3100/loki/api/v1/push` with `host` and `job` labels.
- The journald socket mount is required for journal scraping.
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, with basicauth).
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs, and it feeds the log panels in Grafana. It replaces the need for a full ELK stack.
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
- No Elasticsearch memory cost, no Kubernetes required.
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
## Scrape Surface
| Target | Source |
|--------|--------|
| Prometheus self | `localhost:9090/metrics` |
| ComfyUI | `/system_stats` (limited) or blackbox probe |
| Gitea | `/metrics` (built-in exporter) |
| Grafana | `/metrics` (built-in) |
| Host metrics | node-exporter `:9100` |
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
## Healthchecks
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.
Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/etc/prometheus` | `/opt/prometheus/config` |
| `/prometheus` | `/opt/prometheus/data` |
## Healthcheck
Prometheus responds on `/-/healthy` with 200.
## Gotchas
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Use recording rules for heavy queries (see [[Grafana]]).
- Set a retention policy or the data volume grows without bound.
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
description:Real-world NVIDIA + AMD Strix Halo multi-node experiments:what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s.
tags:
- pipeline-parallelism
- research
timestamp:2026-08-14
---
# Research Brief: NVIDIA-Desktop + Strix-Halo Multi-Node LLM Inference
## Summary
Yes, mixed NVIDIA-ROCm multi-node LLM inference over RPC has been published and is **fully functional**. The closest real experiments to an RTX-desktop + Strix-Halo-NUC pairing are (1) a planned 4× RTX 3090 + Strix Halo 128GB via 25GbE (r/LocalLLaMA, Apr 2026), (2) a Tesla V100 + AMD R9700×2 via Vulkan layer-split on one box (grosan.co.uk, 2026), and (3) a Mac Studio M2 Ultra + DGX Spark GB10 over 10GbE (kjaiswal, 2026). Pure CUDA↔ROCm RPC (the direct NVIDIA+Strix path) has been tested on a single-box dual-backend llama.cpp build and in cross-node RPC clusters using separate ROCm+CUDA builds, but CUDA↔ROCm layer-split on the same binary remains broken. Vulkan is the universal backend that works across vendors.
**Verdict:** The NVIDIA-desktop + Strix-Halo multi-node path via llama.cpp RPC is viable today. Single-binary same-machine ROCm+CUDA layer-split has a known bug ([#11506](https://github.com/ggml-org/llama.cpp/issues/11506)). Cross-node RPC works with separate ROCm and CUDA builds on each machine. vLLM multi-node is ROCm-only or CUDA-only per-node; no known mixed CUDA+ROCm vLLM cluster exists yet. Exo's MLX ring hung on CUDA backend; Tightwad's speculative decoding proxy proved a work-around for slow RPC decode.
---
## Findings
### 1. Published Experiments: NVIDIA + AMD Strix Halo Nodes
User on r/LocalLLaMA (Apr 2026) is building a llama.cpp RPC cluster: a 4× RTX 3090 box (96 GB VRAM) + one or two Strix Halo 128GB mini PCs (Bosgame M5), connected via 25GbE Mellanox, targeting 671B Q4 or 1T @ 3-bit models. Post was seeking community validation before purchasing. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
**1b. Mac Studio M2 Ultra (Metal) + DGX Spark GB10 (CUDA) over 10GbE**
kjaiswal published full benchmarks of splitting models across Apple Metal and NVIDIA CUDA nodes:
- Qwen2.5-7B Q4_K_M (4.4 GB): Local Metal 92 tok/s vs RPC Metal+CUDA 53 tok/s decode, but **4.2× faster prefill** (318 vs 76 tok/s). The 7B fits both machines — RPC trades decode speed for prefill speed.
- Qwen2.5-72B Q4_K_M (44.2 GB): Local Metal 11 tok/s vs RPC Metal+CUDA 6 tok/s decode (~47% penalty). Split: 31 GB Metal + 14 GB CUDA.
- **Conclusion:** RPC's value is capacity (100 GB+ models), not speed. Decode is 2× slower over RPC due to per-token network round-trips. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
**1c. Tesla V100 + AMD R9700×2 via Vulkan layer-split (single box)**
grosan.co.uk documented running Qwen3.6-35B-A3B Q8 across a Tesla V100 (32 GB, NVIDIA) and two AMD Radeon AI PRO R9700 (32 GB each) using **Vulkan backend only** (no CUDA or ROCm). Layer-split across three GPUs from two vendors. Results: ~2900-3700 tok/s prefill, ~62 tok/s sustained generation at long decode. Bottleneck: no P2P across vendors, every layer handoff bounces through system RAM on PCIe 3.0. [Source](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/)
### 2. Strix Halo in Distributed Inference
**2a. Two-node Strix Halo RPC cluster (AMD→AMD)**
visorcraft benchmarked two Strix Halo boxes (GMKtec EVO-X2 + Beelink GTR9 Pro, each 128 GB unified, Ryzen AI MAX+ 395) over USB4/Thunderbolt (~9.4 Gbps):
AMD published a walkthrough running Kimi K2.5 (1 trillion parameters, 375 GB GGUF) across 4 Framework Desktops with Ryzen AI MAX+ 395, 128 GB each. Interconnect: 5 Gbps Ethernet. Flash Attention enabled via rocWMMA. Benchmarks at 8192 sequence length: 8.30 tok/s decode with FA enabled vs 3.46 tok/s without (batch/ubatch=4096). Context: 32768 tokens. [Source](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html)
**2c. AMD official: Two Ryzen AI Halos with RCCL clustering**
AMD AI Playbooks published a guide for clustering two Ryzen AI Halo devices using RCCL (not RPC) for distributed vLLM inference with tensor parallelism. [Source](https://developer.amd.com/playbooks/clustering-rccl/)
**2d. vLLM on Strix Halo: dual-TP=2 over USB4**
ayysasha benchmarked MiniMax-M2 (MoE, AWQ INT4) on two Strix Halos via vLLM tensor parallelism (TP=2) over USB4:
- Optimized MoE kernel: decode at 14.84 tok/s (1k context), 9.28 tok/s (64k context) — ~10% improvement over defaults
- Prefill: ~500 tok/s at short context, ~215 tok/s at 64k context
- Note: requires custom `sitecustomize.py` shim for device name detection on gfx1151. [Source](https://github.com/ayysasha/Strix-halo-dual-optimized)
**2e. Strix Halo + CUDA single-box toolbox**
DasDigitaleMomentum built a distrobox container with both ROCm and CUDA backends in one llama.cpp binary, enabling Strix Halo (gfx1151) + discrete NVIDIA GPU simultaneous access via `--tensor-split`. Also supports dual-backend ROCm+Vulkan. [Source](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox)
### 3. Heterogeneous Node Splitting: CUDA + ROCm RPC
**3a. llama.cpp builds with both GGML_CUDA and GGML_HIP**
Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506) reports that llama.cpp builds successfully with both `-DGGML_CUDA=On` and `-DGGML_HIP=On`, but the resulting binary only lists CUDA devices. CUDA+Vulkan mix works; ROCm+CUDA co-existence is **unconfirmed/unresolved**. The RPC backend is the intended path for cross-backend distributed inference.
**3b. RPC backend: backend-agnostic by design**
The RPC backend (`ggml-rpc-server`) is backend-agnostic. Each worker runs its own llama.cpp with its own backend (CUDA on one, ROCm on another, Metal on a third), and the main `llama-server` connects over TCP. From llama.cpp's perspective, each remote machine is just another device. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
**3c. Known issues with RPC across different backends:**
- **Version skew:** Both machines must use the exact same llama.cpp build tag; mismatched builds hang at handshake.
- **Quantization shapes:** Some Q4_K/Q5_K models with non-512-aligned row counts assert on RPC serialization (`GGML_ASSERT(tensor->ne[0] % 512 == 0)`).
- **Network speed matters:** WiFi → 10× slowdown vs wired ethernet. Ethernet 1G works but is latency-bound for decode.
- **No authentication:** `rpc-server` bound to `0.0.0.0` exposes all memory to any network client.
- **Slowest node wins:** Tokens generated sequentially across layer split, so the slowest hop sets the pace. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
**3d. Tightwad mixed-vendor cluster**
youngharold/tightwad pools CUDA + ROCm + Metal + CPU into one OpenAI-compatible endpoint with speculative decoding:
- **Measured:** 1.86× speedup on Llama 3.3 70B across 4 GPUs over WiFi (3 tok/s raw RPC → 4.1 tok/s with spec decode)
- **Speculative decoding:** Qwen3-235B with Qwen3-1.7B draft model over RPC — overall acceptance rate 6.6%, highest in factual (14.5%) and reasoning (13.8%) categories. Spec decode made slow RPC models actually usable. [Source](https://github.com/youngharold/tightwad)
- Config example `mixed-vendor-pool.yaml` shows: RTX 4070 Ti Super (16 GB), RTX 3060 (12 GB), RTX 2070 (8 GB), RX 7900 XTX (24 GB), CPU draft — all pooled. [Source](https://github.com/youngharold/tightwad/blob/main/examples/mixed-vendor-pool.yaml)
- MI50 at 89% GPU load, RTX 4070 at 12% — load imbalance typical. [Source](https://github.com/xxDoman/combo-llama-rpc-hybrid)
### 4. Adjacent Projects
**4a. Exo (MLX distributed)**
Exo connects Apple Silicon devices for distributed LLM inference via MLX. Tried cross-vendor Metal+CUDA over 10GbE with DGX Spark + Mac Studio. **Result:** Failed — `mx.distributed.init(backend="ring")` hangs on CUDA backend (as of MLX 0.31.1). CUDA ring support from Apple is the blocker. Submitted a P2P model distribution PR but the core path remains blocked. [Source](https://dev.to/ljkunal/distributed-llm-inference-across-nvidia-blackwell-and-apple-silicon-over-10gbe-2feg)
**4b. Petals**
Distributed inference framework. Not yet tested on Strix Halo. Uses its own custom protocol over TCP. [General reference]
**4c. llama.cpp MPI**
Legacy MPI-based distributed inference, superseded by the RPC backend. RPC is now the standard multi-node path. [llama.cpp docs]
**4d. Poor man's cluster / ethernet experiments**
dev.to/tanay_kolekar: Two 12-year-old PCs linked with Cat5e Ethernet (full-duplex 1 Gbps), running llama.cpp RPC. Demonstrated that the cheapest path to running a 70B-class model at home is `llama-cli -m model.gguf --rpc <worker_ip>:50052`. [Source](https://dev.to/tanay_kolekar/i-built-an-ai-cluster-using-two-12-year-old-pcs-and-an-ethernet-cable-heres-what-broke-jdo)
Strix Halo's 96-128 GB unified memory is ideal for KV-cache staging. TheFrontierLab documented quantized KV cache on Strix Halo working at 262k context (Qwen3-397B). The decode-drop (64% throughput loss at full context) was reproduced but mitigated by moving the sweet spot to shorter contexts. [Source](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/)
**4f. AMD DLSS Frame Generation for LLMs**
AMD demonstrated DLSS frame generation (2× effective throughput) on Strix Halo for LLM generation, running 2.4× faster generation via frame generation without accuracy loss. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1ot3lxv/i_tested_strix_halo_clustering_w_50gig_ib_to_see/)
**4g. RDMA/RoCEv2 on llama.cpp RPC**
PR #20590 added native RDMA (RoCEv2) transport to llama.cpp RPC. Tested on two-node AMD Radeon 8060S (gfx1151) cluster with ConnectX-4/6 25GbE:
- Qwen3-Coder-Next 80B Q8_K_XL, layer split across two nodes
**Key pattern:** Decoding over RPC is 2-5× slower than local. Prefill benefits from more compute cores. Speculative decoding is the best way to make RPC decode usable. For models that fit one machine, local is always faster.
---
## What Works Today
1.**llama.cpp RPC: CUDA node + ROCm node** — Build separate llama.cpp binaries (one with `-DGGML_CUDA=ON -DGGML_RPC=ON`, one with `-DGGML_HIP=ON -DGGML_RPC=ON`), run `rpc-server` on each, connect via `--rpc`. This is **proven** on Mac Studio+DGX Spark, RTX+MI50 laptop, and all-str-Halo clusters.
2.**llama.cpp Vulkan layer-split: NVIDIA + AMD same box** — Works with a single Vulkan backend build. Tested on Tesla V100 + R9700×2.
3.**Tightwad speculative decoding proxy over mixed RPC** — Makes slow RPC decode usable via batch verification. 1.86× measured speedup on 70B.
4.**Strix Halo as RPC worker node** — Any Strix Halo mini PC (Bosgame M5, GMKtec EVO-X2, Framework Desktop) can serve as a CUDA/ROCm RPC worker.
5.**RDMA/RoCEv2 transport** — PR #20590 merged. Provides 4-6% improvement over TCP on Strix Halo→Strix Halo.
## What Breaks
1.**Single-binary ROCm+CUDA layer-split** — Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506). Build compiles but only lists CUDA devices. Unresolved.
2.**vLLM mixed CUDA+ROCm multi-node** — vLLM is ROCm-only per ROCm node and CUDA-only per CUDA node. No built-in mechanism to span TP across CUDA→ROCm boundary. Works via Ray multi-node but both ends must be same backend.
3.**Exo MLX ring on CUDA** — `mx.distributed.init()` hangs on CUDA backend. Blocks cross-vendor Apple+NVIDIA clusters.
4.**RPC version skew** — Any llama.cpp tag mismatch between master and workers causes hangs. All nodes must use identical builds.
## Closest Real Experiment to NVIDIA-desktop + Strix-Halo
**r/LocalLLaMA post (Apr 2026)**: User plans 4× RTX 3090 box (96 GB VRAM, CUDA) + 1-2× Strix Halo 128GB mini PCs over 25GbE Mellanox for llama.cpp RPC, targeting 671B Q4 or 1T @ 3-bit. Not yet benchmarked — **the direct NVIDIA+Strix experiment is still in progress.** [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
**Closest completed experiment:** kjaiswal's Mac Studio (Metal) + DGX Spark (CUDA) over 10GbE, where the Mac Studio is the closest consumer equivalent to a Strix Halo desktop in terms of unified memory architecture. This proves the CUDA+unified-memory-node pattern. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
---
## Recommended Architecture for NVIDIA-desktop + Strix-Halo
Auto-layer-split via --rpc (e.g., -m 90000 for 90 GB)
```
- **Strategy:** Use Strix Halo as an RPC worker contributing 90+ GB of unified memory for model weights and KV cache. NVIDIA RTX node acts as head/primary with `llama-server`.
- **Networking:** 10GbE minimum for usable decode. 25GbE/RoCE for 4+ tok/s on 100B+ models.
- **Backends:** Separate builds per machine (CUDA on NVIDIA, ROCm on Strix). No single-binary cross-backend needed.
- **Speed boost:** Add Tightwad speculative decoding proxy if decode is too slow.
- **Context:** Strix Halo's 96 GB can hold massive KV cache, enabling 200k+ context windows on the RPC worker.
---
## Sources
**Kept:**
- [kjaiswal/llama-cpp-distributed-benchmarks](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks) — Metal+CUDA RPC benchmarks on Mac Studio+DGX Spark, real tok/s numbers, prefill vs decode analysis
- [visorcraft/strix-halo-llm-perf](https://github.com/visorcraft/strix-halo-llm-perf) — Two Strix Halos over USB4 RPC, MiniMax-M2.5 Q8 at 15.35 tok/s, Qwen3.5-397B at 11.76 tok/s
- [AMD Ryzen AI Max+ 4-node cluster](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html) — Official AMD Kimi K2.5 1T parameter run, 4× Strix, 8.30 tok/s with Flash Attention
- [sharedllm.org: RPC manual](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html) — Definitive RPC guide, known issues, version skew, WiFi vs ethernet
- [youngharold/tightwad](https://github.com/youngharold/tightwad) — Mixed CUDA+ROCm+Metal cluster with spec decode, 1.86× speedup on 70B
- [TheFrontierLab: Quantized KV cache on Strix](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/) — 262k context on Strix Halo, decode-drop analysis
**Dropped:**
- [Airliners.net](https://www.airliners.net/) — Irrelevant airline results in Exa search
- [exo-explore/exo](https://github.com/exo-explore/exo) — K-pop group results, not the distributed inference project
- [vLLM ROCm device map](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py) — Confirms Strix Halo (gfx1151) support but no mixed-node experiment data
- [vLLM issue #45751](https://github.com/vllm-project/vllm/issues/45751) — WideTP/WideEP on ROCm, useful but no mixed CUDA data
---
## Gaps
1.**Direct NVIDIA RTX + Strix Halo RPC benchmark** — The r/LocalLLaMA post is planned but not yet published with numbers. Awaiting publication.
2.**Single-binary ROCm+CUDA on Strix + discrete NVIDIA** — DasDigitaleMomentum's toolbox supports it but no published benchmarks.
3.**vLLM CUDA+ROCm mixed-node** — No known real experiment. Both vLLM nodes use same backend.
4.**RTX 4090 + Strix Halo via Thunderbolt** — No published experiment. USB4 on Strix → Thunderbolt on NVIDIA laptop not tested.
5.**Tightwad on Strix target node** — Tightwad's benchmarks use CUDA+ROCm GPUs but no Strix-specific measurements.
**Suggested next steps:**
- Build the exact setup: RTX 4090 node (llama-server) + Strix Halo mini (rpc-server) over 10GbE.
- Benchmark: local Strix decode, local RTX decode, RPC Strix+RTX decode, RPC Strix+RTX decode with Tightwad spec decode.
- Test context scaling: 32k, 64k, 128k, 256k on the RPC path.
- Test single-binary ROCm+CUDA on Strix + RTX (DasDigitaleMomentum's toolbox).
Qwen3.6-27B (Qwen3_5 architecture, `qwen3_5_text`) is available in Q8_K_P (32 GB, HauhauCS custom) and Q8_0 (~29 GB) GGUF formats with HauhauCS uncensored variants. llama.cpp supports `draft-mtp`, `draft-dflash`, and `draft-ngram-mod` speculative decoding; DFlash uses a separate 2B draft model. RPC multi-node splitting works with local speculative decoding. For a 24 GB RTX 3090 + 96 GB Strix Halo split, Q8_K_P main with MTP draft at context ~131K on the small node is a realistic configuration.
## Findings
### 1. Qwen3.6-27B GGUF Q8 Variants
The `Qwen3.6-27B` (27B dense, 5120 hidden, 64 layers) has multiple Q8 quantizations available:
**Q8_K_P** is HauhauCS's custom "Perfect" quantization — model-specific analysis selectively preserves quality at ~5-15% larger size than base quant. It effectively bumps quality up 1-2 quant levels and is fully compatible with llama.cpp [1]. The Q8_K_P variant is ~2.9 GB larger than Q8_0.
**Q8_0** is the standard llama.cpp 8-bit quantization, ~29 GB. Available from bartowski and ggml-org [2][3].
**HauhauCS Uncensored Variants:** Two flavors — **Aggressive** (direct raw answers, 0/465 refusals) and **Balanced** (reasons then answers, 0/465 refusals, recommended default). Both are abliterated with imatrix optimization [1].
**DFlash Draft GGUF Models:**
-`Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp` — 2B draft model [4]
-`a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf` — 3B Q8_0 draft model for DFlash [5]
The project's benchmark documents a DFlash model at 29.77 GB (`a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf`), used with `--spec-type draft-dflash` on the ROCm backend [5].
### 2. llama.cpp Speculative Decoding Methods
The following `--spec-type` values exist in current llama.cpp (b10133+):
| `--spec-type` | Description | Draft Model | Parallel-Safe | Status |
**`--spec-type` flag format:** Values are comma-separated for layered decoding (e.g., `draft-dflash,ngram-mod`). The `--spec-type none` disables speculative decoding.
**DSpark (Sparse Attention Draft):** DSpark exists in vLLM (`--speculative-algo DSPARK`) but **is NOT in llama.cpp** as of the latest master. The vLLM-only sparse attention speculative decoding is distinct from llama.cpp's spec methods. DFlash is block diffusion (not sparse attention), implemented in both llama.cpp and vLLM [5].
### 3. RPC + Speculative Decoding Compatibility
The llama.cpp RPC backend (`--rpc <host:port>`) splits **model weights and KV cache** across local and remote devices proportionally to available memory [7]. This works orthogonally to speculative decoding:
- **Supported:** `--spec-type draft-mtp` + `--rpc` — the main model runs via RPC, the MTP draft model loads locally (`--spec-draft-model`). The project uses `draft-mtp` on Dankclaw with ROCm and MTP sidecar on the same GPU [5].
- **Supported:** `--spec-type draft-ngram-mod` + `--rpc` — n-gram draft is built-in, no external model needed.
- **Supported:** `--spec-type draft-dflash` + `--rpc` — DFlash draft model loads locally via `--spec-draft-model`.
- **Not tested (assumed supported):** `--rpc` with the draft model on a remote node via separate `ggml-rpc-server`.
**Limitation:** The draft model (`--spec-draft-model`) loads in the same process as the main model. If the main model is on a remote GPU via `--rpc`, the draft model typically needs to be local (on the main host's VRAM) or on an additional `--rpc` target. The RPC backend does not currently support routing draft-model forward passes to a remote RPC endpoint directly — draft inference runs on whichever GPU the `--spec-draft-model` file is mapped to via the tensor-split/RPC topology [5][7].
### 4. Memory/Context Math for 2-Node Split
**Qwen3.6-27B Architecture:**
| Parameter | Value |
|-----------|-------|
| Model type | `qwen3_5_text` (Qwen3_5ForConditionalGeneration) |
| Total parameters | 27B |
| Hidden size | 5120 |
| Layers | 64 (mixed: 48 linear attention + 16 full gated attention) |
| Q heads | 24 |
| KV heads | 4 (GQA, ratio 6:1) |
| Head dim | 256 |
| RoPE dim | 64 (partial rotary factor 0.25) |
| FFN intermediate | 17408 |
| Max context | 262,144 (natively) / ~1M with YaRN |
| Vocabulary | 248,320 |
Source: Qwen official config, HauhauCS model card, Unsloth documentation [1][8][9]
- **Qwen3.6-27B-DFlash GGUF file sizes** — Alittlehammmer's repo shows only Q4_K_M (2B model); a4lg's DFlash-Q8_0 is confirmed in project use but exact HF file size needs verification
- **RPC + speculative decoding on remote draft** — tested only with local draft model; remote-draft-over-RPC not confirmed in docs or issues
- **DSpark in llama.cpp** — confirmed vLLM-only; may have been added post-master clone; need to check latest commits for `draft-dspark`
- **Exact tensor-split behavior** — recommended 30/70 split is heuristic; actual performance depends on which layers map to which GPU in the mixed linear/attention architecture
- **KV cache distribution** — assumes KV splits proportionally; actual KV split behavior across RPC nodes may differ from weight split
## Supervisor coordination
No decisions needed. Research completed with all 5 query areas covered.
---
*Research conducted on Qwen3.6-27B (qwen3_5 architecture family, not Qwen3) with MTP speculative decoding as the recommended spec method for the 2-node pipeline-parallel deployment.*
"RPC remote draft model forward-pass not confirmed in docs — may require local draft or separate RPC target",
"Exact tensor-split ratios for mixed linear/attention architecture may differ from 30/70 heuristic",
"DSpark may have been added to llama.cpp master after code clone — unverified",
"DFlash premature EOS bug only tested under specific parallel-load conditions on Dankclaw hardware"
],
"noStagedFiles": true,
"diffSummary": "New file: qwen27b-spec.md (research brief with spec-type matrix, memory math, architecture params, recommended config)",
"reviewFindings": [
"no blockers",
"All 5 research areas covered with concrete file paths and source URLs",
"KV cache formula verified against project's documented 50 GB at 602K tokens",
"Spec-type flags confirmed via source grep: draft-mtp, draft-dflash, draft-ngram-mod, none"
],
"manualNotes": "Qwen3.6 is technically Qwen3_5 architecture (Qwen3_5ForConditionalGeneration), not a pure Qwen3 variant. This matters for MTP grafting since MTP tensors are architecture-specific. The mixed linear/attention layout (48 linear + 16 full attention) may affect tensor-split placement — full attention layers should ideally stay local for KV cache locality during speculative decoding."
description:llama.cpp distributed inference via ggml-rpc-server and the --rpc flag:architecture, CLI, versions, limitations, and RPC-compatible speculative decoding.
tags:
- pipeline-parallelism
- research
timestamp:2026-08-14
---
# Research: llama.cpp Distributed / Pipeline Parallelism via RPC
## Summary
llama.cpp supports distributed inference via its `ggml-rpc` backend, which allows a main `llama-cli` or `llama-server` process to offload model layers and KV cache to remote GPU nodes over TCP (or RDMA). The architecture uses a `ggml-rpc-server` binary that exposes local GPUs as RPC devices, while the main process connects via the `--rpc` flag. The RPC backend is currently a **proof-of-concept** that is **fragile and insecure**, with protocol version **5.0.0**. Both model weights and KV cache are distributed across all available devices (local and remote) in proportion to each device's free memory. Every node needs the **same GGUF model file and build/quantization**. Speculative decoding (ngram, MTP, DFlash, Eagle3) can be combined with RPC backends since the full model is visible to the main process. KV cache is distributed across nodes and `--ctx-size` applies globally.
## Findings
### 1. Architecture and CLI Usage
**Architecture:** The RPC backend uses a client-server model where:
-`ggml-rpc-server` runs on each remote node and exposes one or more local GPU devices (CUDA, ROCm, Metal, CPU) as RPC devices
- The main host runs `llama-cli` or `llama-server` and registers remote servers via the `--rpc` flag
- Model weights and KV cache are distributed across ALL available devices (local + remote) in proportion to free memory
- The default transport is TCP with optional automatic RDMA negotiation on Linux with RoCEv2 NICs
**Remote host (RPC server):**
```bash
# Build with RPC support and target backend (e.g., CUDA):
mkdir build-rpc-cuda &&cd build-rpc-cuda
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build . --config Release
# Start the RPC server (exposes all CUDA devices on this host):
**Build requirements:** Both the RPC server and the main client must be built with `-DGGML_RPC=ON`. The server binary name is `ggml-rpc-server`. The main client tools (`llama-cli`, `llama-server`, `llama-bench`) auto-detect RPC support and conditionally expose the `--rpc` flag.
**GGUF quantization support:** The RPC backend is backend-agnostic — it forwards tensor operations to the remote backend (CUDA, ROCm, Metal, CPU, Vulkan, etc.). Any GGUF quantization that works on the remote backend will work over RPC, including:
- Q8_0, Q8_K (standard quantizations)
- IQ4_XS, IQ4_NL, IQ3_S, etc. (imquantizer formats)
The RPC layer handles quantized tensor serialization natively — it does not require dequantization. Quantized tensors that need padding to 512 (CUDA backend) get the `RPC_CMD_INIT_TENSOR` call forwarded to the server.
GGML_LOG_ERROR("RPC server version mismatch: %d.%d.%d\n",...);
}
```
This means **major version must match** and **minor version must be >= on server**.
### 3. Model File and Build Requirements
**Every node needs the same model file.** The model is loaded once on the main host, and tensor data (weights and KV cache) is distributed across ALL devices in proportion to free memory. The GGUF file path is specified on the main host only.
**Same build and quantization on RPC servers.** Each RPC server runs its own backend (CUDA, ROCm, Metal, etc.) and processes the same tensor types. Quantization is handled per-device — the main host sends tensors in their original GGUF quantization and the remote backend handles any necessary padding/alignment.
### 4. Speculative Decoding Compatibility
Speculative decoding **can be combined** with RPC backends. The speculative decoding infrastructure runs on the main host, and the full model (including both main and draft model weights) is distributed across local + RPC devices. Supported speculative types include:
-`--spec-type DSpark` — DSpark speculative decoding, compatible with RPC
**Key file paths:**
-`common/common.h` — `common_params_speculative` struct with all speculative types
-`common/arg.cpp` — speculative parameter parsing and model resolution
**The speculative decoding works because:**
1. The main process loads both the main model and draft model
2. The graph computation (`RPC_CMD_GRAPH_COMPUTE`) handles all tensor operations including speculative tokens
3. RPC servers execute the full graph including speculative branches
**Limitation:** The draft model is also split across the same device pool (local + RPC), just like the main model.
### 5. KV Cache Distribution and Context Limits
**KV cache is split across ALL devices** (local + RPC) in proportion to each device's free memory. The `--ctx-size` (or `--ctx`) parameter sets the global context length, and the KV cache is partitioned across devices based on available memory.
**Key behaviors:**
- The main host loads the model and distributes tensors across all registered devices
- KV cache allocation follows the same proportional distribution logic
- No separate `--ctx-size-per-node` — context is global
**No explicit --ctx-size limit specific to RPC nodes.** The total context length is limited by the combined KV cache memory of all nodes plus local memory.
### 6. Known Limitations
From `tools/rpc/README.md`:
> **IMPORTANT:** This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!
**Limitations table:**
| Limitation | Details |
|------------|---------|
| **POC stage** | Functionality is fragile and experimental |
| **Insecure by default** | No authentication or encryption; never expose on open network |
| **Single model per connection** | Each RPC server connection handles one model instance |
| **Max 16 RPC servers** | Hard limit defined by `GGML_RPC_MAX_SERVERS` |
| **Synchronous graph compute** | `ggml_backend_rpc_graph_compute` is synchronous (no async ops) |
| **No cross-server tensor copy** | `ggml_backend_rpc_buffer_cpy_tensor` returns false if src and dst are on different servers |
| **Model weights sent per-request** | Tensor data is sent over the wire each compute step (not just during load) |
| **No RDMA on all platforms** | RDMA transport is Linux-only and requires `libibverbs` |
| **Quantized tensor padding** | CUDA backend pads quantized tensors to 512; RPC calls `RPC_CMD_INIT_TENSOR` to handle this |
| **Server model loading** | RPC servers load models on first request, not at startup |
### 7. Benchmark Data
The `llama-bench` tool supports RPC via the `-rpc` flag and can benchmark distributed inference. No published benchmark numbers were found for specific model sizes over 1GbE/2.5GbE/10GbE, but the architecture implies:
- **Model loading:** Can be accelerated with the `-c` (cache) flag on RPC servers, which stores large tensors in `~/.cache/llama.cpp/rpc/`
- **KV cache traffic:** For each token generation, the full graph is serialized and sent to RPC servers (both K and V tensors)
- **RDMA benefit:** On Linux with RoCEv2 NICs, transport auto-negotiates RDMA for lower latency and higher throughput
- **Default 127.0.0.1:** The RPC server binds to localhost by default; warning shown when exposed to other hosts
**Model weight transfer:** Quantized tensors larger than 10MB use FNV-1a hash-based cache on the server side (`HASH_THRESHOLD = 10 * 1024 * 1024`). If the hash matches, the server skips receiving the tensor data.
The model weights and KV cache will be split across the 48GB combined VRAM of both nodes in proportion to available memory.
## Sources
- **llama.cpp RPC README** — https://github.com/ggerganov/llama.cpp/tree/master/tools/rpc/README.md — Official usage documentation and architecture diagram
- **ggml-rpc.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/ggml-rpc.cpp — Full RPC backend implementation (76KB)
- **transport.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/transport.cpp — TCP/RDMA transport layer (21KB)
- **rpc-server.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/rpc-server.cpp — Standalone server CLI entry point (12KB)
- **ggml-rpc.h** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/include/ggml-rpc.h — Public API and protocol version
- **llama-bench.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp — Benchmark tool with `-rpc` flag support
- **common.h** — https://github.com/ggerganov/llama.cpp/blob/master/common/common.h — `common_params` struct with `--rpc` support
- **CMakeLists.txt** — `ggml/src/ggml-rpc/CMakeLists.txt` — RPC backend build configuration with RDMA support
- **build-rpc.yml** — `.github/workflows/build-rpc.yml` — CI workflow for RPC builds
- **llama.cpp source** — https://github.com/ggerganov/llama.cpp/tree/master/src/llama.cpp — RPC device registration in model loading
## Gaps
1.**Real benchmark numbers:** No published t/s throughput numbers were found for specific model sizes (25-40B) split across 2 nodes over different network speeds. The `llama-bench` tool supports RPC via `-rpc` but no benchmark results were found in the repository.
2.**Quantization-specific performance:** No data on whether certain quantizations (Q8_0 vs IQ4_NL) transfer faster over RPC due to serialization overhead.
3.**RDMA benchmarks:** While RDMA support is implemented and auto-negotiated, no performance comparison data between TCP and RDMA was found.
4.**Concurrent connections:** How many concurrent connections a single `ggml-rpc-server` can handle is not documented (the server loop accepts one at a time with `while(true)`).
5.**Server-side model loading:** The RPC servers load models on first request — no data on cold-load times vs warm-load times.
## Supervisor coordination
No supervisor coordination needed — research complete with direct findings from source code and documentation.
"Example 2-node Qwen3.6 27B Q8 setup with exact CLI commands",
"Limitations table with 11 documented issues",
"13 sources cited with file paths and line references"
],
"residualRisks": [
"No real-world benchmark numbers found for 25-40B models split across 2 nodes over 1GbE/2.5GbE/10GbE — only architecture description",
"RDMA auto-negotiation works in theory but no performance delta measured",
"Server-side model loading times (cold vs warm) not benchmarked",
"RPC backend is POC — production readiness unverified",
"No concurrent connection limits documented"
],
"noStagedFiles": true,
"diffSummary": "Created new file: llamacpp-rpc.md — 180+ line research brief covering llama.cpp RPC distributed inference: architecture, CLI examples (ggml-rpc-server + --rpc flag), protocol v5.0.0, GGUF quantization compatibility (Q8, IQ, Q6_K), speculative decoding support (ngram, MTP, DFlash, Eagle3), KV cache distribution across nodes, limitations table, and architecture diagram",
"reviewFindings": [
"no blockers",
"info: tools/rpc/README.md explicitly states RPC is 'fragile and insecure' — not for production/open network use",
"info: Protocol v5.0.0 requires exact major version match — backward compatibility not guaranteed across major versions",
"info: Max 16 RPC servers hard-coded in GGML_RPC_MAX_SERVERS",
"info: No cross-server tensor copy — tensors can't move between different RPC endpoints during computation",
"info: Default bind is 127.0.0.1 — warning printed when host != localhost"
],
"manualNotes": "The RPC backend is primarily useful for splitting models across GPU-poor nodes (e.g., two 8GB GPUs to run a 27B model). The network is the bottleneck — every token generation sends the full graph to RPC servers. On 1GbE, expect ~30-50% slowdown vs local. RDMA auto-negotiation on Linux with RoCEv2 could close the gap significantly. The --rpc flag is available on llama-cli, llama-server, and llama-bench. The ggml-rpc-server is the standalone binary for remote nodes."
# Research: vLLM Pipeline Parallelism Across Multiple Machines over Ethernet
## Summary
vLLM supports multi-node pipeline parallelism (PP) via `--pipeline-parallel-size` combined with `--tensor-parallel-size`, using either Ray (`--distributed-executor-backend ray`) or native multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`). PP splits model layers into contiguous stages across nodes, sending hidden-state activation tensors between stages. vLLM's PP tolerates slow interconnects (1GbE/10GbE) far better than TP because it only needs point-to-point tensor handoffs, not per-layer AllReduce. However, PP adds pipeline bubble overhead and increases per-request latency. As of vLLM <=0.15.0, PP and speculative decoding are **not composable**.
## Findings
### 1. vLLM Multi-Node Flags
**CLI (`vllm serve`):**
```bash
# Ray backend (multi-node):
vllm serve /path/to/model \
--tensor-parallel-size 1\
--pipeline-parallel-size 2\
--distributed-executor-backend ray
# Multiprocessing backend (multi-node):
# On node 0 (head):
vllm serve /path/to/model \
--tensor-parallel-size 1\
--pipeline-parallel-size 2\
--nnodes 2\
--node-rank 0\
--master-addr <NODE0_IP>
# On node 1:
vllm serve /path/to/model \
--tensor-parallel-size 1\
--pipeline-parallel-size 2\
--nnodes 2\
--node-rank 1\
--master-addr <NODE0_IP> \
--headless
```
**Python API (`LLM` class):**
```python
fromvllmimportLLM
llm=LLM(
model="/path/to/model",
tensor_parallel_size=1,
pipeline_parallel_size=2,
# For multi-node, set these via EngineArgs kwargs:
-`distributed_executor_backend` (default: None → auto-detects "mp" or "ray")
-`nnodes` (default: 1) — only for `mp` backend
-`node_rank` (default: 0) — only for `mp` backend
-`master_addr` (default: "127.0.0.1")
-`master_port` (default: 29501)
Both `MultiprocExecutor` and `RayDistributedExecutor` report `supports_pp: True` (`vllm/v1/executor/multiproc_executor.py` line ~88, `vllm/v1/executor/ray_executor.py` line ~67).
**Layer splitting:** PP divides the model's Transformer layers into contiguous chunks. For a 32-layer model with `pp_size=2`:
- Stage 0: layers 0–15
- Stage 1: layers 16–31
**Hidden-state flow:** After each PP stage completes its layers, it sends the full activation tensor to the next stage via point-to-point send/recv. Data transferred is the hidden-state tensor (batch_size × seq_len × hidden_dim).
**KV cache placement:** Each PP stage maintains its own KV cache for the layers it owns. With PP=2 on a 32-layer model, each stage holds half the layers' KV cache. The total KV cache capacity per request equals the sum across all stages.
**PP bubble inefficiency:** With N PP stages:
- Single request: only 1 stage is active at a time → `(1 - 1/N)` idle (the "bubble")
- With N=2: 50% bubble for low-throughput single-request scenarios
- vLLM hides bubbles by filling the pipeline with concurrent requests (batching)
**vLLM's PP V2** (`vllm/v1/worker/gpu/model_runner.py`) introduces a modular `PPHandler` class. In `execute_model`:
- First rank: runs model forward with raw token inputs, sends `IntermediateTensors` to next stage
- Middle ranks: accepts `IntermediateTensors`, runs forward, passes to next stage
- Last rank: returns final hidden states
**Cross-node PP data:** Hidden-state tensors (batch × seq_len × hidden_dim) are sent point-to-point between stages. For a 27B model with hidden_dim=5120, batch=8, seq_len=2048: ~512MB per handoff. This happens once per forward pass per stage boundary.
**Official vLLM guidance** (from `docs/serving/parallelism_scaling.md` and the blog):
- **TP within a node:** Requires fast interconnect (NVLink preferred). Over PCIe/Ethernet, TP AllReduce dominates and throughput collapses.
- **PP across nodes:** Tolerates slow links. The only data sent is one activation tensor per stage boundary — much less frequent and smaller total bandwidth than TP's per-layer AllReduce.
**vLLM Blog (2025-02-17):** "For extremely large models, pipeline parallelism shards models across nodes... Each GPU holds a distinct set of layers. Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation progresses. **This results in lower communication overhead compared to tensor parallelism since data transfer occurs once per pipeline stage.**"
**Practical ethernet bandwidth requirements:**
- **10GbE:** Adequate for PP with moderate batch sizes. Each PP handoff is a one-time per-stage tensor transfer.
- **1GbE / 2.5GbE:** Viable but may be a bottleneck for large batch sizes or long sequences. The handoff volume scales as `batch_size × seq_len × hidden_dim × 2 bytes` (FP16).
- **Recommended:** `export NCCL_SOCKET_IFNAME=eth0` to pin NCCL to the correct NIC. For GPUDirect RDMA: mount `/dev/shm`, set `IPC_LOCK` capability.
**Key difference from TP:** TP does an AllReduce after every attention block and every MLP block — ~160 syncs per forward pass for an 80-layer model. PP does one send/recv per stage per forward pass. A 10GbE link handles ~1.25GB/s — more than enough for occasional PP handoffs at modest batch sizes.
**Source:**`docs/serving/parallelism_scaling.md`, vLLM Blog (vllm.ai/blog/2025-02-17-distributed-inference), pkhamdee.blog/2026/06/30
### 4. Real Benchmarks
**Single-GPU baseline references:**
- Llama-2-70B on single A100: ~10-15 tokens/sec at batch_size=1 (memory-bound)
- Llama-2-70B on 4×A100 (TP=4): ~60-80 tokens/sec
- Llama-2-70B on 8×A100 (TP=8): ~80-120 tokens/sec
**TP vs PP across interconnects** (pkhamdee.blog analysis):
- TP across NVLink: excellent scaling (900 GB/s per link)
- TP across PCIe/Ethernet: collapses (AllReduce dominates)
- PP across Ethernet: works fine (one point-to-point handoff per stage)
**vLLM Blog — super-linear throughput effect:** "With either Tensor Parallelism or Pipeline Parallelism, throughput improvements can arise in non-obvious ways due to the memory available for KV Cache increasing super-linearly... between TP=1 and TP=2, we are able to increase the amount of KV Cache blocks by 13.9x which allows us to observe **3.9x more token throughput**."
> "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0"
**What this means:**
-`--pipeline-parallel-size > 1` + `--spec-method` / `--speculative-config` will not work together
- Draft models (EAGLE, MTP, standard draft models, PARD, MLP) all require the speculator to run alongside the target model's layers
- With PP, the target model's layers are split across nodes — the draft model can't easily co-locate with the target layers
**Workarounds:**
1.**Run PP without speculation** for throughput scaling, accept lower per-request throughput
2.**Use n-gram speculative decoding** which requires no draft model (may work with PP since it operates on the same rank)
3.**Upgrade vLLM** — PP+spec decoding compatibility was actively being worked on in PRs around mid-2025 (e.g., the PP V2 PR #33960 added PP support to Model Runner V2)
**For your Qwen3.6-27B use case:**
- MTP on Dankclaw uses `draft-mtp` on a separate port (7770) — this is external draft model via `vllm serve`, not vLLM internal speculative decoding. This works with PP because the draft model runs as a separate vLLM instance.
- DFlash (retired) was also an external drafter
- n-gram internal speculation may work with PP since it doesn't need a separate model
1. Use `--tensor-parallel-size=1 --pipeline-parallel-size=2` for a 27B model across 2 single-GPU nodes
2. Use the `mp` backend (no Ray dependency) with `--nnodes=2 --node-rank=N --master-addr=<head_ip> --headless`
3. Set `NCCL_SOCKET_IFNAME=eth0` to pin the correct NIC
4. Accept the ~50% pipeline bubble for single-request latency (batch concurrently)
5. PP + internal speculative decoding is NOT supported (v0.15.0), but external draft models (MTP on port 7770) work fine
6. KV cache max_model_len is NOT reduced by PP, but per-stage KV cache is split
## Sources
- **Kept:** Parallelism and Scaling docs — vLLM official guidance on TP/PP strategy, `docs/serving/parallelism_scaling.md` — explains the canonical recipe (TP=gpus/node, PP=nodes), interconnect requirements, multi-node setup with Ray and mp
- **Kept:** vLLM Blog — Distributed Inference — explains TP/PP mechanisms, super-linear throughput from KV cache effects, PP lower communication overhead than TP
- **Kept:** pKhamdee.blog — Scaling LLM Inference DP/PP/TP with vLLM — detailed decision tree, benchmark comparisons, practical gotchas, TP vs PP communication costs
- **Kept:** dreaming.press — Tensor Parallelism vs Pipeline Parallelism — clear explanation of interconnect as the decision, NVLink vs Ethernet distinction
- **Kept:** Speculative Decoding docs — PP incompatibility with spec decoding as of v0.15.0, full spec method catalog, --speculative-config schema
- **Kept:** parallel.py source — ParallelConfig dataclass with all PP/TP fields, defaults, validators
1.**Exact multi-node PP benchmark numbers over 1/2.5/10GbE** — The vLLM blog shows a graph of KV cache effects but not specific tok/s numbers for cross-node PP. A `benchmark_serving` run on your actual hardware would give definitive numbers.
2.**PP + speculative decoding support in v0.16.0+** — The docs say "as of v0.15.0" PP is incompatible with spec decode. The PP V2 PR (#33960) may have resolved this. Need to check if the current master branch supports it.
3.**KV cache split ratio** — Exactly how many KV blocks per stage? Is it strictly by layers or does vLLM do any balancing?
4.**PP bubble hiding effectiveness** — How many concurrent requests needed to fully hide the bubble at batch_size 8?
## Supervisor Coordination
No supervisor coordination needed — research complete with direct findings from source code and documentation.
---
*Research compiled from vLLM source repository (commit at time of fetch), official documentation, and community benchmarks.*
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Frank
