dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): documentation exists for point of use — destroy the ledgers (−13.7k lines) #31

Merged
dank merged 13 commits from docs/vault-destruction-20261019 into main 2026-10-19 10:59:51 -07:00
Conversation 0 Commits 13 Files Changed 157
+2344 -15158
157 changed files with 2344 additions and 15158 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+309 -486
View File
@@ -47,7 +47,7 @@ per attempt and no separate swap-history file.
The managed skill library is distilled there one page per skill-cluster:
`LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
`Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
pages. There is no `log.md`: the change diary was destroyed 2026-10-19
- `tests/` — bats suite + Playwright e2e
## Git Workflow — trunk-based (fleet law, aligned 2026-10-18)
@@ -318,35 +318,36 @@ the next), so never report request #1.
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.
## Documentation Policy (fleet law, 2026-10-18)
## Documentation Policy (fleet law, 2026-10-19)
**One fact, one owner.** A page owns exactly one thing — an endpoint, a
service, or a technique — and every other file links to it with `[[Page]]`.
Restating a shape, a digest, a port list, or a measured number in a second
file is a defect: it is the next thing to go stale.
**Documentation exists for point of use.** A page earns its bytes only if it
helps whoever performs the next task — the shape to deploy, the flag that must
be passed, the failure signature to recognise, the rollback to run. It is not a
lab notebook and not a ledger. **Losing information is an acceptable, usually
correct outcome.** Never let an orphan census ("this is the only copy of that
fact") decide what to keep: sole custody of a useless fact is still useless.
- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
— current shape, an **Eras** table (`window | payload | runtime | shape |
replaced because | rollback`), the **Laws** the endpoint taught, the wire
contract, and the files that own it. A payload swap edits the shape block
and appends one Era row. It never creates a page, and the retired era's
page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
table.
- **`log.md` records incidents and laws** — anything with a causal story
(why the host went down, why a knob resets at reboot, why the RAM was the
fault). A model swap is not an incident: it is an Era row and nothing else.
- **`AGENTS.md` carries operator-critical numbers only** — port, context
ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
- **The purpose test — before you write, and again before you keep:** would the
reader act differently because of this line? No → delete it.
- **Measurements are disposable.** A rate or throughput figure (t/s, tok/s, ms,
acceptance %, a bench ladder, a measured MiB) for a payload, runtime, or
layout that is not running now and not being compared now is destroyed, not
filed — not in an Eras row, not on a technique page, not "preserved for
history". AI moves faster than this vault: the number is wrong before anyone
reads it, and a wrong number in a runbook gets believed. Keep the **law** the
measurement taught; drop the measurement.
- **One fact, one owner**, every other file links with `[[Page]]`. The env file
beside the serve script is the authority for every live number; the endpoint
page is the authority for the shape and the wire contract.
- **One page per endpoint:** current shape, **Eras** (`window | payload |
runtime | replaced because | rollback` — the rollback is a commit, not a
number), the **Laws**, the wire contract, the files that own it. A swap edits
the shape block and appends one row; the retired era's own page is
**destroyed** to `.AGENTTRASH/`, never merged into anything.
- **`log.md` is retired** (2026-10-19: 2,380 lines destroyed). Git holds the
chronology. A causal law belongs to the page that owns the thing it governs,
or to *Key Lessons Learned* below if it is cross-cutting — never to a diary.
- **Index pages are locators:** one row, one clause, no numbers.
- **A measured number is written once:** in the env-file header beside the
serve script that consumes it, and in the endpoint page. Every other page
cites it by link.
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
technique pages: they stop carrying a per-endpoint throughput table or a live
shape, but a campaign ledger that exists nowhere else is owned by the page that
measured it. The defect is two pages carrying one figure, not a number living on a
technique page — move a figure only when an endpoint page or an env header is its
home; otherwise leave it in place and name the era it measured.
## Model Config (harness)
@@ -477,481 +478,302 @@ linked runbooks.
**PCIe / multi-GPU**
- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
them when the device has no atomics, and the failure surfaces as
`hipErrorIllegalState` ("the operation cannot be performed in the present
state") from `hipLaunchKernel`, not as anything that names atomics. The
value is consumed at driver probe time, so writing the AtomicOp Requester
bit with `setpci` at runtime changes nothing. Check the *chain*:
`lspci -vv -s <gpu>` and every upstream bridge must show
`AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
and the rest behind the chipset (usually not) — that asymmetry is what
blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
short-circuits at `world_size=1`.
- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
`PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
precondition the old test plan named. Bandwidth was never the constraint (a
27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
and under 0.4 ms at Gen5 x8).
**Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
frank-medium serves on it; llama.cpp's RCCL path was built too
(`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
device code (`invalid device function`), which is why R4D is the path vLLM
uses at all. Evidence matrix and the measured three-arm verdict:
[[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The
message that identified the cause came from `AMD_LOG_LEVEL=3`
(`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
prints the failing HIP call with its return code.
- **The software path that puts both cards on ONE model without a collective:
llama.cpp's layer split.** On the old board vLLM could not do it in any of its
parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
died on the card that could not dispatch an RCCL kernel. `--split-mode
layer` in llama.cpp needs no
collective: layers are distributed in order and only the boundary activation
crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
fallback).
This was what `frank-medium` ran until 2026-09-20, when the board swap made
collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
[[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
[[frank-medium-llamacpp-dflash2]].
- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
across devices: `--override-tensor <group>=<device>` sets
`has_tensor_overrides()`, a REQUIRED term of the scheduler's
`pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
that reads it), and that flag is what sets `n_copies` in
`ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
target that works, because the CPU backend is last in the list. Pure Vulkan
changes nothing here — the gate is the override, not the driver mix. See
[[x4-pair-experiment]].
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
ceiling — with async no better than sync and pageable within 1 % of pinned,
i.e. the copies are link-limited rather than staging-limited (the usual
"pageable costs 2x" rule does not apply here). The card is x16-capable and the
Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
`00:03.1`), so the x4 lives in the adapter or cable; the link idles at
2.5 GT/s x4 and trains up under load. Consequence for any engine that
*streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
20 % resident cache), against **20.46 tok/s** for the same model with the
experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
this streaming shape — also loads only modelopt block-FP8 safetensors for
`qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
the question. See [[x4-pair-experiment]].
- **RCCL collectives need PCIe atomics on every participating GPU, and the failure never names
atomics.** The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on
a device without atomics and the error surfaces as `hipErrorIllegalState` from `hipLaunchKernel`,
not as anything that names atomics. The value is consumed at driver PROBE time, so writing the
AtomicOp Requester bit with `setpci` at runtime changes nothing. Check the CHAIN: `lspci -vv -s
<gpu>` and every upstream bridge must show `AtomicOpsCap: Routing+` (a `Routing-` bridge is a
hardware wall), and the endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
failing HIP call with its return code. [[PCIe Link and Atomics]]
**ROCm / GPU**
- **Layer split needs no collective** — layers go in order and only the boundary
activation crosses PCIe, so it is the fallback wherever a collective cannot be
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to
a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned
draft under tensor mode aborts the KV build, and on a split its home is the
card holding the target's last layers. Deployed history:
[[frank-medium-llamacpp-dflash2]].
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
one directory level. rocBLAS and hipBLASLt look for their device code at a
fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
and the flattened copy fails at *request* time, not load time: the engine
reads the weights, answers `/health` and dies on the first completion with
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
: gfx1201`. Declare the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
`-ot` (23 → 13.4 t/s).
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve
scripts export env files with `set -a`, so a tunable named `TEMP` leaks
into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every
load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable
these; check with
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a trap.** With a missing
`DST`, cp creates `DST` as a copy of `SRC` and drops one directory level. rocBLAS and hipBLASLt
look for their device code at a FIXED depth (`../lib/rocblas/library`,
`../lib/hipblaslt/library/<arch>`), so the flattened copy fails at REQUEST time, not load time:
the engine reads the weights, answers `/health` and dies on the first completion with
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201`. Declare
the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
and pinning the PLE tensor with `-ot` was slower than letting it page. [[Llama Cpp Serving Sizing]]
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
`1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
tunable these; check with
`strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash
attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+
t/s, measured 2026-08-17).
- **Flash-Next MTP heads do not load on any current loader** — they need full
per-layer `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified
3x). Don't re-attempt without new loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
the agentionai FAST layer set; the kingjones STRIX set works unpatched.
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
116,732 MiB (`llama-server --list-devices`;
`gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
"64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
CPU-side carve-out — intentional, do not change.
- **`GGML_CUDA_FA_ALL_QUANTS` is dead; do not build for it.** Verified on this host 2026-10-19:
master's `ggml/cmake/common.cmake` warns it is deprecated in favour of `GGML_CUDA_FA_QUANTS`, the
successor's DEFAULT pair list (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`) already compiles the
quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old
macro anywhere under `ggml/src/ggml-cuda/`. The old "~12× prefill collapse" belongs to the older
source line. Prove a build's KV ability from its `CMakeCache.txt`. [[frank-zealot]]
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
`blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
loader support.
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
`/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
`free`/`dmidecode` show the CPU-side carve-out — intentional, do not change. [[Dankclaw System Specs]]
**Load / crash diagnosis**
- Model load is slow (minutes cold); health returns 503 while loading; do
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On
halogen, the FIRST request after a (re)start can go silent ~45–60 s
(health `engine_unresponsive`, 503s) and then self-recovers — do not
restart over it.
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` + kernel line via
`journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in
HIP); (2) run serve script, then raw binary — different outcome isolates
the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash
scripts/serve-<svc>.sh` — llama-server buffers stdout and loses the log
tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed`
spam is a symptom, not a root cause.
- Journal signatures: client abort logs `W srv stop: cancel task, id_task =
N`; natural stop logs only `slot release` with `truncated = 0`. The server
never preempts a running generation; excess tasks queue when slots are
busy.
- Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process,
`deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On halogen the FIRST request after a (re)start
can go silent for ~45–60 s (`engine_unresponsive`, 503s) and then self-recovers — do not restart
over it. [[frank-dragoon]]
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` AND the kernel line via `journalctl -k` (a
segfault at address 8 in `libamdhip64` is a NULL unref in HIP); (2) run the serve script, then the
raw binary — a different outcome isolates the env file; (3) `gdb -batch -ex run -ex "bt 25"
--args bash scripts/serve-<svc>.sh`, because llama-server buffers stdout and loses the log tail on
SIGSEGV: trust gdb, not the log; (4) `amdgpu: SVM mapping failed` spam is a symptom, not a root
cause. [[LLM Performance Diagnosis]]
- Journal signatures: a client abort logs `W srv stop: cancel task, id_task = N` while a natural stop
logs only `slot release` with `truncated = 0`. The server never preempts a running generation;
excess tasks queue when slots are busy.
**Host memory / swap**
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
of the old one, or live pages must land in RAM that `MemAvailable`
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom.
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
after a write the kernel rejected. Never prescribe a manual sysfs write as the
recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
proof — combined, those two instruments said "ruled out" while the RAM
was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
not a margin guarantee. When the fault is data-shaped (NaN states,
garbage tokens) and a software revert doesn't move it, price in the RAM
before the runtime — and remember a revert tested under the same marginal
hardware is itself confounded. See the runbook's 2026-10-05 section.
- **Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` of the old one**, or
live pages must land in RAM that `MemAvailable` overstates. Dankclaw's 64 GiB `/swap.img`
(fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool
is worse than the OOM it prevents. Host swap state is invisible to `deploy.sh` (no repo file):
`swapon --show` before reasoning about headroom. [[Swap Configuration]]
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults to half of RAM, so
a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the
start, and `Restart=always` re-ran that refusal 6,919 times in ~20 h — **a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no engine log is a
PREFLIGHT refusal, not an engine fault.** Repo law now: `scripts/ttm-preflight.sh` raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only after a write the
kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that
can boot before a human reaches it. [[frank-archon]]
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over
8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was
the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is
data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM
before the runtime; a revert tested under the same marginal hardware is itself confounded.
[[frank-archon]]
**Runtime-specific guards**
- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
dies at container create, not at runtime.** runc reports
`error setting rlimit type 7: operation not permitted` (type 7 =
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
are fine, so `memlock=-1:-1` works iff the unit grants
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
podman to runc without any raise). Upstream halogen's own run lines pass
NO `--ulimit` at all — copy vendor run lines before adding flags
(measured 2026-10-05 on the frank-dragoon cutover).
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
`third_party/composable_kernel/include/ck/config.h` generated manually
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
non-greedy requests and reasoning budgets. Re-test chat before production.
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
stock binaries fail the draft load with a tensor-shape mismatch.
- **A GGUF's custom tensor type picks the runtime — read the histogram before
promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
the only door. Counting tensor types out of the GGUF KV section takes one
python read and settles a runtime argument in minutes. The build identity
matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
`ggml_compute_forward_set_rows` on this same file, so
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
`--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
`--spec-type`, or it measures a crash instead of the absence of drafting.
- **An orphaned scratch server taxes the whole host.** A bench child left
alive by a client-side exception kept ~26 GiB of weights resident on a
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
Record the child pid and kill it in a `finally`, then prove `free` came back
before believing a number. Kill by pid — never `pkill -f llama-server`,
which also hits frank-zealot.
- **Rootless podman cannot RAISE rlimits, so `--ulimit nofile=1048576` dies at container CREATE**,
not at runtime: runc reports `error setting rlimit type 7: operation not permitted` (type 7 =
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on
Dankclaw). Inherited-equal limits are fine, so `memlock=-1:-1` works iff the unit grants
`LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
before adding flags. [[frank-dragoon]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a
file to a loader.** A non-standard tensor type in the KV section makes stock loaders refuse
the file at load, so a fork can be the only door rather than a preference. Counting tensor
types takes one python read and settles a runtime argument in minutes; build identity matters
as much as the fork, so pin the commit by SHA in any build script. [[GGUF Compatibility]]
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
(the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
measures a crash instead of the absence of drafting. [[frank-zealot]]
- **An orphaned scratch server taxes the whole host.** A bench child left alive by a client-side
exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
back before believing a number — and kill by pid, never `pkill -f llama-server`.
**llama.cpp serving**
- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
`--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
the MAIN model's weights (`shares_model`), so there is no second copy of a
12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
`--spec-type a,b` builds every enabled implementation and chains drafting
through them, in a fixed internal order (ngram types first, then the
drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
types (the ngram family) are added unconditionally, while `draft-mtp`,
`draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
CONTEXT. So a chain may hold one context-backed drafter plus any number of
ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
drafter alone on prompt-echo traffic, byte-identical to it on prose at the
fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
type MTP requested but model doesn't contain MTP layers"*, because MTP
builds its context against whatever model the draft context holds. Two
context-backed types cannot share one process, and MTP's head only loads
when MTP **is** the draft model. **A LEGAL chain can still lose on a
recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
logs `recurrent state rollback is not compatible with 'ngram-mod' -
disabling rollback support`, and MTP verification depends on that rollback,
so acceptance drops on ALL traffic even where the matcher fires. Measure
chains per architecture, never by analogy.
- **A drafter's KV scales with the TARGET's context, and its window policy
decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with
no flag to cap it, so a drafter declaring NO sliding window caches the whole
window on `--spec-draft-device` and can OOM the first draft graph capture
AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF
header before sizing, and prefer a skewed `-ts` over a context cut. Worked
example: [[frank-medium-llamacpp-dflash2]].
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
single-card endpoint that squeeze can cost more than the drafter wins.** With
`--parallel N` the drafter pays the pool, not the slot, plus its own compute
buffers. So the MTP question is never "does the file carry the head" but "what
do draft KV + buffers cost against what the batch knobs leave for compute" —
a per-shape answer. Measure the arm, never the flag; the booted arm table is
on [[frank-zealot]].
- **A quant's scale overhead belongs in its byte rate; doubling a measured
figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element
(32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is
**1.882×**, not 2×. Derive byte rates from element count × block layout, then
check them against the boot ledger — this repo carried an invented 2× number
for weeks. Current KV math: [[frank-zealot]].
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is
BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land
about TWICE as heavy on the card that holds the output layers and the drafter,
so a split balanced at one batch is unbalanced at another: re-measure the
balance after any batch change. The logical `--batch-size` is nearly free.
A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit
— no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]]
(R9700 split) and [[frank-zealot]] (CUDA 3090).
- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A
prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. When a bench
IS ordered, A/B it by restarting the SERVICE through a transient
`FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one
server. Numbers: [[frank-zealot]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB
each with its own KV), so balance by measurement. An affine fit reproduces the
deployed point and overstates the gaps in between.
[[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
`adding speculative implementation 'draft-dflash'` and nothing at all for the
matcher, so a chain can look unengaged while it is working. Identify it
behaviourally (`draft_n` > 0 only where text repeats) or by the per-task
`draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB
host-RAM hash table) and its window decides which traffic it can see: too
short and it proposes on ordinary prose fragments, which costs verification
without acceptance.
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with
a 400 even when the pool has room, so `models.yml` `contextWindow` must follow
every reslice. Both slot directions cost or buy window, never pool; shorter
slots decode slightly faster (less attention to scan, one fewer recurrent
state). Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
costs ~1.5× its file size** (the encoder's activation and compute buffers ride
with it). Subtracting two booted rows isolates it exactly; extrapolating
weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7.
Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. The two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
model still invents an answer if the client used the legacy top-level
`images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
failure is silent because a hallucinated color is a plausible answer. Probe
vision with a solid color and expect an exact word.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
then served frank-dragoon: every tool-using agent on that endpoint was broken
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
option and the same ~12x prefill collapse described in the ROCm note above.
f16 KV needs nothing; frank-zealot ran f16 for that reason until
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
endpoint could take q8_0 KV.
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
`-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
table's own `dev` column was printing the truth the whole time. Any
placement-dependent number needs an independent check: sample the other
- **MTP is a property of the FILE, not of the flags.** The `-mtp` export carries the head as
`blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`); `--spec-type draft-mtp` sets
`load_mtp` and the draft context runs on the MAIN model's weights (`shares_model`), so there
is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
tensors at all, and pointing `--model-draft` at that file loads it twice. [[frank-zealot]]
- **What can be chained: ONE context-backed drafter plus any number of draftless ngram types.**
`--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
`ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
`draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
legal and it wins on prompt-echo traffic, but `draft-mtp,draft-dflash`
dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
hybrid loses, because adding the ngram layer logs
`recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
[[frank-dragoon]]
- **A drafter's KV scales with the TARGET's context, and its window policy decides the price.**
The draft context is built at `llama_n_ctx(ctx_tgt)` with no flag to cap it, so a drafter
declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
[[Llama Cpp Serving Sizing]]
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
cost against what the batch knobs leave for compute. Measure the arm, never the flag.
[[frank-zealot]]
- **A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the
f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per
32-block), f16 is exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. Derive byte rates from element
count × block layout, then check them against the boot ledger — this repo carried an invented
2× number for weeks. Current KV math: [[frank-zealot]]
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never
carry it across hosts.** Its compute buffers land about TWICE as heavy on the card that holds
the output layers and the drafter, so a split balanced at one batch is unbalanced at another:
re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. The optimum is a property of
the backend, so re-find it after any device or payload change. When a bench IS ordered, A/B it
by restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
port: one card holds one server.
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
[[Llama Cpp Serving Sizing]]
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
too short and it proposes on ordinary prose fragments, costing verification without acceptance.
[[Llama Cpp Serving Sizing]]
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
`contextWindow` MUST follow every reslice. Both slot directions cost or buy window, never
pool; shorter slots decode slightly faster. Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file
size** (the encoder's activation and compute buffers ride with it). Subtracting two booted rows
isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free
when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. **The projector is NOT billed in llama.cpp's own fit check**: the engine
prints its worst case and fits WITHOUT it, so price a shape from settled `nvidia-smi` after a
real boot — the projector, and any drafter context, lands after `/health` would already have
answered (same class as the DSpark drafter OOM). Two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.**
`--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the
client used the legacy top-level `images` field on `/v1/chat/completions` — frank-zealot
answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part.
The failure is silent because a hallucinated color is a plausible answer; probe with a solid
color and expect an exact word. [[frank-zealot]]
- **`--jinja` is what `tools` needs.** A subagent died with `500 tools param requires --jinja
flag` on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that
endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults
`--jinja` on and the serve scripts pin it; halogen also accepts `tools` and returns
`tool_calls`. Check tool support before routing a role at an endpoint.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts
only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500,
not a clamp — so the harness's levels must be folded onto that set in `models.yml`
(`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request and
the failure names Jinja, not the model: every subagent on `frank-zealot` died with
`Unexpected reasoning effort high` until `high: xhigh` was added. Halogen accepts the wider
set, so the two entries legitimately differ.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe reads as
blindness.** A probe at `max_tokens: 64` returned `content: ""` with `finish_reason: "length"`
and half a thought in `reasoning_content` while the image HAD reached the model (322 prompt
tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision.
- **The FIRST request after a boot is a warmup artifact**, and the 3× spread on one shape makes
graph-capture cost read as a model fault. Benchmark the second request or later.
[[frank-zealot]]
- **A fork build hides its own dtype laws.** Vendor builds carry their own compile options, so
prove a build's KV behaviour from the KV buffer line in the boot ledger and its
`CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics`
natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from
`/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot
re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]]
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
drop-in sourcing the repo env) outlives the bench unless retired: a leftover
`frank-zealot.service.d/ptq1.conf` made the cutover deploy boot a **different model at a
different shape** while `/health` answered 200, `/v1/models` echoed the alias (a flag, not a
fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service:
`systemctl show <svc> -p DropInPaths` (must be empty), then
`tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot ledger — and retire drop-ins
to `~/.AGENTTRASH/<date>-<topic>/` + `daemon-reload`. [[frank-zealot]]
- **A bench verdict that never reaches the repo is indistinguishable from a rogue arm.** That
same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at
141k) that lived only in `/tmp` env files and journald, so the next agent read the surviving
drop-in as an unfinished experiment and reported a hijack. Write the measurement into the
endpoint page and the arm script's header in the session that takes it — `/tmp` and the journal
are not records.
- **`llama-bench` separates a LIST with `/` and SWEEP values with `,`** — for both `-dev` and
`-ts`. `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run,
and `-ts 0.85,0.15` is two sweep values each putting everything on device 0. Write
`-dev A/B -ts x/y`. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve
one model") from two bogus OOMs, and the bench table's own `dev` column was printing the truth
the whole time. Any placement-dependent number needs an independent check: sample the other
device's memory (`nvidia-smi`) while the run is loaded.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
raises on anything else — HTTP 500, not a clamp — so the harness's levels
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
A role whose level is unmapped fails on its FIRST request, and the failure
names Jinja, not the model: every subagent on `frank-zealot` died with
`Unexpected reasoning effort high` until `high: xhigh` was added on
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
differ.
- **The vision projector is NOT billed in llama.cpp's own fit check.** The
engine prints `[mtmd] estimated worst-case memory usage of mmproj is
873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected
21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from
settled `nvidia-smi` after a real boot, never from
`model + context + compute` — the projector (and any drafter context) lands
after `/health` would already have answered. Same failure class as the
DSpark drafter OOM.
- **The FIRST request after a boot is a warmup artifact.** Measured on
Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then
1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a
3× spread on the same shape. Benchmark the second request or later, or the
graph-capture cost reads as a model fault.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe
reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64`
returned `content: ""` with `finish_reason: "length"` and half a thought in
`reasoning_content` — the image HAD reached the model (322 prompt tokens vs
42 for the text alone). At 512 tokens the same image answers "Red". Budget
≥512 for any thinking model you probe for vision.
- **Quantized KV is not the only dtype law a fork build can hide.** Vendor
builds carry their own compile options: frank-zealot's PrismML prebuilt
served `--cache-type-v q8_0` under flash attention at the exact 34,816
B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
proven by the KV buffer line in the boot ledger, not by the build flags you
remember passing.
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.**
Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape
fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy
service — that is how `frank-zealot` sat red across four payload generations
(the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already
0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090
is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
alert that has been firing for weeks.
- **A leftover bench drop-in silently hijacks the next deploy, and no health
gate sees it.** The sanctioned bench-window method — a transient
`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env` drop-in, the arm env
`source`ing the repo env and then overriding it — outlives the bench unless
it is retired. On 2026-10-18 a leftover `frank-zealot.service.d/ptq1.conf` —
the arm Frank had ordered at 02:36 and measured from 03:14 — made the cutover
deploy boot a **different model at a different shape** (3 slots of 133,888,
not 3 × 262,144) while `/health` answered 200, `/v1/models` echoed the alias
(a flag, not a fact) and the Playwright gate passed 33/33. After deploying any
recently-benched LLM service: `systemctl show <svc> -p DropInPaths` (must be
empty), then `tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot
ledger — and retire drop-ins to `~/.AGENTTRASH/<date>-<topic>/` +
`daemon-reload`.
- **A bench verdict that never reaches the repo is indistinguishable from a
rogue arm.** Same window: PTQ1_0 served production for 24 minutes with real
numbers (~48 t/s decode, 801 t/s prefill at 141k), but the figures lived only
in `/tmp` env files and journald, so the next agent on the host read the
surviving drop-in as an unfinished experiment and reported a hijack. Write the
measurement into the runbook and the arm script's header in the session that
takes it — `/tmp` and the journal are not records.
**Ops**
- Never `pkill -f` a pattern that appears in your own command line — it kills
your ssh session. Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
without it.
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
`AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
world-writable — user-unit spawns carry no supplementary groups). It must NOT
pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
2026-09-16, when a stale device-less container was found restarted 9,146
times (`RuntimeError: No HIP GPUs are available`).
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
ports off the gluetun service.
- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in
`dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh`
(diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by
hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11),
adopted on the vendor's 13–16 % prefill figure and never measured here: the
local A/B on byte-identical traffic says identity mode costs **nothing**
(+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free —
`amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old
`err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree
2026-09-24, Linux 7.4) still skips the page walk outright for the privileged
iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`.
- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
only the CURRENT instance; history across recreations lives in
`journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
(per-user storage — `sudo podman logs` finds nothing).
- **A dead rootless podman pause process breaks ALL container management on the
host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
command fails with `invalid internal status … could not find any running
process`, every container start fails binding podman's bridge addresses
(`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
assign requested address`), and every `Restart=always` quadlet loops. Container
storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
token, not a pid) and the rootless netns holding the bridges — deleting the
stale token is not enough, and `podman system migrate` segfaults in
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
had no other word on it). Treat the pause process as host state that can die
for its own reasons, and reach for the reboot when it does.
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
netavark DNATs *published* ports only: from a bridge container
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
gateway `10.89.1.1` are both refused. Use `host.containers.internal`
(169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
was open from that bridge — it never was).
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
`.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
+ 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
sha256, concatenate in numeric order — that is how the deployed 111 GiB file
was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
Q4_K_S look like it does not exist.
- **A fetch script must `-f` and verify, or it reports success on an access
error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
as the "artifact": a gated repo yielded three 153-byte shards with identical
sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
`-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
return 403 even with the host's HF token until that account accepts the gate.
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup
campaign, four read-only scouts worked the same pages: one classified era pages as
stale *because* they document retired payloads (that is an era page's content), one
reported files that do not exist, one reported duplications that grep disproved
(`457.9 t/s` and the ping-RTT table are solo-held by `Ethernet Bandwidth Math`),
and one cited a line number past the end of the file it judged. Re-verify every
claimed duplicate and every claimed staleness against the file and today's fleet
state before editing; a scout yield buys no evidence.
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
`GroupAdd=keep-groups`, plus the udev rule that makes those nodes world-writable — user-unit
spawns carry no supplementary groups) and must NOT pass `--cpu`: the gfx1151 ROCm wheels exist
and a stale device-less container was found restarted 9,146 times. [[ComfyUI]]
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun
service. [[SearXNG]] [[Gluetun]]
- **Dankclaw's kernel command line is a repo file.** It boots **`iommu=pt`** since 2026-10-18,
tracked in `dankclaw/host/grub`, and moves ONLY through `scripts/set-dankclaw-bootline.sh`
(diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by hand-editing
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
says identity mode costs nothing, so the XDNA2 NPU returned for free.
[[dankclaw-iommu-translation-off]]
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
[[Quadlet Lifecycle]]
- **A dead rootless podman pause process breaks ALL container management on the host, and
`podman system migrate` panics in 5.7.0.** Every `podman` command fails with
`invalid internal status`, every start fails binding the bridges
(`failed to bind udp listener on 10.89.x.1:53`), every `Restart=always` quadlet loops — while
container storage stays intact. What is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
token) and the rootless netns; deleting the stale token is not enough. Repair: **reboot**
(`/run/user` clears and the quadlets return on their own — check `Linger=yes` and
`WantedBy=default.target` first). The 2026-09-16 occurrence had no OOM in `journalctl -k`:
treat the pause process as host state that can die for its own reasons. [[Quadlet Lifecycle]]
- **A container cannot reach a `Network=host` service at the host's LAN IP**: netavark DNATs
*published* ports only, so the host bind and the `10.89.1.1` gateway are both refused from a
bridge container while the published port answers. Use `host.containers.internal`
(169.254.1.2). This 502'd `hermes.franksdank.cloud` until its Caddyfile upstream changed.
[[Container Host Reachability]]
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as `.partNofM`
fragments** while the smaller rungs are single files, so filtering `endswith(".gguf")` makes a
whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256,
concatenate in numeric order. [[Model Swap and Rollback]]
- **A fetch script must `-f` and verify, or it reports success on an access error.**
`curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
declared size, and check the LFS sha256 — the live `scripts/download-frank-*.sh` scripts do
all three; ad-hoc fetches tend not to. Gated repos 403 even with the host's HF token until
that account accepts the gate. [[Model Swap and Rollback]]
- **A verdict is a lead — the scout's and the tool's alike.** In the 2026-10-19
documentation campaign six scouts worked the same pages and produced invented
bullet IDs, files that do not exist, "duplications" grep disproved, line
numbers past the end of the file they judged, and Eras tables with invented
payloads. The mechanical tool was worse: `scripts/port-census.py` built its
corpus from the directory containing the page under test, so each page was held
by *itself* and ten era pages reported "0 sole-record lines" — which reads as
"pure duplication, trash it". Nothing was cut on a verdict; every holder was
re-grepped and every tool re-measured before it was believed.
- **Never let a census decide what to keep.** That same campaign then concluded
the era pages were irreplaceable because each held 14–63 sole-record lines —
wrong in the opposite direction: those were throughput figures for payloads that
no longer run. Point of use is the only ownership test, the operator applies it,
and 53 pages plus the 2,380-line change diary were destroyed as a result. A tool
can say where a fact lives; it cannot say whether anything needs it.
## Runbook Index
@@ -966,12 +788,13 @@ per-attempt page and no per-swap ledger anywhere else.
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
| Deployment flow | `serverAppConfigs-docs/Architecture/Deployment Flow.md` |
Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.
There is no page for a retired endpoint and no change diary. A payload swap
appends one row to the live endpoint's **Eras** table; when the endpoint itself
dies, its page is destroyed. Rollback is a commit, not a document.
`serverAppConfigs-docs/index.md` is the locator for every page that survives;
retired scripts live in `old/` and `.AGENTTRASH/`.
## Why
scripts/axe-measurements.py
Executable
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""Destroy measurement ledgers in the surviving pages.
Point of use (AGENTS.md → Documentation Policy): a rate/throughput figure for
anything not running and not being compared is destroyed, not filed. This pass
deletes
1. any section whose heading is a bench/throughput/rate/ladder/verdict ledger,
2. any table row or standalone bullet that REPORTS a rate (`123.4 t/s`,
`tok/s`, `ms/token`, `TTFT`, acceptance %, `llama-bench`, a bench arm),
and it never touches a shape number (ctx, slots, MiB budgets, ports) or a bullet
under a *Laws* / *Wire contract* / *Rollback* heading, where the sentence is the
payload and the number is decoration. Dry run by default; `--apply` writes.
Usage: axe-measurements.py [--apply] [--verbose]
"""
import pathlib
import re
import sys
VAULT = pathlib.Path("serverAppConfigs-docs")
RATE = re.compile(
r"\d[\d,.]*\s*(t/s|tok/s|tokens?/s|tok s|tps|ms/token|µs/token|GB/s|GiB/s)\b|"
r"\b(llama-bench|bench (arm|ladder|window)|decode ladder|prefill ladder)\b|"
r"\bacceptance (rate|of)\b|\bdraft_n\b|\bttft\b|\bServer-Timing\b",
re.I,
)
# A heading is a ledger only when the noun LEDS it and nothing instructional
# follows: `## Throughput` yes, `## 6. The context ladder (find the max bootable
# window by measurement)` no — that one is the method, which is point-of-use.
LEDGER_HEAD = re.compile(
r"^#{2,4}\s+(?:\d+[.) ]\s*)?(bench(es|mark|marks)?|throughput|results?|timings?|"
r"measured (figures|numbers|results|rates|rates)|rates|verdict|acceptance|ladder)"
r"\b[^|]{0,40}$",
re.I,
)
INSTRUCTIONAL = re.compile(r"how|why|law|policy|method|procedure|find|by measurement|execution|flip|checklist", re.I)
PROTECTED = re.compile(
r"^#{2,4}\s+.*(law|wire|rollback|contract|shape|eras|files|health|see also|related|"
r"boot|preflight|environment)",
re.I,
)
def blank_rate_cells(text):
"""Kill throughput COLUMNS wherever they appear: a cell that reports a rate
becomes an em-dash, so the row keeps whatever else it records (the payload
name, the reason, the rollback) and loses only the dead number."""
out = []
for line in text.split("\n"):
st = line.strip()
if st.startswith("|") and st.endswith("|") and RATE.search(line):
cells = line.split("|")
for i, c in enumerate(cells):
if RATE.search(c):
cells[i] = " — "
line = "|".join(cells)
out.append(line)
return "\n".join(out)
def collapse_eras(text):
"""One Eras row per era is fleet law; the NUMBERS in the old rows are not.
Keep the current era verbatim, fold everything older into one row that says
the numbers were destroyed and where the rollback actually lives (git)."""
lines = text.split("\n")
out = []
in_eras = False
ncol = 0
folded = 0
for line in lines:
if line.startswith("#"):
in_eras = bool(re.match(r"^#{2,4}\s+.*eras\b", line, re.I))
if folded:
out.append("| " + " | ".join(
[f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured "
"number destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1)))
folded = 0
out.append(line)
continue
st = line.strip()
if in_eras and st.startswith("|"):
cells = [c.strip() for c in st.strip("|").split("|")]
if set("".join(cells)) <= set(":- "):
ncol = len(cells)
out.append(line)
continue
if ncol == 0:
ncol = len(cells)
if re.search(r"2026-1[01]|current|active|running", st, re.I):
out.append(line)
else:
folded += 1
continue
if folded and not st.startswith("|"):
out.append("| " + " | ".join(
[f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured number "
"destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1)))
folded = 0
ncol = 0
out.append(line)
if folded:
out.append("| " + " | ".join(
[f"≤ 2026-09 — {folded} earlier eras destroyed 2026-10-19"] + ["—"] * max(0, ncol - 1)))
return "\n".join(out)
def main(argv):
apply = "--apply" in argv
verbose = "--verbose" in argv
total_lines = 0
pages = 0
for page in sorted(VAULT.rglob("*.md")):
s = str(page)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
text = page.read_text(errors="replace")
before = len(text.split("\n"))
if s.startswith("serverAppConfigs-docs/LLM/"):
text = collapse_eras(blank_rate_cells(text))
lines = text.split("\n")
out = []
skip = False
under_protected = False
cut = 0
for line in lines:
if line.startswith("#"):
under_protected = bool(PROTECTED.search(line))
skip = (bool(LEDGER_HEAD.match(line)) and not under_protected
and not INSTRUCTIONAL.search(line))
if skip:
cut += 1
continue
if skip:
cut += 1
continue
stripped = line.strip()
out.append(line)
if under_protected:
# Eras rows are the rollback record, Law bullets are the payload:
# the number is decoration inside them, and the row/bullet is not
# a ledger entry. Leave the whole section alone.
continue
if stripped.startswith(("|", "-", "*")) and RATE.search(line):
# Destroy only when the line EXISTS to report a number: a table
# row, or a bullet whose verb is measurement.
if stripped.startswith("|") or re.search(
r"\b(measured|measures|bench|benched|ran at|runs at|clocked|hit|"
r"landed at|came in at|peak|peaks at|sustained|wall.clock)\b", line, re.I):
out.pop()
cut += 1
final = re.sub(r"\n{3,}", "\n\n", "\n".join(out))
saved = before - len(final.split("\n"))
if saved or cut:
pages += 1
total_lines += max(saved, cut)
print(f" {s.replace('serverAppConfigs-docs/', '')}: -{max(saved, cut)}")
if apply:
page.write_text(final.rstrip("\n") + "\n")
print(f"{'APPLIED' if apply else 'dry run'}: {pages} pages, {total_lines} lines of ledger")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/check-okf.py
Executable
+293
View File
@@ -0,0 +1,293 @@
#!/usr/bin/env python3
"""OKF v0.1 conformance linter for a docs bundle (default serverAppConfigs-docs).
Checks the hard rules of the Open Knowledge Format spec plus the maintenance
guidance, so "conforms to OKF" is a test and not a claim:
FRONTMATTER every non-reserved .md has parseable YAML frontmatter with a
non-empty `type` from the spec's value set
TYPE the value is one of reference / howto / tutorial / explanation
DESC `description` is present, non-blank, not the title, <= 120 chars,
and does not restate a fleet quantity
TIMESTAMP `timestamp` present, ISO 8601, not in the future
LENGTH lines between 1 and 500 inclusive
INDEX-FM no index.md may carry frontmatter (index.md is reserved)
LOG-DATE log.md date headings are `## YYYY-MM-DD` or `## YYYY-MM-DD — Title`,
one entry per date, newest first
INDEX-COVER every index.md links each of its sibling pages; the root index
links every page in the bundle
INDEX-NUM an index table row restates a fleet quantity — a digit run of
three or more, a digits-plus-unit figure, a port, or a
multiplier. Names (Qwen3.6, RDNA4, RTX 3090, PICO-8, P4) are
not quantities and are allowed
CITATIONS informational: pages with factual claims and no Citations
section are counted, never failed
LENGTH lines between 1 and 500 inclusive. A page may opt out of the
ceiling with `okf-exempt: length` in its frontmatter: AGENTS.md
makes one page per endpoint permanent, and an endpoint page with
a full Eras table cannot split without creating a second owner
of the same shape (tests/docs_completeness.bats enforces that)
Exit 0 when there are no violations.
"""
import datetime
import re
import sys
from pathlib import Path
TYPES = {"reference", "howto", "tutorial", "explanation"}
RESERVED = {"index.md", "log.md", "AGENTS.md"}
# A quantity, as opposed to a name: a comma-grouped figure, a run of 3+ digits,
# a unit-bearing figure, a port (":7770"), or a multiplier ("2x", "3 *").
QUANTITY = re.compile(
r"\d{1,3}(?:,\d{3})+" # 262,144
r"|\d{3,}" # 200000, 7770, 3090
r"|\d+(?:\.\d+)?\s*(?:GiB|MiB|GB|MB|TB|tok/s|t/s|x\b|%)"
r"|[:×]\s*\d"
r"|\d+\s*[x×]\s*\d"
)
# Product / model / standard names that legitimately contain digits.
NAMES = re.compile(
r"Qwen[\d.]*[-\w]*|Gemma\s?\d[\w.]*|MiMo[-\w.\d.]*|Ornith[-\w.\d]*"
r"|Bonsai[-\w.\d]*|Ling[-\w.\d.]*|DeepSeek[-\w.\d]*|GLM[-\w.\d]*"
r"|PICO-\d+|RDNA\s?\d+|R\d{4}|RTX\s?\d+|XDNA\d|gfx\d+|CUDA\s?[\d.]+"
r"|\d+\s?GbE|Wi-?Fi\s?\d|v?\d+\.\d+(?:\.\d+)?|b\d+|P\d\b|A\d+\b"
r"|frank-\w*2026-\d\d-\d\d|2026-\d\d-\d\d",
re.I,
)
DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}")
FM_KEY = re.compile(r"^([A-Za-z][\w-]*):\s*(.*)$")
def split_frontmatter(text):
if not text.startswith("---"):
return None, text
end = text.find("\n---", 3)
if end < 0:
return None, text
body_start = text.find("\n", end + 1)
return text[3:end], text[body_start + 1 :] if body_start >= 0 else ""
def fields(raw):
out = {}
if not raw:
return out
key = None
for line in raw.splitlines():
if not line.strip():
continue
m = FM_KEY.match(line)
if m:
key, val = m.group(1), m.group(2).strip().strip("\"'")
out[key] = val
elif line.startswith(" ") and key:
out[key] = (out[key] + " " + line.strip()).strip()
return out
def strip_names(value):
return NAMES.sub(" ", value)
def check_file(path: Path, root: Path, length_exempt=frozenset()):
v = []
rel = str(path.relative_to(root))
text = path.read_text(errors="replace")
lines = text.splitlines()
name = path.name
raw, body = split_frontmatter(text)
fm = fields(raw)
if name not in RESERVED:
if raw is None:
v.append(("FRONTMATTER", rel, "no parseable frontmatter"))
else:
t = fm.get("type", "")
if not t:
v.append(("TYPE", rel, "missing type"))
elif t not in TYPES:
v.append(("TYPE", rel, f"type '{t}' not in {sorted(TYPES)}"))
d = fm.get("description", "")
if not d:
v.append(("DESC", rel, "missing description"))
else:
if len(d) > 120:
v.append(("DESC", rel, f"description is {len(d)} chars (max 120)"))
if d.strip("# ").lower() == fm.get("title", "\0").strip("# ").lower():
v.append(("DESC", rel, "description equals title"))
if QUANTITY.search(strip_names(d)):
v.append(("DESC", rel, "description restates a quantity"))
ts = fm.get("timestamp", "")
if not ts:
v.append(("TIMESTAMP", rel, "missing timestamp"))
else:
try:
stamp = datetime.date.fromisoformat(ts[:10])
if stamp > datetime.date.today():
v.append(("TIMESTAMP", rel, f"timestamp {ts} is in the future"))
except ValueError:
v.append(("TIMESTAMP", rel, f"unparseable timestamp {ts!r}"))
if name == "index.md" and raw is not None:
v.append(("INDEX-FM", rel, "index.md must not carry frontmatter"))
if name == "log.md":
dates = []
for line in lines:
m = re.match(r"^##\s+(\S.*?)\s*$", line)
if not m:
continue
head = m.group(1)
if not DATE_RE.match(head):
continue
dates.append(head[:10])
# OKF requires the heading to USE the date format; a title after an
# em-dash is allowed and costs zero lines, so `## 2026-08-17 — Why the
# pair was rebuilt` is conformant and `## 2026-08-17Rebuilt` is not.
if head != head[:10] and not re.match(r"^\d{4}-\d{2}-\d{2}\s+—\s+\S", head):
v.append(("LOG-DATE", rel,
f"date heading must be the date alone or `date — Title`: '## {head}'"))
if len(dates) != len(set(dates)):
v.append(("LOG-DATE", rel, "duplicate date heading"))
if dates != sorted(dates, reverse=True):
v.append(("LOG-DATE", rel, "date headings not newest first"))
if raw is not None and name not in RESERVED:
n = len(lines)
body_lines = len([ln for ln in body.splitlines() if ln.strip()])
opted_out = "length" in fm.get("okf-exempt", "").lower()
if n > 500:
if rel in length_exempt:
v.append(("LENGTH-OK", rel,
f"{n} lines — over the hint, pinned by the "
"one-page-per-endpoint law"))
elif opted_out:
v.append(("LENGTH-OK", rel,
f"{n} lines — over the hint, opted out by the page"))
else:
v.append(("LENGTH", rel, f"{n} lines (max 500)"))
elif body_lines < 15:
v.append(("LENGTH", rel, "stub: fewer than 15 body lines"))
# A second `type:` key inside the body means the page carries two copies of
# itself with a frontmatter block in the middle — the shape a naive
# frontmatter rewrite produces. It doubles the length, splits the wikilinks
# and makes the page's facts ambiguous, so it is a hard failure.
if raw is not None and name not in RESERVED:
dupes = [i + len((raw or "").splitlines()) + 2
for i, ln in enumerate(body.splitlines())
if re.match(r"^(type|title|description):", ln)]
if dupes:
v.append(("DOUBLED", rel,
f"frontmatter keys inside the body at line(s) "
f"{', '.join(str(d) for d in dupes[:5])} — the page "
"contains a duplicated copy of itself"))
return v
def index_rows(text):
rows = []
for line in text.splitlines():
if line.startswith("|") and "[[" in line:
rows.append(line)
return rows
def check_links(root: Path):
v = []
pages = [
p for p in sorted(root.rglob("*.md"))
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
]
for index in [p for p in pages if p.name == "index.md"]:
text = index.read_text(errors="replace")
links = set(re.findall(r"\[\[([^\]|#]+)", text))
if index.parent == root:
targets = [p for p in pages if p.name not in RESERVED]
else:
targets = [p for p in index.parent.glob("*.md") if p.name != "index.md"]
for target in targets:
if target.stem not in links and target.name not in links:
v.append(("INDEX-COVER", str(index.relative_to(root)),
f"does not link sibling {target.name}"))
for index in [p for p in pages if p.name == "index.md"]:
for row in index_rows(index.read_text(errors="replace")):
if QUANTITY.search(strip_names(row)):
v.append(("INDEX-NUM", str(index.relative_to(root)),
"row restates a quantity: " + row[:80]))
return v
def live_endpoints(root: Path):
"""Pages pinned by AGENTS.md's one-page-per-endpoint law.
Derived, never hardcoded — the same derivation as
tests/docs_completeness.bats: an endpoint is live when scripts/ holds both
its env and its unit. Those pages carry the Eras table for their whole
history, so the OKF 500-line split hint cannot apply to them without
creating a second owner of the same shape.
"""
scripts = root.parent / "scripts"
out = set()
if scripts.is_dir():
for envf in scripts.glob("frank-*.env"):
if (scripts / f"{envf.stem}.service").exists():
out.add(f"LLM/{envf.stem}.md")
return out
def retired_era_pages(root: Path):
"""Era pages of a retired endpoint: campaign ledgers, one owner each.
AGENTS.md lets a campaign ledger that exists nowhere else live on the page
that measured it, and a retired endpoint keeps exactly one page per era.
Splitting such a ledger to satisfy the 500-line hint would fragment one
campaign across two owners, so the tag opts the page out.
"""
out = set()
for path in root.rglob("*.md"):
s = str(path)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
raw, _ = split_frontmatter(path.read_text(errors="replace"))
if raw is not None and "retired-endpoint" in fields(raw).get("tags", ""):
out.add(str(path.relative_to(root)))
return out
def main(argv):
root = Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
exempt = frozenset(live_endpoints(root)) | retired_era_pages(root)
violations = []
notes = []
citation_gaps = 0
pages = 0
for path in sorted(root.rglob("*.md")):
s = str(path)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
pages += 1
for code, rel, detail in check_file(path, root, exempt):
(notes if code == "LENGTH-OK" else violations).append((code, rel, detail))
text = path.read_text(errors="replace")
if len(text.splitlines()) > 40 and not re.search(
r"^##\s+(Citations|Sources)\b", text, re.M):
citation_gaps += 1
violations += check_links(root)
for code, rel, detail in violations:
print(f" {code:12s} {rel}: {detail}")
for code, rel, detail in notes:
print(f" {code:12s} {rel}: {detail}")
print(f"{pages} concept pages, {len(violations)} OKF violations"
f" ({citation_gaps} pages without a Citations section — soft;"
f" {len(notes)} length exemptions honoured)")
return 1 if violations else 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/check-vault-links.py
+6 -1
View File
@@ -31,7 +31,12 @@ def strip_md(path: str) -> str:
def build_index(vault: pathlib.Path):
pages = sorted(vault.rglob("*.md"))
# `.AGENTTRASH` is deleted-but-held state, not vault content: it must not be
# scanned (its relative links resolve against the trash dir, not the vault)
# and must not answer as a link target — a live page pointing into the trash
# is an unresolved link, not a resolved one.
pages = sorted(p for p in vault.rglob("*.md")
if ".AGENTTRASH" not in str(p))
stems: dict[str, list[pathlib.Path]] = {}
for page in pages:
stems.setdefault(page.stem.casefold(), []).append(page)
scripts/destroy-vault.py
Executable
+109
View File
@@ -0,0 +1,109 @@
#!/usr/bin/env python3
"""Destroy the vault's dead weight: era ledgers, retired runtimes, dead experiments.
`AGENTS.md` was treating a measurement as a record worth keeping forever. It is not: a
throughput figure for a payload that no longer runs, on a runtime that no longer exists,
is a liability — the next agent spends context on a number that AI progress has already
obsolete. This pass moves those pages to `.AGENTTRASH/` (repo law: never `rm`) and then
deletes every reference to them from the surviving pages, including the sentence that
carried the link when the link was the sentence's only content.
Usage: destroy-vault.py <list-file>
List file: one vault-relative path per line; a directory path destroys the whole tree.
"""
import pathlib
import re
import shutil
import sys
VAULT = pathlib.Path("serverAppConfigs-docs")
TRASH = VAULT / ".AGENTTRASH" / "20261019-destruction"
def targets(list_file):
out = []
for line in pathlib.Path(list_file).read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
p = VAULT / line
if p.is_dir():
out += [f for f in sorted(p.rglob("*.md"))]
elif p.is_file():
out.append(p)
else:
print(f" SKIP (missing): {line}")
return out
def main(argv):
if len(argv) < 2:
print(__doc__)
return 2
doomed = targets(argv[1])
# Stems come from the trash too, so re-running the pass (or running it after a
# partial destruction) still knows which names are dead and keeps scrubbing.
stems = {p.stem for p in doomed}
if TRASH.is_dir():
stems |= {p.stem for p in TRASH.rglob("*.md")}
moved = 0
for p in doomed:
dest = TRASH / p.relative_to(VAULT)
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(p), str(dest))
moved += 1
print(f"destroyed {moved} pages ({sum(len(p.read_text(errors='replace').splitlines()) for p in TRASH.rglob('*.md'))} lines staged in trash)")
# Repoint-or-delete: a link to a destroyed page is itself dead information.
link_wl = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|([^\]]*))?\]\]")
link_md = re.compile(r"\[[^\]]*\]\(([^)#]*\.md)(?:#[^)]*)?\)")
pages_cleaned = links_dropped = lines_dropped = 0
for page in sorted(VAULT.rglob("*.md")):
s = str(page)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
text = page.read_text(errors="replace")
lines = text.split("\n")
out = []
hits = 0
def dead(target):
# NOT pathlib.Path(...).stem: that strips a real suffix off names like
# `Lemonade and sd.cpp`, so a link to a destroyed page survives and the
# vault then fails its own link check.
t = target.strip()
if t.endswith(".md"):
t = t[:-3]
return t in stems or pathlib.PurePosixPath(t).name in stems
for line in lines:
found = [t for t in link_wl.findall(line) if dead(t[0])]
for m in link_md.finditer(line):
if dead(m.group(1)):
found.append(m.group(1))
if not found:
out.append(line)
continue
hits += len(found)
stripped = link_wl.sub(
lambda m: m.group(2) if (m.group(2) and not dead(m.group(1))) else "",
line)
stripped = link_md.sub(
lambda m: "" if dead(m.group(1)) else m.group(0), stripped)
# What is left of the line after the dead links are gone?
residue = re.sub(r"[\s\-*•·|:;,.\u2014\u2013]+", "", stripped)
if residue:
out.append(re.sub(r"\s+$", "", re.sub(r"^(\s*[-*]\s*)[\s:;,.\u2014-]+", r"\1", stripped)))
else:
lines_dropped += 1
if hits:
page.write_text("\n".join(out))
pages_cleaned += 1
links_dropped += hits
print(f"cleaned {pages_cleaned} surviving pages: {links_dropped} dead links dropped, "
f"{lines_dropped} lines left empty and deleted")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/gen-vault-index.py
Executable
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""Regenerate serverAppConfigs-docs/index.md from page frontmatter.
The root index is a locator (AGENTS.md -> *Documentation Policy*): one row per
page, one clause, no numbers. Hand-maintaining it is how rows go missing, and
the docs seam fails when a page is not linked from it, so it is generated:
* preamble everything above the first `## ` in the existing file is kept
verbatim (that is the curated statement of the vault's law)
* section from the page's directory, overridden by its frontmatter tags
(`endpoint` / `retired-endpoint` split the LLM pages)
* row text the page's frontmatter `description`, verbatim
A page with no description is a hard error rather than an empty cell, so the
generator cannot silently produce a silent index. Sections with no pages are
skipped. Deterministic: the same vault always yields the same bytes.
Sub-directory indices are hand-grouped (they carry curated prose), so they are
synced rather than regenerated: dead rows dropped, row text refreshed from the
page's description, uncovered siblings appended.
"""
import pathlib
import re
import sys
RESERVED = {"index.md", "log.md"}
SECTIONS = [
("Inference endpoints", "endpoint"),
("Retired endpoints", "retired-endpoint"),
]
DIRECTORY_SECTIONS = [
("Dankclaw", "Dankclaw services"),
("Dankserver", "Dankserver services"),
("FrankBook-Pro", "FrankBook-Pro"),
("Desktop", "Workstations"),
("Infrastructure", "Cross-host infrastructure"),
("Architecture", "Repo architecture and policy"),
("Operations", "Operations and incidents"),
("Agent", "Agent / harness"),
("Research", "Research and methodology"),
]
PIPELINE_SECTION = "Pipeline Parallelism (series)"
LLM_FALLBACK = "LLM technique and hardware references"
CHANGE_SECTION = "Change history"
ORDER = [t for t, _ in SECTIONS] + [t for _, t in DIRECTORY_SECTIONS] + \
[PIPELINE_SECTION, LLM_FALLBACK, CHANGE_SECTION]
def frontmatter_fields(text):
if not text.startswith("---"):
return {}
end = text.find("\n---", 3)
if end == -1:
return {}
raw = text[3:end]
fields = {}
for key in ("description", "type"):
m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
if m:
fields[key] = m.group(1).strip().strip("\"'")
tags = []
m = re.search(r"^tags:\s*\[(.*?)\]$", raw, re.M)
if m:
tags = [t.strip().strip("\"'") for t in m.group(1).split(",") if t.strip()]
elif re.search(r"^tags:\s*$", raw, re.M):
block = re.search(r"^tags:\s*\n((?:\s+-\s+.*\n?)+)", raw, re.M)
if block:
tags = [l.strip().lstrip("-").strip().strip("\"'")
for l in block.group(1).splitlines() if l.strip()]
fields["tags"] = tags
return fields
def section_for(page, vault, fields):
rel = page.relative_to(vault)
if len(rel.parts) == 1:
return CHANGE_SECTION if page.name == "log.md" else "Uncategorised"
top = rel.parts[0]
if top == "LLM":
if PIPELINE_SECTION_DIR in rel.parts:
return PIPELINE_SECTION
for title, tag in SECTIONS:
if tag in fields.get("tags", []):
return title
return LLM_FALLBACK
for prefix, title in DIRECTORY_SECTIONS:
if top == prefix:
return title
return "Uncategorised"
ROW = re.compile(r"^(\| )\[\[([^\]|]+)\]\]( \|)( .*?)( \|$)")
BULLET = re.compile(r"^(- )\[\[([^\]]+)\]\]( — )(.*?)$")
def sync_subindex(index: pathlib.Path, vault: pathlib.Path):
"""Keep a hand-grouped sub-index honest without erasing its grouping.
Rows are locators, so their text is the linked page's own description; a
row whose page is gone is a dead promise and is dropped; a direct sibling
with no row is appended under `## Pages` (that is what the link-integrity
seam checks). Curated headings and prose are untouched.
"""
here = index.parent
known = {p.stem for p in here.rglob("*.md")
if ".AGENTTRASH" not in str(p) and p.name != "index.md"}
lines = index.read_text().split("\n")
out, dropped, refreshed = [], 0, 0
for line in lines:
# A row whose first cell is empty is a description with no page: the link
# was deleted when the page was destroyed, so the row is a dead promise.
if re.match(r"^\|\s*\|", line.strip()):
dropped += 1
continue
for pat, wrap in ((ROW, "| [[{}]] |{} {} |"), (BULLET, "- [[{}]] — {}")):
m = pat.match(line)
if not m:
continue
stem = m.group(2).strip()
if stem not in known:
dropped += 1
line = None
break
cand = next((p for p in here.rglob(f"{stem}.md")
if ".AGENTTRASH" not in str(p)), None)
desc = frontmatter_fields(cand.read_text(errors="replace")).get("description") \
if cand else None
if desc:
current = m.group(4).strip()
if current != desc:
line = wrap.format(stem, " " if pat is ROW else "", desc) \
if pat is ROW else wrap.format(stem, desc)
refreshed += 1
break
if line is not None:
out.append(line)
text = "\n".join(out)
# Presence must be the wikilink itself: a bare stem is a substring of other
# page names (`frank-medium` lives inside `frank-medium-rocmfpx`), and a
# sibling matched that way silently never gets a row.
missing = sorted(
(p.stem, frontmatter_fields(p.read_text(errors="replace")).get("description", ""))
for p in here.glob("*.md")
if p.name != "index.md" and f"[[{p.stem}]]" not in text
)
if missing:
block = ["", "## Pages", "", "| Page | Owns |", "|---|---|"]
block += [f"| [[{stem}]] | {desc} |" for stem, desc in missing]
text = text.rstrip("\n") + "\n" + "\n".join(block) + "\n"
if text != index.read_text():
index.write_text(text)
return dropped, refreshed, len(missing)
PIPELINE_SECTION_DIR = "Pipeline Parallelism"
def main(argv):
vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
index = vault / "index.md"
if not index.is_file():
print(f"missing {index}")
return 2
existing = index.read_text()
preamble = existing.split("\n## ", 1)[0].rstrip("\n")
pages = [p for p in vault.rglob("*.md")
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
and p.name != "index.md" and p != index]
buckets, errors = {}, []
for page in pages:
fields = frontmatter_fields(page.read_text(errors="replace"))
if page.name != "log.md":
desc = fields.get("description")
if not desc:
errors.append(str(page.relative_to(vault)))
continue
else:
desc = "Chronological incident and law log for the whole bundle."
buckets.setdefault(section_for(page, vault, fields), []).append((page.stem, desc))
if errors:
print("pages with no description (fix the frontmatter, not the index):")
for e in sorted(errors):
print(" " + e)
return 1
out = [preamble, ""]
for title in ORDER + sorted(set(buckets) - set(ORDER)):
rows = sorted(buckets.get(title, []), key=lambda r: r[0].lower())
if not rows:
continue
out += [f"## {title}", "", "| Page | Owns |", "|---|---|"]
out += [f"| [[{stem}]] | {desc} |" for stem, desc in rows]
out.append("")
out.append("<!-- generated by scripts/gen-vault-index.py; edit page frontmatter instead -->")
index.write_text("\n".join(out) + "\n")
touched = 0
for sub in sorted(vault.rglob("index.md")):
if sub == index or ".AGENTTRASH" in str(sub):
continue
dropped, refreshed, added = sync_subindex(sub, vault)
if dropped or refreshed or added:
touched += 1
print(f" {sub}: -{dropped} dead rows, {refreshed} refreshed, +{added} added")
print(f"{index}: {sum(len(v) for v in buckets.values())} rows in "
f"{len([t for t in ORDER if buckets.get(t)])} sections, {touched} sub-indices synced")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/port-census.py
Executable
+100
View File
@@ -0,0 +1,100 @@
#!/usr/bin/env python3
"""Port census: prove a merge/collapse lost no fact.
For every line that existed in the OLD set and not in the NEW set, check that
each of its significant tokens (numbers, long identifiers, paths) still occurs
somewhere in the NEW corpus. A line whose every token survives is a reworded
line; a line with an unheld token is a candidate fact loss and gets printed.
port-census.py <new-glob-dir> <old-file> [<old-file> ...]
Old files may be `git-rev:path` (read from git) or a path (e.g. a trash copy).
"""
import pathlib
import re
import subprocess
import sys
def toks(line):
out = []
for n in re.findall(r"\d[\d,]*(?:\.\d+)?", line):
if len(n.replace(",", "").split(".")[0]) >= 3 or (
"." in n and len(n.split(".")[-1]) >= 2 and len(n.replace(",", "")) >= 4
):
out.append(n)
# Identifiers, flags, paths and backticked spans are the facts worth
# tracking; ordinary prose words are not — they reword on every rewrite.
# A hyphen alone is not evidence of an identifier (`long-standing`), so a
# dashed word counts only with two hyphens (`frank-tts-engine`) or with a
# digit, dot, slash, colon or `--` inside it.
for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line):
core = w.strip(".,;:!?\"')]")
if (core.startswith("--") or core.isupper()
or re.search(r"[\d_/:.]", core) or core.count("-") >= 2):
out.append(core)
out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3]
return out
def held(tok, corpus):
t = tok.replace(",", "")
alt = rf"\b{re.escape(t)}\b"
for c in corpus:
if tok in c or t in c:
return True
if re.search(alt, c.replace(",", "")):
return True
return False
def main(argv):
new_dir = pathlib.Path(argv[1])
def corpus_excluding(exclude):
"""Every vault page except the one under test.
The corpus is what must still hold a fact AFTER the file is gone, so a
file that lives inside the scanned directory has to be removed from it —
otherwise every token is "held" by the file itself and the census proves
nothing (it reports 0 losses for every page, which is the bug this
guards).
"""
out = []
for p in new_dir.rglob("*.md"):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
if exclude is not None and p.resolve() == exclude:
continue
out.append(p.read_text(errors="replace"))
return out
lost = checked = 0
for spec in argv[2:]:
if spec.startswith("git-"):
ref, path = spec[4:].split(":", 1)
old = subprocess.run(["git", "show", f"{ref}:{path}"],
capture_output=True, text=True).stdout
label = f"{ref}:{path}"
exclude = None
else:
old = pathlib.Path(spec).read_text(errors="replace")
label = spec
exclude = pathlib.Path(spec).resolve()
blob = "\n".join(corpus_excluding(exclude))
for line in old.splitlines():
s = line.strip().strip("->*|# ").strip()
if len(s) < 30 or s.startswith("```"):
continue
checked += 1
missing = [t for t in toks(s) if not held(t, [blob])]
if missing:
lost += 1
print(f" {label}\n L: {s[:150]}\n unheld: {', '.join(missing[:6])}")
print(f"{checked} old lines checked, {lost} with unheld tokens")
return 1 if lost else 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/repair-damage.py
Executable
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env python3
"""Repair the two artifacts the destruction passes introduced.
1. The link stripper could eat the space after a `- ` marker (`-`X``) and leave a
dangling `see .` where a wikilink used to be.
2. The Eras collapse could swallow a table's HEADER row (it looked like just
another non-current row), leaving a separator with no header above it.
Usage: repair-damage.py [--apply]
"""
import pathlib
import re
import sys
VAULT = pathlib.Path("serverAppConfigs-docs")
ERAS_HEADER = "| Window | Payload | Runtime | Shape | Replaced because | Rollback |"
def main(argv):
apply = "--apply" in argv
marker = dangling = header = 0
for p in sorted(VAULT.rglob("*.md")):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
lines = p.read_text(errors="replace").split("\n")
out = []
# Only a pointer whose target was deleted reads as `… see .` — never touch
# a line merely because it wraps on a comma. (An earlier version of this
# rule stripped the trailing comma off 660 wrapped prose lines, which
# `scripts/restore-punctuation.py` then had to undo from HEAD.)
for line in lines:
new = line
if re.match(r"^(\s*)[-*](?![-* ])[$`[(*_]", new):
new = re.sub(r"^(\s*)[-*]", r"\1- ", new)
marker += 1
if re.search(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", new):
new = re.sub(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", "", new)
dangling += 1
out.append(new)
# A separator row whose line above is not a header gets its header back.
fixed = []
for i, line in enumerate(out):
if re.match(r"^\|-+", line.strip()) and (not fixed or not fixed[-1].strip().startswith("|")):
fixed.append(ERAS_HEADER)
header += 1
fixed.append(line)
if apply and "".join(fixed) != "".join(lines):
p.write_text("\n".join(fixed))
print(f"{'APPLIED' if apply else 'dry run'}: marker spacing {marker}, "
f"dangling pointers {dangling}, restored table headers {header}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/restore-punctuation.py
Executable
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Undo trailing-punctuation damage by restoring lines that match HEAD exactly
except for a stripped trailing `,` / `;` / removed trailing word.
The repair pass over-reached: its "dangling pointer" cleanup also matched every
wrapped prose line ending in a comma, and deleted the comma. A line whose text
is otherwise byte-identical to its HEAD version, differing only by trailing
punctuation, is not an edit — it is damage, so HEAD's line goes back.
Usage: restore-punctuation.py [--apply]
"""
import pathlib
import subprocess
import sys
REPO = pathlib.Path(".")
VAULT = REPO / "serverAppConfigs-docs"
def head_text(rel):
r = subprocess.run(["git", "show", f"HEAD:{rel}"], capture_output=True, text=True)
return r.stdout if r.returncode == 0 else None
def main(argv):
apply = "--apply" in argv
restored = 0
pages = 0
for p in sorted(VAULT.rglob("*.md")):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
rel = s
head = head_text(rel)
if head is None:
continue
cur = p.read_text(errors="replace").split("\n")
# Key on (length, prefix) of the HEAD line minus trailing punctuation so a
# damaged line can find its original.
index = {}
for hl in head.split("\n"):
h = hl.rstrip()
if h.endswith((",", ";")) or True:
key = (len(h.rstrip(",;. ")), h.rstrip(",;. ")[:60])
index.setdefault(key, h)
touched = 0
for i, cl in enumerate(cur):
c = cl.rstrip()
if not c or c.endswith((".", ":", "|", "`", ")", "," , ";")):
continue
key = (len(c), c[:60])
hl = index.get(key)
if hl and hl != c and hl.startswith(c):
cur[i] = hl
touched += 1
if touched:
pages += 1
restored += touched
if apply:
p.write_text("\n".join(cur))
print(f"{'APPLIED' if apply else 'dry run'}: {restored} lines restored across {pages} pages")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
serverAppConfigs-docs/Agent/Agent Docs and Edits.md
+4 -4
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Agent Docs and Edits
description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
description: "Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each."
tags: [agents, docs, markdown, hashline, edit-recovery]
timestamp: 2026-10-18T00:00:00Z
---
@@ -21,7 +21,7 @@ Damage catalogue:
no longer frontmatter (title/tags/description become body text).
2. A rewrite **appended** instead of merged → duplicate headings and a stray
second `---` mid-file.
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`
`+- `, `- - `.
4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
5. Line-range replacement that dropped a sentence tail → dangling fragment,
@@ -46,7 +46,7 @@ Repair order (matters):
2. If it merely duplicated: **salvage unique content first** — cut the
one-of-a-kind section, paste it into the copy you keep, THEN delete the
duplicate ranges. Never delete first.
3. Re-add missing sentence tails (insert before the orphaned fragment),
3. Re-add missing sentence tails (insert before the orphaned fragment)
keeping list continuation lines indented.
4. Re-run all checks above.
@@ -57,7 +57,7 @@ safe, but its snapshot tag changes on every edit — re-read before each
subsequent edit. `.AGENTTRASH/` is gitignored (`.gitignore:25`), so NEW trash
stays local — but do not read that as "the repo has none of it": on 2026-10-18
`git ls-files | grep -c AGENTTRASH` returns **37** tracked paths
(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`,
(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`
`tests/.AGENTTRASH/`), because gitignore never untracks what was committed
before the rule. Trash is a temporary hold, not an archive.
serverAppConfigs-docs/Agent/Command Center Dotfiles.md
+1 -1
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Command Center Dotfiles
description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
description: "How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas."
tags: [dotfiles, omp, deploy, rsync, blankpad]
timestamp: 2026-10-05T00:00:00-07:00
---
serverAppConfigs-docs/Agent/OMP Harness.md
+12 -12
View File
@@ -1,7 +1,7 @@
---
type: reference
title: OMP Harness
description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
description: "The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits."
tags: [omp, harness, compaction, models-yml, context, samplers]
timestamp: 2026-10-18T00:00:00Z
---
@@ -16,7 +16,7 @@ this file is repo-managed; edits are harness-side only.
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and
`frank-zealot` (Dankclaw) — engine, port and card behind each id live on
[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`,
[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
files, runbook pages); pages written in the old era name the service,
the harness names the model. The wire now matches: each endpoint's
@@ -26,7 +26,7 @@ picker/grants, the homepage docs and the raw API all use one id per model
note on the endpoint echoing the requested name).
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
192.168.0.143:7771 each answer with the unit name (`frank-archon`,
192.168.0.143:7771 each answer with the unit name (`frank-archon`
`frank-dragoon`, `frank-zealot`).
## Compaction chain arithmetic (verify before ANY window change)
@@ -40,11 +40,11 @@ breaks compaction.
Where the contract lives:
- `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL:
`http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`),
`methodOrder` (`shake → snapcompact → soft → handoff → remote`,
`http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`)
`methodOrder` (`shake → snapcompact → soft → handoff → remote`
`config.yml:77-82`), `thresholdPercent` (**95**, `config.yml:84`), idle
settings (`idleEnabled`, `idleThresholdTokens: 100000`)
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`
`compactionModel` references
- Implementation: omp plugin
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
@@ -104,16 +104,16 @@ input. Boot-shape OOM laddering is a different problem
## Per-model sampler config (`compat.extraBody`)
`models.yml` has **no per-model sampler keys**. The global knobs are
`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`
`repetitionPenalty`, and a knob reaches the wire only when it is set — the
bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`),
bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`)
and this workstation's `config.yml` sets **none** of them. The 2026-10-05 note
that the unset default is `-1`/"not sent" is retired: there is no `-1`
sentinel, an unset key is simply absent. The per-model hook is
`compat.extraBody`:
**Why it works:** `applyOpenAIExtraBody` runs **last** in the
openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
openai-completions builder — after `applyChatCompletionsCompatPolicy` —
`Object.assign(params, extraBody)`. So extraBody overrides global settings
and compat-policy output.
@@ -191,7 +191,7 @@ assistant turn carrying a long `reasoning_content`, then compare
`usage.prompt_tokens` with the flag true vs false. A large delta means the
field changed template rendering; zero delta means it was ignored.
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`)
never `rm`.
## Static-preamble audit ("the context feels bloated")
@@ -205,7 +205,7 @@ What loads statically every session (measure each, don't guess):
```bash
python3 - <<'EOF'
import glob, re, os
for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
for root in [os.path.expanduser("~/.omp/agent/managed-skills")
os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
n=tot=0
for sk in glob.glob(root+"/*/SKILL.md"):
@@ -305,4 +305,4 @@ standalone; fold go/no-go arithmetic skills into their probe skill
- [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
- [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
- [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
- measuring the endpoints this config routes
serverAppConfigs-docs/Agent/index.md
+4 -15
View File
@@ -1,23 +1,12 @@
---
type: index
description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
tags: [agent, index]
timestamp: 2026-10-05T00:00:00-07:00
---
# Agent
Pages about the agent's own tooling and context — how omp is configured, how
docs get edited safely, and how the agent's home is deployed.
- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
`compat.extraBody`, preamble and skill-library audits
- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
hashline edit recovery
- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
transport, override packages
- [[OMP Harness]] — The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits.
- [[Agent Docs and Edits]] — Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each.
- [[Command Center Dotfiles]] — How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas.
## Related
- [[Testing Policy]] — the seams these pages' checks ride
- [LLM](../LLM/index.md) — the fleet these agents operate
- the fleet these agents operate
serverAppConfigs-docs/Architecture/Container Network Namespaces.md
+3 -4
View File
@@ -1,8 +1,7 @@
---
type: reference
title: Container Network Namespaces
description: Which local services are addressable from which Podman network namespace on
Dankclaw, and why bridge-network containers cannot dial host-netns loopback listeners.
description: "How Podman builds container networks here: the network inventory, the netns sharing mechanism, and the isolation limits."
tags:
- networking
- podman
@@ -48,7 +47,7 @@ network namespace**:
`Network=` values are read from the live Quadlet sources in
`/home/dank/.config/containers/systemd/*.container`; runtime network names on the
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`,
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`
`systemd-searxng`, `systemd-open-webui`.
Two consequences that are easy to miss:
@@ -84,7 +83,7 @@ podman exec <name> cat /proc/net/route # default route → gateway IP
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?'
```
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`,
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`
`10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns,
published) and `:9119` (its own published port) connect; `:8080` and `:8787`
are **refused instantly** even though both listeners are bound to `0.0.0.0` in
serverAppConfigs-docs/Architecture/Deploy State Triage.md
+6 -6
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Deploy State Triage
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
description: "Reading deploy-state markers against live containers and deciding, safely, whether to redeploy."
tags: [deployment, deploy-state, systemd, podman, triage]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -36,7 +36,7 @@ possibly zero behavioral change.
2. Probe the target (ssh as `dank`, passwordless sudo present):
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
- `git -C ~/serverAppConfigs rev-parse HEAD`
- `systemctl is-active/is-enabled <svc>`;
- `systemctl is-active/is-enabled <svc>`
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
- the on-host env file already carries the new keys (it landed via the
ff-pull).
@@ -119,8 +119,8 @@ each, kill stale ones by explicit PID.
A killed deploy leaves behind:
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
explicit stop — start it yourself);
- the state marker **pending** (pre-deploy or in-flight revision);
explicit stop — start it yourself)
- the state marker **pending** (pre-deploy or in-flight revision)
- the e2e gate un-run (the deployer runs it last).
Reconcile the marker ONLY after the service itself is verified healthy — a
@@ -142,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts
numbers — which looks exactly like your change having no effect. Key on
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
See for the full capture
procedure.
## Related
- [[Deployment Flow]] — the deploy pipeline itself
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
- journal capture
serverAppConfigs-docs/Architecture/Deployment Flow.md
+2 -3
View File
@@ -1,5 +1,5 @@
---
type: guide
type: howto
title: Deployment Flow
description: Service-scoped Git deployment for the two homelab hosts.
tags:
@@ -68,7 +68,7 @@ service on the other host.
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
The shared chat template maps to the inference service on the selected target.
The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
The `frank-npu` endpoint is retired () and has no deploy handler
an unrelated large-model deployment never touches it.
## Caddy safety
@@ -125,6 +125,5 @@ routing. See [[Testing Policy]].
## See Also
- [[Testing Policy]]
- [LLM deployment](../LLM/deployment.md)
- [[Caddy]]
- [[SearXNG]]
serverAppConfigs-docs/Architecture/E2E Suite.md
+2 -2
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "`tests/e2e/` drives the **live fleet** through a real browser: fresh contexts"
description: "The Playwright suite that drives the live fleet through a real browser, its project set, and the FrankBook conditional."
tags:
- architecture
- e2e
@@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior.
*arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
sleeping Mac mini must never fail an unrelated deploy. The project joins only
when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`
`shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
that touches the arr links must therefore be paired with a `19-frankbook` run
serverAppConfigs-docs/Architecture/Network Architecture.md
+6 -6
View File
@@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
else), and accepted traffic returns to Caddy's internal listener on `:8016`,
else), and accepted traffic returns to Caddy's internal listener on `:8016`
which proxies every request whole to the Authentik server with the Host
header preserved; Authentik's embedded outpost routes on that Host against
each proxy provider's `external_host`, serves the login flow in place for
@@ -67,12 +67,12 @@ graph TB
```
Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`):
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`),
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`)
prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta
(`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.`
(the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP
clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP),
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP)
books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts
that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own
auth"). Grafana is **not** exempt even though it signs in through authentik as
@@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through
`Network=searxng.network` (runtime network `systemd-searxng`), while
`searxng.container`, `marreta.container` and `marreta-browser.container` join
the tunnel the modern way — `Network=gluetun.container`
(`dankclaw/searxng/quadlet/searxng.container:18`,
`dankclaw/marreta/quadlet/marreta.container:34`,
(`dankclaw/searxng/quadlet/searxng.container:18`
`dankclaw/marreta/quadlet/marreta.container:34`
`dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet
equivalent of the retired compose `network_mode: service:gluetun`. **Five**
containers share the VPN exit, not three, and Marreta's outbound fetches leave
through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`);
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`)
everything else on Dankclaw uses the normal LAN route. This matches KB Pattern
2 in [[Gluetun]].
See [[Container Network Namespaces]] for which addresses are reachable from that
serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md
+6 -6
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Quadlet Lifecycle
description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
description: "The Quadlet discipline: ownership audits, compose cutover, drift and crash-loop triage, generator quirks, log retrieval."
tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -36,14 +36,14 @@ done
## 2. Cutover: compose service → Quadlet unit
Checklist (precedents to copy:
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
`dankclaw/observability/quadlet/`):
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
line for line: `Image=` (`localhost/` prefix for locally built images +
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
containers by name), `Network=`, `User=0:0` when the image's own user
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved)
`Exec=` for compose `command:`, `PublishPort=`, health keys,
`[Service] Restart=always` + `RestartSec=10`, `[Install]
WantedBy=default.target`. **Never `systemctl --user enable` the generated
@@ -56,7 +56,7 @@ Checklist (precedents to copy:
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
`~/.config/systemd/user/<svc>.service` that shells out to
`podman compose up` collides BY NAME with the generated unit);
`podman compose up` collides BY NAME with the generated unit)
`podman rm -f <container>` so the unit's container can take the name.
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
@@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever.
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
= the §2 cutover. **Verify the capability, not the status:** `podman exec
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
<name> python3 -c "import torch; print(torch.cuda.is_available())"`
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
@@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
unhealthy but logs look fine, compare the baked
`{{.Config.Healthcheck.Test}}` array against the intended command before
believing the status.
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
- **Probe a new key before shipping:** drop a throwaway `probe.container`
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
"unsupported key"`, `systemctl --user cat probe.service` (shows the
generated ExecStart verbatim), then remove + reload.
serverAppConfigs-docs/Architecture/Stale Signal Audit.md
+1 -1
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Stale Signal Audit
description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
description: "The audit battery for surfaces that rot without failing: dead probes, vacuous assertions, gates that never run."
tags: [testing, observability, audit, signals, gates]
timestamp: 2026-10-05T00:00:00-07:00
---
serverAppConfigs-docs/Architecture/Testing Policy.md
-1
View File
@@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con
## See Also
- [[Deployment Flow]]
- [[index]]
serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md
+7 -7
View File
@@ -1,6 +1,6 @@
---
type: howto
description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next."
description: "Proven Qwen-Image text-to-image workflows on Dankclaw: the recipes, the API call shape, and which models to add next."
tags:
- dankclaw
- comfyui
@@ -56,8 +56,8 @@ sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed
fixed in the file (`1930` in the art workflow, `2026` in the text one) —
read from `dankclaw/comfyui/workflows/*.json` node `9`.
Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`,
`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`,
Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`
`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`
`SaveImage` (ids 3–11).
| Workflow | File | Shape | Steps | Made for |
@@ -113,9 +113,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`,
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`,
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`,
all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
`huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk
yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18).
@@ -126,7 +126,7 @@ yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18)
| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) |
| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) |
Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`
#3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
sha256 checks) or a plain `hf download` with the token in the environment —
serverAppConfigs-docs/Dankclaw/ComfyUI.md
+3 -3
View File
@@ -43,7 +43,7 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not
## GPU device access (rootless), and why the unit exists
The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`,
The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`
`dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses
`GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman
cannot map the group name into the user namespace. The companion udev rule
@@ -115,7 +115,7 @@ generated units.
The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/`
2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats`
reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The
health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`,
health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`
bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
(`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs
`http://127.0.0.1:8188/` inside the container, and the container reports
@@ -135,5 +135,5 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
- [[Dankclaw Stack]]
- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
- [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
- [[Lemonade and sd.cpp]] — the ggml-family alternative
- the ggml-family alternative
- Knowledge base: [[ComfyUI]] (tools)
serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md
+4 -5
View File
@@ -52,11 +52,10 @@ graph LR
The LLM rows are bare-metal systemd services, not containers, and this repo
owns their scripts, units and knowledge base (see
[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on
). Which payload and runtime sits on
each port is era state and lives on the endpoint page, not here. `frank-npu`
(the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon`
on the Dankserver port) are both retired — see [[frank-npu]] and
[[frank-medium-radiance-mxfp4]].
on the Dankserver port) are both retired — see and
## Boot Persistence
@@ -72,7 +71,7 @@ Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-
## See Also
- [[index]] — repository index
- repository index
- [[Network Architecture]] — routing and proxy
- [[Observability]] — Grafana, Prometheus, logs
- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
- the model-serving endpoints on this host
serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md
+3 -3
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker, on a private podman bridge behind frank-tts — deployed 2026-10-18, and it publishes no port on purpose."
description: "The engine behind the fleet's speech endpoint: its runtime, private bridge, and the surface it publishes."
tags:
- dankclaw
- tts
@@ -37,7 +37,7 @@ timestamp: 2026-10-18
## Why this engine, and not the alternatives
The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in [[Local TTS and Voice Cloning]], which is the source of these numbers):
The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in , which is the source of these numbers):
| Measure | Resident `tts-server` (arm A, chosen) | Mainline per-request CLI path (arm B) |
|---|---|---|
@@ -128,6 +128,6 @@ still open.
| Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
| `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
| A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see |
| The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this |
| A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces |
serverAppConfigs-docs/Dankclaw/Frank TTS.md
+9 -6
View File
@@ -1,11 +1,13 @@
---
type: reference
description: "frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in front of frank-tts-engine; deployed 2026-10-18 and wired as the harness speech role."
description: "The fleet's cloned-voice speech endpoint and the one voice surface consumers address; wired as the harness speech role."
tags:
- endpoint
- dankclaw
- tts
- voice
- quadlet
timestamp: 2026-10-19
---
# Frank TTS
@@ -16,10 +18,10 @@ audio in that cloned voice. Everything else in the voice stack sits behind it
the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers
never learn a path.
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`,
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`
both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
the same change. Probes on the deployed endpoint: `/health` answers
`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`,
`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`
and two different voices rendered the same sentence into **different** bytes
(24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
pass-through of one default. The operator harness routes its `speech` role here;
@@ -145,7 +147,7 @@ A `/health` probe job would therefore be structurally never-green, and because
`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
Signal every few minutes for an endpoint that is fine. The engine cannot be
scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
scraped at all: it publishes nothing. Status lives in `systemctl --user status`
`podman healthcheck run`, the two unit journals, and the homepage card, which
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
2026-10-18 deploy.
@@ -153,7 +155,7 @@ probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
## The harness speech role
The operator harness reaches this endpoint through
`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`,
`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`
`baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api
`openai-speech` on the **model** row (naming a runner api at *provider* level
fails the whole file's validation and disables every custom provider, which is
@@ -169,7 +171,7 @@ harness bundle and measured the day of the flip:
`speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list,
and it never reaches an HTTP request. So a harness call arrives voice-less by
default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared
pack answers a voice-less request and the journal marks that line `(default)`,
pack answers a voice-less request and the journal marks that line `(default)`
while an **unknown** name still 400s with the list that exists.
- **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not
select a pack; it hands the vocalizer a value outside its enum. (The first
@@ -202,3 +204,4 @@ harness bundle and measured the day of the flip:
| First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
| The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
---
serverAppConfigs-docs/Dankclaw/Gitea.md
+2 -2
View File
@@ -1,5 +1,5 @@
---
type: configuration
type: reference
title: Gitea
description: Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service.
tags:
@@ -33,7 +33,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net
| `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) |
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* |
The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)
The LLM fleet knowledge base:
## Management
serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md
+1 -1
View File
@@ -30,7 +30,7 @@ entrypoint does not start the dashboard. The dashboard is a separate process, so
it runs in its own Quadlet unit
([`dankclaw/hermes/quadlet/hermes-dashboard.container`](../../dankclaw/hermes/quadlet/hermes-dashboard.container):
image `localhost/hermes-agent:latest`, `Pull=never`, `Exec=hermes dashboard
--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`,
--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`
`RestartSec=5`). All three Hermes containers mount the same `/data` bind mount.
Unlike its two siblings — `hermes.container` and `hermes-webui.container` both
serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md
+6 -6
View File
@@ -1,7 +1,7 @@
---
type: configuration
type: reference
title: Hermes-WebUI (Hermex server)
description: The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH tunnel, from the whole LAN (it binds all interfaces), and from https://hermes.franksdank.cloud (internal-only vhost).
description: "The iPhone companion app for Hermes on Dankclaw and the ways it is reached from the LAN."
tags:
- hermes
- hermes-webui
@@ -40,7 +40,7 @@ from another LAN host: `/health` → 200, `/` → 302) — **Vhost:**
## Why a container, not a host process
The gateway's state files are owned by the container-mapped uid (`109999`,
The gateway's state files are owned by the container-mapped uid (`109999`
from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` →
`dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 =
109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless
@@ -82,12 +82,12 @@ the sibling dashboard on `9119` is not.
- Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994,
shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only
*(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
- (unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
without `sudo`)*; the account can do nothing except
authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel
while the transport stays up for port forwarding.
- sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31)
scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`,
scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`
`X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are
present under `Match User tunnel` (`cat` 2026-10-18).
- Key options on the authorized key pin the blast radius:
@@ -130,7 +130,7 @@ only if inactive (it is never restarted on deploy, so in-flight browser
sessions survive), re-tags the image alias, then restarts
`hermes.service`, `hermes-dashboard.service` and
`hermes-webui.service` in the **user** manager of `dank` (linger must be on for
them to come up at boot — `loginctl show-user dank -p Linger` → `yes`,
them to come up at boot — `loginctl show-user dank -p Linger` → `yes`
2026-10-18), so gateway/dashboard/webui restarts stay isolated from
inference deploys.
Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
serverAppConfigs-docs/Dankclaw/Hermes.md
+7 -7
View File
@@ -18,7 +18,7 @@ bindings, 2026-10-18). Note the **container name drift**: the unit is
`podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so
`podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no
`ContainerName=` key even though its header comment says "container name:
hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached
`hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A
record; the name resolves from the Pi-hole hosts file,
`dankclaw/pihole/addn-hosts:39`) — **Docs:**
@@ -37,7 +37,7 @@ designed wiring; the remedy for both halves is in the log line
**Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted
at `/data` in all three Hermes containers (gateway, webui, dashboard —
`hermes.container:36`, `hermes-webui.container:46`,
`hermes.container:36`, `hermes-webui.container:46`
`hermes-dashboard.container:24`). Each
container additionally gets an anonymous volume at `/opt/data`, which is
re-created empty on every container recreation — anything that lives only there
@@ -80,7 +80,7 @@ is not durable. The tracked, sanitized configuration baseline is
`frank-dragoon` — verified from the host with `curl
http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
`127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
and is IPv4-only);
and is IPv4-only)
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
2026-10-18, that endpoint serves only `frank-archon`** (`GET
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
@@ -115,7 +115,7 @@ per session; the browser itself lives in a dedicated daemon container, the
digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host
network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows
`127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image
`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`,
`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`
`dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal
gateway and the webui server — run in the host network namespace (`podman
inspect` → `net=host` for both, 2026-10-18), so they can
@@ -248,7 +248,7 @@ What actually breaks, and the fix:
effort: none # mandatory, see below
```
Before copying this: which id is served on which port is endpoint state — see
Before copying this: which id is served on which port is endpoint state —
[[frank-zealot]] and [[frank-archon]]. The provider/model
pairing above is the shape that was landed in 2026; re-derive it from the live
config before reusing it.
@@ -282,7 +282,7 @@ Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth
on match, 401 on miss. No username field. `/health` answers 200
unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with
`{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w
%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via
%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth
those**; probe
login with a deliberately wrong password (expect 401) and the real one
(expect 200). Details on the webui surface: [[Hermes-WebUI]].
@@ -311,7 +311,7 @@ login with a deliberately wrong password (expect 401) and the real one
instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18;
off-network ingress is SSH tunnel only —
user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth
*(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*,
- (unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*
forward remote end
localhost:8787).
serverAppConfigs-docs/Dankclaw/Marreta.md
+11 -11
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Marreta on Dankclaw
description: The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet pair that shares SearXNG's Gluetun netns for egress and sits behind the Anubis + Authentik guard chain at marreta.franksdank.cloud.
description: "The content-cleaning proxy SearXNG borrows egress from: a quadlet pair in the VPN netns behind the guard chain."
tags:
- dankclaw
- marreta
@@ -19,7 +19,7 @@ returns a stripped, readable version of the page — no paywall chrome, no
tracking, no cookie walls. It is a Laravel 13 app on the
`serversideup/php:8.4-fpm-nginx` image (`composer.json` requires
`php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` —
<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>),
<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>)
with an optional headless-browser companion for JavaScript-heavy pages.
**Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained)
@@ -31,7 +31,7 @@ with an optional headless-browser companion for JavaScript-heavy pages.
- Fetches a target URL with a cascade of strategies: plain HTTP (curl), a
headless browser (Lightpanda) for JS pages, and the Internet Archive
(Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php`
files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`,
files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`
`WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`).
- Strips paywall/consent/tracking markup and re-renders clean HTML.
- Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` /
@@ -69,7 +69,7 @@ graph LR
| `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) |
Both units are boot-persistent (`WantedBy=default.target`) and self-healing
(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`,
(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`
`marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live
2026-10-18 both report `systemctl --user is-enabled` → `generated` and
`is-active` → `active`, and `podman inspect` reports marreta **healthy**
@@ -87,14 +87,14 @@ The fix is a single env var, verified against the serversideup source:
- The image sets `ENV NGINX_HTTP_PORT="8080"` and the site template is
`listen ${NGINX_HTTP_PORT} default_server;`.
- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build);
- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build)
the entrypoint `10-init-webserver-config.sh` runs `envsubst` over every
`NGINX_*` variable **at container start**.
- So `Environment=NGINX_HTTP_PORT=8085` in the quadlet relocates the listen
port off the collision with no custom build.
Marreta then publishes **nothing** of its own — the netns owner (Gluetun)
publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`),
publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`)
exactly as it publishes `8081:8080` for SearXNG. Caddy dials
`192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the
relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085`
@@ -127,7 +127,7 @@ on the mechanism: `BrowserFetchStrategy.php` only calls
(`dankclaw/Caddyfile:159-163`).
2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies
whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`).
3. **Post-auth routing** — authenticated traffic returns to `http://:8017`,
3. **Post-auth routing** — authenticated traffic returns to `http://:8017`
where `@marreta host marreta.franksdank.cloud` routes to
`reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`).
@@ -164,7 +164,7 @@ Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0
— that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and
this container sets no `TRUSTED_PROXIES`, so the default is what runs
(`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL`
starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`,
starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`
`URL::forceScheme('https')`). With
`APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded
scheme and client IP through the whole chain, so there is no redirect loop
@@ -234,7 +234,7 @@ netns down and would otherwise orphan it).
Deployed via `deploy.sh dankclaw --service gluetun --service marreta --service
caddy` (gluetun first to recreate the tunnel with the new `8082:8085` publish,
which also cycles SearXNG's netns ref). Authentik `ProxyProvider` `marreta`
registered (pk 13, `external_host=https://marreta.franksdank.cloud/`,
registered (pk 13, `external_host=https://marreta.franksdank.cloud/`
`internal_host=http://caddy:8017`). Results:
- Unauth edge → Anubis challenge page (not 502). TLS cert issued via DNS-01.
@@ -245,14 +245,14 @@ registered (pk 13, `external_host=https://marreta.franksdank.cloud/`,
(`GET 127.0.0.1:9222/json/version` → `Lightpanda/1.0`).
- SearXNG unaffected: `:8081` → 200 after the tunnel recreate.
> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`,
> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`
> which 500s every app route (see Secrets above). Fixed by re-keying with the
> `base64:` prefix before any encrypted data was persisted.
## Citations
- Marreta upstream: <https://github.com/manualdousuario/marreta> (README,
`compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`,
`compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`
`lang/` directory listing).
- serversideup/php fpm-nginx port mechanism: `serversideup/docker-php`
`src/variations/fpm-nginx/Dockerfile` (`NGINX_HTTP_PORT="8080"`) and
serverAppConfigs-docs/Dankclaw/Open Notebook.md
+2 -2
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out — script written by frank-zealot, voices rendered by frank-tts, no cloud provider anywhere in the path."
description: "The fleet's podcast generator: notes in, multi-speaker episode out, script and voices both served in-house."
tags:
- dankclaw
- open-notebook
@@ -13,7 +13,7 @@ timestamp: 2026-10-18
open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook)) is the notebook layer that turns existing notes and sources into a **spoken episode**: an outline, then a dialogue transcript, then one synthesized clip per speaker turn, concatenated with ffmpeg into an MP3. In the fleet it is the consumer that closes the loop Frank's audio stack was built for — the script is written by **frank-zealot**, the voices are rendered by **frank-tts**, and nothing leaves the LAN.
**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Deploying it is a deliberate next step, not a pending one.
**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Re-censused 2026-10-19 and unchanged: no `*notebook*` unit in `systemctl --user list-unit-files`, nothing installed under `~/.config/containers/systemd/` by that name, no container in `podman ps` — the repo's three quadlet files have never been installed on the host. Deploying it is a deliberate next step, not a pending one.
**Host:** Dankclaw (rootless Podman, user `dank`) — **Ports:** 8502 UI and 5056 API, published on the LAN, **no vhost and no Caddy route** — **Deployer:** `open-notebook` (one service, two units) — **Docs:** https://github.com/lfnovo/open-notebook
serverAppConfigs-docs/Dankclaw/OpenWebUI.md
+10 -13
View File
@@ -111,7 +111,7 @@ The Ollama connection is disabled (`ollama.enable=false`).
- Working dir for imports: `/app/backend`; package name `open_webui`
(underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
BEFORE importing any `open_webui` module (import-time hard requirement);
BEFORE importing any `open_webui` module (import-time hard requirement)
the secret lives at `/app/backend/.webui_secret_key`.
- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
module (`podman exec open-webui python3 -c "import sqlite3;
@@ -122,8 +122,8 @@ The Ollama connection is disabled (`ollama.enable=false`).
pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
`status`/`enabled` attribute — use `is_active` from the DB.
- **API auth for verification:** `create_token` lives in
`open_webui.utils.auth` (NOT `open_webui.utils.utils`);
`get_verified_user` requires an `id` claim (not just `sub`);
`open_webui.utils.auth` (NOT `open_webui.utils.utils`)
`get_verified_user` requires an `id` claim (not just `sub`)
`SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
container:
@@ -148,7 +148,7 @@ The Ollama connection is disabled (`ollama.enable=false`).
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
`host.containers.internal`), not a model problem.
**Durability:** the model table + connections are DB-only (not git-managed);
**Durability:** the model table + connections are DB-only (not git-managed)
a fresh data volume loses them — including the public access grant above.
For reproducibility, add an idempotent first-start seed script or document
the setup in the service doc.
@@ -156,7 +156,7 @@ the setup in the service doc.
## API access (auth-gated)
Every `/api/*` endpoint requires an authenticated session (SSO login JWT)
or a per-user API key (`auth.enable_api_keys=true`; keys are minted per
or a per-user API key (`auth.enable_api_keys=true`; keys are minted
account under Profile → API Keys). Anonymous web visitors get the `pending`
role (`ui.default_user_role=pending`), which cannot chat. Verified
2026-10-07 from loopback `:3005` and over the public vhost:
@@ -166,13 +166,12 @@ bogus bearer token); only `/api/config`, `/api/version` and `/health`
`POST /api/chat/completions` all return **401** with no key (also with a bogus bearer
token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of
them expose the model list. An external program needs an account + API key — there is
no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`,
no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`
`/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and
`POST /api/chat/completions` → 401; `/api/config` additionally publishes
`features.enable_login_form=false`, `features.enable_signup=false` and
`oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside.
## Web search
Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
@@ -186,7 +185,7 @@ Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
The Anubis→Authentik guard chain is a browser mechanism (proof-of-work +
session flow); a plain HTTP client cannot solve it, and the internal port
bypasses Caddy entirely.
- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`,
- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`
a hardcoded Python constant — no config knob). The `pass_ip` list is checked
FIRST and short-circuits the whole botdetection chain, and OpenWebUI's bridge
egress (`10.89.x`) is in `pass_ip`, so the cap never applies. Verified: an
@@ -249,7 +248,6 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
restore the password form: set `ui.enable_login_form` back to `true` in the
Admin Panel (or DB) and drop the two env vars from the quadlet.
## Boot persistence (Quadlet)
The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
@@ -296,7 +294,7 @@ TTS is the fleet's one OpenAI-compatible speech endpoint —
`http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url`
in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That
port is the rootless quadlet `frank-tts` (this repo:
`dankclaw/voiceservice/quadlet/frank-tts.container`,
`dankclaw/voiceservice/quadlet/frank-tts.container`
`scripts/frank-tts.env`), and its shim resolves `voice=<name>` against
`/home/dank/voices/<name>/reference.wav` — a name without a reference is a
400 listing the voices that exist, never a silent fallback.
@@ -310,7 +308,7 @@ setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
`browser-kokoro` is chosen the page synthesizes locally and rejects any
voice outside its fixed pack — so `narrator1` fails with
`Voice: "narrator1" not found. Should be one of: af_heart, …` **without a
single request reaching the server**. Personal audio settings are per
single request reaching the server**. Personal audio settings are
browser: no `user.settings` row and no container env carries an audio
override, yet the toast still appears ⇒ the choice lives in that browser's
own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**,
@@ -325,7 +323,6 @@ the failing click — that contrast is the instrument: a server-path failure
leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves
neither.
## Retired instances
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -355,7 +352,7 @@ neither.
began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
id-keyed objects had to move in the same window as the alias flip:
`model.id`+`name` (`frank-zealot` → `frank-zealot`, `params`
`{"context":{"length":200704}}` and the `meta.description` kept),
`{"context":{"length":200704}}` and the `meta.description` kept)
`access_grant` (`resource_type='model'`, `resource_id`, principal
`user`/`*`, permission `read`), and `openai.api_configs` per-backend
`model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1
serverAppConfigs-docs/Dankclaw/Postgres.md
+1 -1
View File
@@ -88,7 +88,7 @@ never enter this repo. See [[Gitea]] and [[Authentik]].
## Boot persistence (Quadlet)
Both units carry `Restart=always` (`gitea-postgres.container:28`,
Both units carry `Restart=always` (`gitea-postgres.container:28`
`authentik-postgres.container:38`) and `[Install] WantedBy=default.target`
(`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed
under `~/.config/containers/systemd/`, and report `generated` / `active` with
serverAppConfigs-docs/Dankclaw/SearXNG Providers.md
+6 -6
View File
@@ -143,7 +143,7 @@ engines:
disabled: false
```
Key per-engine keys (each overwrites the engine-module default; see
Key per-engine keys (each overwrites the engine-module default
`settings_engines.html`):
| Key | Effect |
@@ -164,7 +164,7 @@ Key per-engine keys (each overwrites the engine-module default; see
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
`google2` still returns results, and vice-versa.
**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
**Parser caveat (read before fiddling with `parsers`):** the modern `google`
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
[scrapy_html]` under `name: google` mostly does nothing. To make Google
@@ -376,14 +376,14 @@ done
**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. It answers 200 on this instance today
(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`,
(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`
`categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
[[SearXNG]]). `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.
**3. `check.searx.space`** — the hosted official checker probes every engine of
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
plain HTTP probe misses) and posts results on searx.space. Add your instance via
plain HTTP probe misses) and posts results on searx.space. Add your instance
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
to reach the SearXNG port.
@@ -417,10 +417,10 @@ misleading).
already replaced — checked 2026-10-18; see [[SearXNG]]). It is
also immune to the shared-VPN-exit-IP blocking this page is about: it
authenticates, so reputation of the exit IP is irrelevant.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to
exactly which engines are down and why. Confirm the running SearXNG version
(check whether `parsers`/`rate_limit` exist in the pinned build).
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds
upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
(`:27`), and retries once (`:32`). This addresses
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
serverAppConfigs-docs/Dankclaw/SearXNG.md
+16 -17
View File
@@ -108,7 +108,7 @@ engines:
disabled: true
```
The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas);
The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas)
the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked
`secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is
censused below.
@@ -154,7 +154,7 @@ what keeps the `trusted_proxies` entry meaningful.
## Unblocking Google & Search Providers
The current request policy allows a modest amount of VPN latency (`5.0s` per
The current request policy allows a modest amount of VPN latency (`5.0s`
request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting
retries to one (`:32`). This can recover slow
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
@@ -169,10 +169,10 @@ upstream request volume. SearXNG **suspends** offending engines — which is wha
up as "Google went unavailable". The default suspension is shorter than the folklore:
see the table in [[SearXNG Providers]]. This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
if Google still re-blocks — route the Google engines at a clean-IP proxy via
if Google still re-blocks — route the Google engines at a clean-IP proxy
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
root cause, the suspend-model table, the `engines:` config, and the
`/json` verification recipe.
@@ -212,17 +212,16 @@ root cause, the suspend-model table, the `engines:` config, and the
(`searxng.container:21`), and that is one of the variables SearXNG substitutes at
load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a
config change, out of scope for docs.
- Keep the valkey cache container on the network SearXNG can actually reach — see
- Keep the valkey cache container on the network SearXNG can actually reach —
[[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's
netns), so the network that matters is the one Gluetun and `searxng-valkey` share.
## Kagi (paid API engine)
Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native
`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`),
`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`)
which sets `Authorization: Bearer <key>` (`kagi.py:103`) and issues
`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`),
`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`)
and declares `require_api_key: True` (`kagi.py:63`).
- **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
@@ -231,10 +230,10 @@ and declares `require_api_key: True` (`kagi.py:63`).
`enabled: true`, categories `general, web` (2026-10-18).
- **Cost control**: `paging: false` — exactly one API credit per aggregated
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0)
(`shared/searxng/settings.yml:74-77`; upstream paging default is `True`,
(`shared/searxng/settings.yml:74-77`; upstream paging default is `True`
`kagi.py:67`, so the `false` here is a real override).
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
`deploy-remote.sh` bakes the real key into the live settings.yml via
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`
`deploy-remote.sh` bakes the real key into the live settings.yml
`scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
(owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are
the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`).
@@ -262,15 +261,15 @@ and declares `require_api_key: True` (`kagi.py:63`).
- 262 engines in the build, **75 enabled**, categories of the enabled set led by
`other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7).
- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`,
`github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`,
`arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`,
`wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`,
- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`
`github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`
`arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`
`wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`
`youtube`, `vimeo`, `dailymotion`, `bing videos`.
- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`,
- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`
`brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`.
- The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target
engines that **no longer exist in this build** — `/config` lists no `startpage`,
engines that **no longer exist in this build** — `/config` lists no `startpage`
`startpage images` or `startpage news`, so those overrides are no-ops kept as
intent. *(labelled 2026-10-18)*
serverAppConfigs-docs/Dankclaw/Seerr.md
+3 -3
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line)."
description: "The request manager for the *arr stack — the maintained successor to the archived Overseerr line."
tags:
- dankclaw
- seerr
@@ -40,8 +40,8 @@ the next outpost reload), and the missing app bounced post-login to
authentik's own user UI instead of Seerr. Both backfilled live: the three
standard flows copied from a known-good provider, plus an application
(`seerr-guard`) linked to the provider. Verified by the hardened SSO sweep
(`tests/e2e/specs/13-sso.spec.ts`) — see [[Authentik]] for the two
provisioning rules this incident established.
(`tests/e2e/specs/13-sso.spec.ts`) — see [[Authentik Outpost Reference]] for the
two provisioning rules this incident established.
## *arr wiring
serverAppConfigs-docs/Dankclaw/Swap Configuration.md
+4 -4
View File
@@ -1,7 +1,7 @@
---
type: configuration
type: reference
title: Dankclaw Swap Configuration
description: The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit. Covers why it was raised from 8 GiB, the safe resize ordering that avoids an OOM on a host whose MemAvailable is inflated by pinned model weights, and the swappiness caveat.
description: "The host swapfile: why it is sized as it is, the resize ordering that avoids an OOM, and the swappiness caveat."
tags:
- dankclaw
- host
@@ -144,8 +144,8 @@ frank-dragoon `health=200` throughout.
The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at
`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging
directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only
`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`,
`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`,
`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`
`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`
`20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`.
The old swapfile has therefore been purged; nothing on the host still holds it.
serverAppConfigs-docs/Dankclaw/Voice Packs.md
+7 -7
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material gate scripts/build-voice-pack.sh enforces, how to add a fourth voice, and what to do when a pinned engine bump changes the extractor's output sizes."
description: "How a voice pack is built: the files every pack must carry, the source-material gate, adding a voice, engine-bump drift."
tags:
- dankclaw
- frank-tts
@@ -23,7 +23,7 @@ voice is how a wrong-voice endpoint stays wrong for weeks.
state that stays out of the repo, mounted read-only at `/voices` by the façade
unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in
`scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:**
arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]].
arm A `qwentts.cpp`, decided in .
Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade
container runs as uid 10000 and reads the mount from outside its own uid map.
@@ -137,7 +137,7 @@ invariant to every crop in the tracked pack list.
1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop
offset is a claim about one exact file, so the digest goes in the env with it.
2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once:
`ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`,
`ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`
build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest
on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the
`.rvq` falls under the floor.
@@ -188,15 +188,15 @@ loud, not a bug. Work through it in order:
1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec`
prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , …
codebooks @ … Hz` on every run, and the pinned tree's
`tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`;
`tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`
`codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`.
2. Decide whether the new number is intended. A changed `enc_dim` means the old
`.spk` files are unreadable by the new engine, so **every** pack must be
rebuilt, not just the failing one.
3. Update the four arithmetic lines in `scripts/voice-packs.env`
(`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`,
(`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`
and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and**
`VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`,
`VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`
`VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes
without moving the commit is the mistake to avoid.
4. Rebuild all three packs with `--force` (the retired directories keep the old
@@ -226,7 +226,7 @@ loud, not a bug. Work through it in order:
narrator1 and narrator2). The builder's first real run is therefore a
migration, and it should be a `--force` run per voice so the old directories
are retired rather than left half-updated.
- The three source files (`/home/blank/audiosample.mp3`,
- The three source files (`/home/blank/audiosample.mp3`
`audiosample2.m4b`, `audiosample3.m4b`, digests pinned in
`scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the
builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so
serverAppConfigs-docs/Dankserver/Calibre.md
+1 -1
View File
@@ -80,7 +80,7 @@ image's own nginx (the desktop), so the content server runs on internal 8090.
**Auth model (calibre 9.x):** users live in a SQLite userdb whose `pw` column
is **plaintext** — the file is mode 600. The single admin account is
bootstrapped once by the deploy handler
(`deploy_calibre_server` in `scripts/deploy-remote.sh`) via
(`deploy_calibre_server` in `scripts/deploy-remote.sh`)
`calibre-server --manage-users -- add`, the password piped on stdin with
`printf` (a trailing newline from `echo` lands in the stored password and
breaks every login). The desktop's basic auth (`CUSTOM_USER`/`CUSTOM_PASSWORD`)
serverAppConfigs-docs/Dankserver/Copyparty.md
+15 -8
View File
@@ -83,7 +83,7 @@ The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum co
## Volumes
Live deployment: the whole LUKS array is served at URL `/w` via
Live deployment: the whole LUKS array is served at URL `/w`
`Volume=/mnt/entropy:/w:ro` in the quadlet unit ([[Entropy Volume]]) —
read-only **at the mount level**, so no containerized consumer can write into
the array through this path regardless of account permissions. The ac edition
@@ -111,7 +111,7 @@ convention; see [[#Command-line interface (ac edition)]] above.
```
`${COPAY_ACCOUNT}`/`${COPAY_PASSWORD}` are **never** written into the unit:
they live in the host-local `/home/dank/Containers/copyparty/env` (0600),
they live in the host-local `/home/dank/Containers/copyparty/env` (0600)
loaded via `[Service] EnvironmentFile=` so systemd expands them into
`Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD}` at start. The password still
lands in the container argv — visible to anyone who can run
@@ -143,13 +143,13 @@ Copyparty v1.20.9+ supports WOPI (Web Application Open Platform Interface) for e
## Security Advisories
| Version | CVE | Issue |
|---------|-----|-------|
| v1.20.19 | GHSA-phv8-wgjp-g4p9 | FTP upload outside volume |
| v1.20.17 | GHSA-x5pq-m9p8-f4vx | dirkey/filekey confusion |
| v1.20.9 | CVE-2026-27948 / GHSA-62cr-6wp5-q43h | XSS via file upload |
| Version | Advisory | CVE | Issue |
|---------|----------|-----|-------|
| v1.20.19 | GHSA-phv8-wgjp-g4p9 | CVE-2026-104102 | FTP upload outside the defined volumes (limited path traversal) |
| v1.20.17 | GHSA-x5pq-m9p8-f4vx | CVE-2026-70657 | dirkey/filekey confusion |
| v1.20.9 | GHSA-62cr-6wp5-q43h | CVE-2026-27948 | reflected XSS via the `?setck=` URL parameter (not the upload path — the separate upload/DOM-XSS advisory is GHSA-m2jw-cj8v-937r) |
Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`.
Upgrade to v1.20.19+ to cover all known advisories. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`. Each row's patched version is the advisory's own `first_patched_version`.
## Gotchas
@@ -174,3 +174,10 @@ Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters
- [[Dankserver Stack]]
- [[Caddy]] — TLS termination and authentication in front
- [[ARR-Stack]] — Copyparty as a viewer alongside automated download management
## Citations
- GitHub Security Advisories, `9001/copyparty` —
https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19)
each row's patched version read from `first_patched_version` in the
`https://api.github.com/advisories/<GHSA-id>` response.
serverAppConfigs-docs/Dankserver/Dankserver Stack.md
+2 -2
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "Dankserver (192.168.0.235) runs the media and library stack."
description: "Service inventory and routing overview for the media and library host."
tags:
- dankserver
timestamp: 2026-09-03
@@ -54,5 +54,5 @@ The following services ran on Dankserver and are now retired from this stack:
## See Also
- [[index]] — repository index
- repository index
- [[Network Architecture]] — routing and proxy
serverAppConfigs-docs/Dankserver/Jellyfin.md
+4 -4
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Jellyfin
description: Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and hardware-transcode status.
description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
tags: [jellyfin, dankserver, media-server]
timestamp: 2026-09-03T00:00:00Z
---
@@ -14,7 +14,7 @@ serving music, movies, and shows over LAN. It replaced the earlier research
brief; it runs alongside [[Plex]], which remains the primary server for some
clients.
**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`,
**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`
public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
[[Authentik]] outpost (post-auth route proxies `localhost:8096`).
@@ -67,7 +67,7 @@ user to verify the password.
- LDAP provider `Jellyfin LDAP` (Base DN `dc=franksdank,dc=cloud`, bind/search
mode `direct`) + application `Jellyfin`.
- LDAP outpost `Jellyfin LDAP Outpost` → container `authentik-outpost-ldap`
(quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`),
(quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`)
publishing `389`/`636` on the Dankclaw host for cross-host binds.
- Service account `svc-jellyfin-ldap` (DN
`cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud`).
@@ -80,7 +80,7 @@ user to verify the password.
**Jellyfin side** (plugin config, set via admin → Plugins → LDAP-Auth):
- Server `192.168.0.143:389` (plain LDAP over the trusted LAN), Base DN
`dc=franksdank,dc=cloud`, bind user = the service-account DN, search
attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`,
attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`
`CreateUsersFromLdap=true`, all folders enabled.
**Gotcha — the bind credential is a PASSWORD, not a token.** The LDAP bind is
serverAppConfigs-docs/Dankserver/Plex.md
+1 -1
View File
@@ -60,7 +60,7 @@ root runs as the `dank` user, and the deployer adopts the data root with
## Integration points
- **Caddy:** Plex is exempt from the Anubis bot challenge
(plain-HTTP legacy clients cannot solve JS challenges) — see
(plain-HTTP legacy clients cannot solve JS challenges) —
[[Anubis]] "Exempt by construction".
- **Tautulli** ([[Tautulli]]) is the companion: it polls PMS on the appnet
for statistics and notifications and makes no sense without Plex.
serverAppConfigs-docs/Dankserver/RomM-Scraping.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver."
description: "Metadata providers, cover art, ROM naming and the scraping workflow behind the RomM library."
tags:
- dankserver
- romm
serverAppConfigs-docs/Dankserver/RomM.md
+2 -2
View File
@@ -32,7 +32,7 @@ Anubis → Authentik chain ([[Caddy]] routes it post-auth to
## Architecture
Current RomM releases require MariaDB plus Redis (or Valkey) for the task
queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit),
queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit)
not valkey.
```mermaid
@@ -114,7 +114,7 @@ scan:
## Deployment (Quadlet)
Since 2026-09-01 the trio runs as repo-tracked quadlet units, deployed with
`./scripts/deploy.sh dankserver --service romm` (any of `romm`,
`./scripts/deploy.sh dankserver --service romm` (any of `romm`
`romm-mariadb`, `romm-redis` triggers the full ordered cutover):
- Units: `dankserver/romm/quadlet/{romm,romm-mariadb,romm-redis}.container`
serverAppConfigs-docs/Dankserver/mariadb.md
+1 -1
View File
@@ -1,7 +1,7 @@
---
type: reference
title: MariaDB (RomM Backend)
description: MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and backup path.
description: "The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path."
tags: [mariadb, romm, dankserver]
timestamp: 2026-09-03T00:00:00Z
---
serverAppConfigs-docs/Dankserver/redis.md
+1 -1
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Redis (RomM Cache / Queue)
description: Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup path.
description: "The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path."
tags: [redis, romm, dankserver]
timestamp: 2026-09-03T00:00:00Z
---
serverAppConfigs-docs/Desktop/Noctalia and Niri.md
+12 -12
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Noctalia and Niri
description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
description: "The noctalia shell on niri across workstations: config layers, remote IPC, custom palettes, output rotation and pinning."
tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -45,14 +45,14 @@ Verified: `msg status` returns live JSON state; unknown panel ids return an
error listing all valid panel ids (clipboard, control-center, launcher,
polkit, session, setup-wizard, test, tray-drawer, wallpaper).
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`
`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`
`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`
`media next|previous|playPause` (graceful no-op, rc 0, with no active
player), `brightness-up [target] [step]`, `brightness-down`,
`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
player), `brightness-up [target] [step]`, `brightness-down`
`settings-open [context]`, `settings-toggle [context]`, `config-reload`
`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`
`color-scheme-get|set`, `theme-mode-get`, `templates-apply`
`screenshot-fullscreen`.
**Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
@@ -81,9 +81,9 @@ Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
synced. Missing/invalid file → silent fallback to the builtin palette.
Palette JSON shape = the community file's shape: `dark`/`light` maps of
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`
`mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`
`cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
-o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
*generator* format, not this one. Copy an existing community JSON and mutate
@@ -114,7 +114,7 @@ Procedure:
`noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
`color-scheme-get`.
5. Re-render every configured template: `noctalia msg templates-apply`.
Regenerates `~/.config/ghostty/themes/noctalia`,
Regenerates `~/.config/ghostty/themes/noctalia`
`~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
starship, etc. Symlinked-into-dotfiles paths show up as modified files
worth committing.
@@ -216,7 +216,7 @@ Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
scale, transform, logical pos/size) and `niri --version`. Output **names are
connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
the EDID strings printed before them are decoration. On blankpad the live
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`
and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
repo path (that is what gets committed); check with `ls -ld`.
serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md
+3 -3
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "*arr Stack"
description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every container on the gluetun WireGuard netns (all egress VPN): topology, wiring, auth posture, and gotchas."
description: "The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas."
tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
timestamp: 2026-09-04T00:00:00Z
---
@@ -38,7 +38,7 @@ live call at a time instead of guessing payload shapes:
- `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
- `DefaultMetadataProfileId must be > 0` → fetch real ids from
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
`GET /api/v1/metadataprofile` (rows like `(1,'Standard')`)
`GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
profiles.
- `targetType null` → configContract not registered in that build (this
@@ -87,7 +87,7 @@ settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
pipeline hides a miss and you conclude "empty field" when the grep itself
failed.
4. Decision framing: when posture = gated-with-zero-users, present it as a
decision menu (None behind the front-door wall vs External/authentikarr),
decision menu (None behind the front-door wall vs External/authentikarr)
not a silent change — first-account-wins means whoever logs in first owns
the instance.
serverAppConfigs-docs/FrankBook-Pro/index.md
+7 -7
View File
@@ -7,10 +7,10 @@ Gluetun WireGuard netns. The host overview lives at [[FrankBook-Pro]]
## *arr Stack
- [[ARR-Stack]] — the full *arr stack: indexers, managers, and the download client
- [[sonarr]] — TV show management
- [[radarr]] — movie management
- [[lidarr]] — music management
- [[readarr]] — ebook/audiobook management
- [[prowlarr]] — indexer management
- [[qbittorrent]] — torrent download client
- [[ARR-Stack]] — The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas.
- [[sonarr]] — Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts.
- [[radarr]] — Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts.
- [[lidarr]] — Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts.
- [[readarr]] — The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client.
- [[prowlarr]] — Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas.
- [[qbittorrent]] — The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe.
serverAppConfigs-docs/FrankBook-Pro/prowlarr.md
+4 -4
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Prowlarr
description: Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and verified live gotchas.
description: "Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas."
tags: [prowlarr, arr-stack, frankbook-pro]
timestamp: 2026-09-03T00:00:00Z
---
@@ -70,7 +70,7 @@ http://prowlarr:9696/{indexerId}/download?link=...&file=... (torrent handoff)
The global API key is in `config.xml` (`<ApiKey>`); per-application keys are
only used by Prowlarr's own sync calls, NOT by the feeds. The arr download
clients authenticate to qBittorrent with the permanent WebUI password stored
in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — see
in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver —
[[qBittorrent]] for why that password must be pinned in the conf file.
**Readarr caveat:** Readarr is abandoned upstream; its qBittorrent client
@@ -97,13 +97,13 @@ successor; the indexer row itself is healthy.
- Categories in the indexer row must be a subset of what the feed's `caps`
advertises; extras log "does not support the following categories" and
some feeds reject them.
- **A freshly created indexer row defaults to all-off.** If `enableRss`,
- **A freshly created indexer row defaults to all-off.** If `enableRss`
`enableAutomaticSearch`, or `enableInteractiveSearch` aren't set, the
app logs `Searching indexers for [X]. 0 active indexers` (Lidarr,
verified). Search proves TV + music chains 2026-08-30: Sonarr returned
76 live releases for an anime term, Lidarr 18 for YOASOBI – THE BOOK,
all via the Prowlarr-Nyaa feed.
- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`);
- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`)
GET responses carry `fields: []` — real settings values live in
`prowlarr.db` → `Applications.Settings` JSON (read-only inspection).
- Instance name/`InstanceName=Prowlarr` in `config.xml` does NOT prefix the
serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md
+3 -3
View File
@@ -1,7 +1,7 @@
---
type: reference
title: qBittorrent
description: qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns, PBKDF2 WebUI credential recipe.
description: "The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe."
tags: [qbittorrent, arr-stack, frankbook-pro, gluetun]
timestamp: 2026-09-03T00:00:00Z
---
@@ -13,7 +13,7 @@ qBittorrent is the torrent download client for the *arr stack. The stack runs on
not on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr,
and Readarr via category tagging.
**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`,
**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`
published by the [[Gluetun]] unit because qBittorrent shares that container's
network namespace (a netns-sharer publishes no ports of its own).
@@ -121,7 +121,7 @@ Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
`podman machine ssh`.
**Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
(NOT core — `core` is the podman-machine VM user, reached only via
(NOT core — `core` is the podman-machine VM user, reached only
`podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
low MaxAuthTries: when password-bootstrapping use `-o
serverAppConfigs-docs/FrankBook-Pro/readarr.md
+17 -9
View File
@@ -1,18 +1,19 @@
---
type: reference
title: Readarr
description: Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB download client.
description: "The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client."
tags: [readarr, arr-stack, frankbook-pro]
timestamp: 2026-09-03T00:00:00Z
---
# Readarr
> **RETIRED UPSTREAM (May 2026).** The Servarr Team retired Readarr: its
> metadata became unusable and the community migration to Open Library stalled.
> There is no upstream development. Alternatives: the Bookshelf community fork,
> or the Calibre content server. This instance still runs on a pinned image
> tag; see below.
> **RETIRED UPSTREAM (announced 2025-06-27).** The Servarr Team retired
> Readarr: its metadata became unusable and the community migration to Open
> Library stalled. There is no upstream development — the GitHub repository is
> archived and its last push is 2025-06-27. Alternatives: the Bookshelf
> community fork, or the Calibre content server. This instance still runs on a
> pinned image tag; see below.
Readarr manages ebooks and audiobooks in the *arr stack. The stack runs on
**FrankBook-Pro** (the Mac mini) as quadlet units inside its podman machine —
@@ -69,9 +70,9 @@ Restart=always
## Gotchas
- Readarr is abandoned upstream (retired May 2026; last line was 0.4.x before
the final tags). The pinned `amd64-1.0.980` tag exists precisely because
`:latest` does not exist for this image on LSIO.
- Readarr is abandoned upstream (retired 2025-06-27; last line was 0.4.x
before the final tags). The pinned `amd64-1.0.980` tag exists precisely
because `:latest` does not exist for this image on LSIO.
- Calibre integration is folder-based, not API-based. Readarr drops files into
the books root which [[Calibre-Web]] serves.
- Not all indexers carry all formats (epub, mobi, pdf). Goodreads/StoryGraph
@@ -81,3 +82,10 @@ Restart=always
- [[ARR-Stack]] — full *arr stack documentation
- [[Calibre-Web]] — ebook web frontend that serves Readarr downloads
## Citations
- Servarr wiki, *Readarr* (Retired) — "Announcement: Retirement of Readarr
2025-06-27" — https://wiki.servarr.com/readarr (accessed 2026-10-19)
- `Readarr/Readarr` on GitHub: `archived: true`, last push 2025-06-27 —
https://api.github.com/repos/Readarr/Readarr (accessed 2026-10-19)
serverAppConfigs-docs/Infrastructure/Alertmanager.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through..."
description: "Receives firing rules from Prometheus, groups and dedupes them, and forwards each group to the host-side Signal relay."
tags:
- infrastructure
timestamp: 2026-09-17
serverAppConfigs-docs/Infrastructure/Anubis.md
+3 -3
View File
@@ -93,7 +93,7 @@ and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
it is exempt by absence — and therefore unauthenticated on the LAN at
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`
`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
(`dankclaw/Caddyfile:283-423`).
@@ -140,7 +140,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
instead of allowed, the browser's fetch throws and it renders its own
error page (iOS/macOS browsers phrase it as "the request failed and the
interceptors did not return an alternative response") instead of the
login flow. Verify with a JS-capable client end-to-end — mint via
login flow. Verify with a JS-capable client end-to-end — mint
`make-challenge`, solve, submit with browser-exact query params — before
declaring a challenge chain healthy.
7. **Single-file bind mount + git pull = stale policy.** The policy is
@@ -158,7 +158,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
answers any request lacking `X-Real-Ip` with a 500 whose log line reads
`[misconfiguration] X-Real-Ip header is not set`. The real chain always
sets it (every guarded Caddy listener carries `header_up X-Real-Ip
{remote_host}` — the bats seam enforces the line), so users never see
{remote_host}` — the bats seam enforces the line), so users never
this. But any direct probe of `:8923` (scratch container, curl, a new
reverse-proxy hop) MUST set the header or it looks like a policy failure.
Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`.
serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md
+230
View File
@@ -0,0 +1,230 @@
---
type: reference
description: "Outpost, proxy-provider, OAuth2, LDAP and flow configuration for the homelab's Authentik SSO."
tags:
- infrastructure
- authentik
timestamp: 2026-10-19
---
# Authentik Outpost Reference
Configuration lookup for the [[Authentik]] deployment: the live proxy-provider
inventory behind the guarded vhosts, the provisioning rules the embedded outpost
enforces, this fork's OAuth2 relying-party quirks, the LDAP provider objects, the
self-serve user-settings flow, and the live database shape. What the service is,
how it is deployed, wired and operated is [[Authentik]]; the three core
containers are [[Authentik Processes]].
## Live proxy-provider inventory
Live provider inventory (read from the `authentik-postgres` container with
read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
members in `authentik_outposts_outpost_providers`), all with
`internal_host = http://caddy:8017`, all carrying the **same three flow
UUIDs** (`default-authentication-flow`
`default-provider-authorization-implicit-consent`
`default-provider-invalidation-flow`), and each backed by an Application
(the `slug` column below):
| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |
| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |
That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
the lockstep rule in [[Authentik]] holds today. Two corrections against the 2026-09-02 table:
**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
not proxy providers, so the proxy outpost does not deserialize them: pk 9
`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
below rejects), and pk 14 `Open WebUI`
(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
`external_host` (the runbook-recipe style) and bare origins both intercept
correctly — the outpost matches on hostname.
## Provisioning rules for proxy providers
Two provisioning rules, each verified live by an outage:
1. **All three flows are required.** A ProxyProvider created via raw API/ORM
with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
null wedges the embedded outpost's config fetch: the Rust side rejects the
whole config JSON (serde: `invalid type: null, expected a formatted UUID
string`), logs `Failed to fetch outpost from API, retrying in 3 seconds`
forever, and **everything behind the outpost dies at the next outpost
reload** — including hosts whose providers were fine (verified 2026-09-02
when three providers provisioned without any flows took down every guarded
vhost after a server restart). Backfill all three flows (copy from pk 1:
`default-authentication-flow`
`default-provider-authorization-implicit-consent` (the implicit-consent
authorization flow, `default-invalidation-flow`) and the retry loop
recovers on its next 3-second cycle. The silent killer: while the outpost
process keeps its last good in-memory config, everything works — the
outage only surfaces on restart/reload.
2. **Every proxy provider needs an attached Application.** The outpost
resolves the post-login redirect through the provider's application; a
provider with no app completes the login (authentik session established)
but bounces to authentik's own user UI (`/if/user/#/library`) instead of
the app. No error is logged anywhere — the SSO "works" and the app never
appears. Create the Application with `provider_id` set (the UI does this
automatically; raw API/ORM drives must do it explicitly).
## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
implementation diverges from upstream in ways that bit us; all verified live:
- **Application row required.** `OAuth2Provider.objects.create()` does not
create the single-table-inheritance `Application` row; without it the flow
executor can never complete an authorize request. Create both rows.
- **`grant_types` defaults to empty.** Until the provider's `grant_types`
contains `"authorization_code"` (enum value with underscore), the token
endpoint rejects every exchange with `invalid_request`.
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
`parse_token_request` see no `grant_type` → `unsupported_grant_type`
POST-body client secrets are not accepted — authenticate the confidential
client with `Authorization: Basic base64(client_id:client_secret)`.
- **Provider resolution is by `client_id` query param** on the global
`/application/o/authorize/` route; per-slug authorize paths do not exist in
this build. Token/userinfo live at `/application/o/token/` and
`/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
`PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
`BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
becomes the *expression source* instead of the context; every call raises a
swallowed `PropertyMappingExpressionException` and the claim is skipped.
Scope-grant filtering has the same class of bug: `ScopeMapping` has no
provider FK (linkage lives in the `ProviderPropertyMapping` through model)
so `filter(provider=...)` always yields an empty queryset and tokens asked
to carry scopes get zero. Net effect: userinfo/introspection answers with
`{sub}` only — email/groups/entitlements claims cannot be delivered by this
build until the evaluator is patched or upgraded. Clients consuming this
provider must key identity off `sub`.
- Scope mappings for this provider exist — `grafana-email-scope` and
`grafana-profile-scope` are both rows in `authentik_core_propertymapping`
(read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
the evaluator bug is fixed; no config change is needed then.
## LDAP provider objects (provisioned via `ak shell` ORM)
The LDAP outpost's own deployment — unit, image digest, published ports, token
env var — is in [[Authentik]]. The objects it serves:
Read from the live database (read-only `SELECT`s against the
`authentik-postgres` container, 2026-10-18):
| Object | Value |
|---|---|
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — an earlier revision's "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — an earlier revision recorded `intent=verification`; the live value is `app_password` |
| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (an earlier revision said slug `jellyfin`, which does not exist) |
Hard-won rules (the first is now contested by the live database):
- **Token intent: contested, do not trust either value blindly.** An earlier
revision asserted that a bind token with `intent=app_password` is rejected by
the flow's password stage with `Invalid password` and only
`intent=verification` passes. The live token for `svc-jellyfin-ldap` is
`app_password` (`authentik_core_token`, 2026-10-18), so either the rule is
wrong or the current Jellyfin bind path is not that token. Nothing observable
settles it today: the LDAP outpost's retained log holds only its 5-minute
`initialized direct binder` / `Update providers` config cycles and no bind
activity *(unverified 2026-10-18: no bind was performed — reproducing it
requires the bind secret)*. Re-verify with a real bind before changing the
token's intent either way.
- **Search needs `search_full_directory`.** Without the RBAC grant the
outpost returns only the bind user itself — a `(memberOf=…)` filter then
matches zero entries and the plugin logs `Found no users matching X`.
The permission is object-level on the provider; grant it via an RBAC role
and retest.
- `ak shell` import path for the provider model in 2026.8:
`from authentik.providers.ldap.models import LDAPProvider`.
- (unverified 2026-10-18: importing it outside a configured Django shell fails
with `ImproperlyConfigured`, and no `ak shell` was driven against the live
auth service for this check)*
## Self-serve user-settings flow (password change)
Users change their own account details through the `default-user-settings-flow`
(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the
target of the homepage footer's "account" link. It is a `stage_configuration`
designated flow, so it renders every bound `ConfigurableStage` for the logged-in
user.
Stages, in execution order (the `FlowStageBinding.order` field, not the M2M
insertion order):
| order | stage | type | what it does |
|---|---|---|---|
| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** |
| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user |
**Password change is a prompt field, not a stage.** The `UserWriteStage`
writes whatever the preceding `PromptStage` collected: for the key `password`
it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix
is a `password` field on the `default-user-settings` prompt
(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a
`PasswordStage`. A `PasswordStage` is an *authentication* stage: its
`validate_password` runs `authenticate()` against the backends to check the
*current* password; it never sets a new one. (A `PasswordStage` was added on
2026-09-25 and removed the same day for exactly this reason.)
Two ORM gotchas: `Flow.stages.add()` fails on this through model
(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
MTI), so create the `FlowStageBinding` row directly with an explicit `order`
and a prompt field is a `Prompt` row linked through `PromptStageField`
(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).
**A blank password must not set an empty password.** `set_password("")` would
make the account log in with no password, so a validation policy
(`default-user-settings-no-empty-password`, bound to the prompt
`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
is `""` and allows it when set. Verified with the `PolicyEvaluator`
(empty → deny, set → allow, absent → allow) and end-to-end by driving
`UserWriteStageView.update_user` on a throwaway user (new password valid, old
invalid).
## Live database shape
For live DB truth prefer the postgres container, and it works without any
credential — the local socket is trusted, so a read-only audit is just
`podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
before guessing: 2026.x renamed both tables and columns, and the models are
MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
three `*_flow_id` columns; the proxy table is
`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
table in 2026.8), the LDAP table is
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`
`authentik_core_application` attaches to a provider through `provider_id`
(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
is the same story from the ORM side: `type` lives in the subclass tables.
Never select secret columns (`client_secret`, `key`, `password`) into a log.
## Cross-references
- [[Authentik]] — what the service is, its deployment, wiring, operations and
failure modes
- [[Authentik Processes]] — the postgres, server and worker containers
- [[Caddy]] — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [[Jellyfin]] — consumes the LDAP outpost via its LDAP-Auth plugin
- [[Grafana]] — guarded vhost *and* OAuth2 relying party of this provider
serverAppConfigs-docs/Infrastructure/Authentik Processes.md
+76
View File
@@ -0,0 +1,76 @@
---
type: reference
description: "The three core Authentik containers — database, server and worker — with their units, images, roles and probes."
tags:
- infrastructure
- authentik
- postgres
timestamp: 2026-10-19
---
# Authentik Processes
The three core containers of the [[Authentik]] stack on Dankclaw. The two
outpost units are not covered here — they live on [[Authentik]] and
[[Authentik Outpost Reference]]. Server and worker run the same image; the
`command` selects the role (`server` / `worker`). Data persists under
`/home/dank/Containers/authentik/` as bind mounts (repo policy — no named
volumes), moved from `/home/dank/authentik/` in the 2026-09-01 quadlet
standardization; the retired stubs this page replaces recorded the pre-move
paths `/home/dank/authentik/data` (server and worker) and
`/home/dank/authentik/postgres` (database).
## authentik-postgres
The PostgreSQL 16 database for Authentik — the only stateful store in the stack
(users, flows, sessions, events). Sessions, caching, and WebSockets moved to
PostgreSQL in 2025.10 (Redis removed).
- **Image:** `docker.io/library/postgres:16-alpine`
- **Data:** `/home/dank/Containers/authentik/postgres` (bind mount; no named
volumes)
- **Health:** `pg_isready -d authentik -U authentik`
- **Retired wording:** the stub called this a "Compose service … in
`dankclaw/compose.yaml`"; the compose stack is gone and the unit is a quadlet.
- The side-by-side with `gitea-postgres` (network, published ports, secret
source, boot persistence) is owned by [[Postgres]].
## authentik-server
The Authentik server — the core, the admin API, and the embedded Caddy outpost.
It also serves the login page at `https://auth.franksdank.cloud`.
- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container`
(`authentik-server.service`)
- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (`command: server`)
- **Port:** `9000` (published) — the outpost, the admin API and the login UI all
live here
- **Data:** `/home/dank/Containers/authentik/data` (bind mount, shared with the
worker)
- **Health:** recorded by the retired stub as
`http://localhost:9000/-/health/`; superseded — [[Authentik]] → Operations
owns the live probe.
- **Retired wording:** the stub described "the Caddy `forward_auth` vhosts"
proxying here; the chain Caddy actually runs is [[Authentik]] → Caddy
integration.
## authentik-worker
The Authentik worker — background tasks: event processing, provider
synchronization, scheduled flows, and certificate renewal.
- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container`
(`authentik-worker.service`)
- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (`command: worker`)
- **Runs as:** `0:0` (rootless: container root maps to the host user that owns
the data dir)
- **Data:** `/home/dank/Containers/authentik/data` (bind mount, shared with the
server)
- **No port** — the worker has no HTTP endpoint.
## Cross-references
- [[Authentik]] — what the service is, its deployment, wiring, operations and
failure modes
- [[Authentik Outpost Reference]] — provider, flow and token configuration
- [[Postgres]] — both PostgreSQL clusters on Dankclaw, side by side
serverAppConfigs-docs/Infrastructure/Authentik.md
+46 -235
View File
@@ -15,26 +15,33 @@ application-level authentication with its proxy outpost. Not wrapped: Gitea,
OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
they handle their own session. Grafana is the exception that is both: it is a
guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
`dankclaw/Caddyfile:374-377`; live provider table below).
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`
`dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]).
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`
2026-10-18)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack: five quadlet units, no compose.** `authentik-postgres`,
`authentik-server`, `authentik-worker` (`deploy_authentik`,
- **Stack: five quadlet units, no compose.** `authentik-postgres`
`authentik-server`, `authentik-worker` (`deploy_authentik`
`scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`
`deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`
`scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
`active`, `Restart=always`, `WantedBy=default.target`, with linger on
(`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
pre-quadlet era, retired.
- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.
**Split off from this page:** [[Authentik Processes]] owns the per-container
facts for `authentik-postgres`, `authentik-server` and `authentik-worker`
[[Authentik Outpost Reference]] owns the provider/flow/token lookup material —
the live proxy-provider inventory, the proxy-provider provisioning rules, the
OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow
design, and the live database shape.
## Why Authentik
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was
@@ -100,23 +107,10 @@ contract:
The chain terminates there — nothing dials back into Authentik from
`:8017`.
## The three compose services
## The stack's processes
| Service | Role | Image |
|---|---|---|
| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.3` |
| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.3` |
- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
moved to PostgreSQL in 2025.10 (Redis removed).
- **Server** and **worker** run the same image; the `command` selects the role
(`server` / `worker`).
- The worker runs as `0:0` (rootless: container root maps to the host user that
owns the data dir).
- Data persists under `/home/dank/Containers/authentik/{data,postgres}`
(bind mounts, repo policy — no named volumes; moved from `/home/dank/authentik/`
in the 2026-09-01 quadlet standardization).
Per-container facts — unit path, image and role, port, run-as, data directory,
health probe — are owned by [[Authentik Processes]].
## Caddy integration
@@ -142,110 +136,15 @@ routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.
Live provider inventory (read from the `authentik-postgres` container with
read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
members in `authentik_outposts_outpost_providers`), all with
`internal_host = http://caddy:8017`, all carrying the **same three flow
UUIDs** (`default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
`default-provider-invalidation-flow`), and each backed by an Application
(the `slug` column below):
| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |
| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |
That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
the lockstep rule holds today. Two corrections against the 2026-09-02 table:
**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
not proxy providers, so the proxy outpost does not deserialize them: pk 9
`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
below rejects), and pk 14 `Open WebUI`
(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
`external_host` (the runbook-recipe style) and bare origins both intercept
correctly — the outpost matches on hostname.
Two provisioning rules, each verified live by an outage:
1. **All three flows are required.** A ProxyProvider created via raw API/ORM
with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
null wedges the embedded outpost's config fetch: the Rust side rejects the
whole config JSON (serde: `invalid type: null, expected a formatted UUID
string`), logs `Failed to fetch outpost from API, retrying in 3 seconds`
forever, and **everything behind the outpost dies at the next outpost
reload** — including hosts whose providers were fine (verified 2026-09-02
when three providers provisioned without any flows took down every guarded
vhost after a server restart). Backfill all three flows (copy from pk 1:
`default-authentication-flow`,
`default-provider-authorization-implicit-consent` (the implicit-consent
authorization flow, `default-invalidation-flow`) and the retry loop
recovers on its next 3-second cycle. The silent killer: while the outpost
process keeps its last good in-memory config, everything works — the
outage only surfaces on restart/reload.
2. **Every proxy provider needs an attached Application.** The outpost
resolves the post-login redirect through the provider's application; a
provider with no app completes the login (authentik session established)
but bounces to authentik's own user UI (`/if/user/#/library`) instead of
the app. No error is logged anywhere — the SSO "works" and the app never
appears. Create the Application with `provider_id` set (the UI does this
automatically; raw API/ORM drives must do it explicitly).
## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
implementation diverges from upstream in ways that bit us; all verified live:
- **Application row required.** `OAuth2Provider.objects.create()` does not
create the single-table-inheritance `Application` row; without it the flow
executor can never complete an authorize request. Create both rows.
- **`grant_types` defaults to empty.** Until the provider's `grant_types`
contains `"authorization_code"` (enum value with underscore), the token
endpoint rejects every exchange with `invalid_request`.
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
`parse_token_request` see no `grant_type` → `unsupported_grant_type`;
POST-body client secrets are not accepted — authenticate the confidential
client with `Authorization: Basic base64(client_id:client_secret)`.
- **Provider resolution is by `client_id` query param** on the global
`/application/o/authorize/` route; per-slug authorize paths do not exist in
this build. Token/userinfo live at `/application/o/token/` and
`/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
`PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
`BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
becomes the *expression source* instead of the context; every call raises a
swallowed `PropertyMappingExpressionException` and the claim is skipped.
Scope-grant filtering has the same class of bug: `ScopeMapping` has no
provider FK (linkage lives in the `ProviderPropertyMapping` through model),
so `filter(provider=...)` always yields an empty queryset and tokens asked
to carry scopes get zero. Net effect: userinfo/introspection answers with
`{sub}` only — email/groups/entitlements claims cannot be delivered by this
build until the evaluator is patched or upgraded. Clients consuming this
provider must key identity off `sub`.
- Scope mappings for this provider exist — `grafana-email-scope` and
`grafana-profile-scope` are both rows in `authentik_core_propertymapping`
(read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
the evaluator bug is fixed; no config change is needed then.
The live proxy-provider inventory (the per-vhost `external_host` → app mapping)
the two provisioning rules the outpost enforces, and this fork's OAuth2
relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
## LDAP outpost (Jellyfin auth, live 2026-09-01)
The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
single-platform digest**
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
@@ -260,51 +159,16 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
.digest'` and pin that digest.
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
(`PublishPort=389:3389`, `636:6636`,
(`PublishPort=389:3389`, `636:6636`
`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
(2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN`
from the host `dankclaw/.env` (git-ignored).
### Authentik-side objects (provisioned via `ak shell` ORM)
Read from the live database (read-only `SELECT`s against the
`authentik-postgres` container, 2026-10-18):
| Object | Value |
|---|---|
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — the page previously recorded `intent=verification`; the live value is `app_password` |
| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (the page previously said slug `jellyfin`, which does not exist) |
Hard-won rules (the first is now contested by the live database):
- **Token intent: contested, do not trust either value blindly.** This page
long asserted that a bind token with `intent=app_password` is rejected by the
flow's password stage with `Invalid password` and only `intent=verification`
passes. The live token for `svc-jellyfin-ldap` is `app_password`
(`authentik_core_token`, 2026-10-18), so either the rule is wrong or the
current Jellyfin bind path is not that token. Nothing observable settles it
today: the LDAP outpost's retained log holds only its 5-minute
`initialized direct binder` / `Update providers` config cycles and no bind
activity *(unverified 2026-10-18: no bind was performed — reproducing it
requires the bind secret)*. Re-verify with a real bind before changing the
token's intent either way.
- **Search needs `search_full_directory`.** Without the RBAC grant the
outpost returns only the bind user itself — a `(memberOf=…)` filter then
matches zero entries and the plugin logs `Found no users matching X`.
The permission is object-level on the provider; grant it via an RBAC role
and retest.
- `ak shell` import path for the provider model in 2026.8:
`from authentik.providers.ldap.models import LDAPProvider`.
*(unverified 2026-10-18: importing it outside a configured Django shell fails
with `ImproperlyConfigured`, and no `ak shell` was driven against the live
auth service for this check)*
The objects behind it — LDAP provider, bind DN, service account and bind token,
RBAC grant, groups, application slug — and the hard-won bind rules are in
[[Authentik Outpost Reference]].
## Configuration (secrets)
@@ -332,7 +196,7 @@ every outpost session (see the ORM playbook).
core quadlets, starts postgres health-gated (`wait_container_healthy
authentik-postgres 120`), then the server (300 s health wait) and the worker.
The outposts are separate handlers: `--service authentik-outpost` and
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`
dispatch at `:1579-1588`).
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
creates the superuser (email + password) and generates the DB schema.
@@ -340,12 +204,13 @@ every outpost session (see the ORM playbook).
Applications** → add an application + a **Proxy** provider whose
`external_host` is exactly the browser-facing origin
(`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
flows set (see the provisioning pitfall above). Attach the provider to the
flows set (see the provisioning rules in
[[Authentik Outpost Reference]]). Attach the provider to the
embedded outpost.
4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
the app with the correct redirect URI, and paste the `client_id` /
`client_secret` into the app's SSO settings. The relying parties that exist
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14)
**Gitea has no provider row at all** (live provider table, 2026-10-18) — it
authenticates itself and stays off the chain by design
(`dankclaw/Caddyfile:110-120`).
@@ -373,53 +238,12 @@ every outpost session (see the ORM playbook).
the LDAP section) — bumping only the tag would leave the LDAP outpost behind.
Authentik migrates the schema on startup.
## Self-serve user-settings flow (password change)
Users change their own account details through the `default-user-settings-flow`
(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the
target of the homepage footer's "account" link. It is a `stage_configuration`
designated flow, so it renders every bound `ConfigurableStage` for the logged-in
user.
Stages, in execution order (the `FlowStageBinding.order` field, not the M2M
insertion order):
| order | stage | type | what it does |
|---|---|---|---|
| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** |
| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user |
**Password change is a prompt field, not a stage.** The `UserWriteStage`
writes whatever the preceding `PromptStage` collected: for the key `password`
it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix
is a `password` field on the `default-user-settings` prompt
(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a
`PasswordStage`. A `PasswordStage` is an *authentication* stage: its
`validate_password` runs `authenticate()` against the backends to check the
*current* password; it never sets a new one. (A `PasswordStage` was added on
2026-09-25 and removed the same day for exactly this reason.)
Two ORM gotchas: `Flow.stages.add()` fails on this through model
(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
MTI), so create the `FlowStageBinding` row directly with an explicit `order`;
and a prompt field is a `Prompt` row linked through `PromptStageField`
(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).
**A blank password must not set an empty password.** `set_password("")` would
make the account log in with no password, so a validation policy
(`default-user-settings-no-empty-password`, bound to the prompt via
`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
is `""` and allows it when set. Verified with the `PolicyEvaluator`
(empty → deny, set → allow, absent → allow) and end-to-end by driving
`UserWriteStageView.update_user` on a throwaway user (new password valid, old
invalid).
## Outpost claim cache (`proxysession`) — 403 after a group add
The embedded outpost does not re-evaluate group membership per request. Each
completed authorize writes one row to `authentik_providers_proxy_proxysession`
whose `session_data` JSON holds the claims (`groups` included) that get served
for that browser session until the row expires (observed TTLs 1–24 h per
for that browser session until the row expires (observed TTLs 1–24 h
row). A session authorized **before** a user was added to a group keeps
serving the old group list — Caddy-side RBAC header gates then 403 despite
correct membership, and no authentik event is generated (the 403 happens at
@@ -457,7 +281,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
(the inventory table carries both bare origins and trailing slashes;
routing matches on hostname, verified 2026-09-02).
`internal_host = http://caddy:8017`, all three flows set, and an
Application attached — the two provisioning rules above.
Application attached — the two provisioning rules in [[Authentik Outpost Reference]].
2. **Route flip in ONE commit/deploy**: the vhost becomes
`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
inside the `http://:8017` listener add `@<svc> host <vhost>` +
@@ -485,7 +309,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
000 (connection refused) — the post-auth listener lives in the caddy
container's netns; probe inside it or grep the config. Non-browser clients
cannot solve the JS challenge — before flipping, audit service-to-service
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
traffic: ingestion/push paths must target internal addresses (`loki:3100`)
never a public hostname of a guarded service. Exempt-by-construction
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
endpoints, services with their own complete auth — document the exemption
@@ -512,31 +336,18 @@ EOF
`sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
the exact banner text and the 229-object count — no `ak shell` was driven
against the live auth service for this audit)*.
- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
- Import paths (each wrong guess costs a failed exec): `User`, `Group`
`Application`, `ApplicationEntitlement`, `PropertyMapping` →
`authentik.core.models`; `ProxyProvider` →
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
`LDAPProvider` → `authentik.providers.ldap.models`;
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`)
`LDAPProvider` → `authentik.providers.ldap.models`
`authentik.core.models.application` does NOT exist as an import path; there
is no `authentik.policies.group` module in this build.
*(unverified 2026-10-18: these module paths were not re-executed — importing
- (unverified 2026-10-18: these module paths were not re-executed — importing
outside a configured Django shell fails with `ImproperlyConfigured`)*
- For live DB truth prefer the postgres container, and it works without any
credential — the local socket is trusted, so a read-only audit is just
`podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
before guessing: 2026.x renamed both tables and columns, and the models are
MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
three `*_flow_id` columns; the proxy table is
`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
table in 2026.8), the LDAP table is
`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
`authentik_core_application` attaches to a provider through `provider_id`
(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
is the same story from the ORM side: `type` lives in the subclass tables.
Never select secret columns (`client_secret`, `key`, `password`) into a log.
- The live database's table and column names (the 2026.x renames, the MTI parent
keys, the read-only `psql` audit recipe) are reference lookup:
[[Authentik Outpost Reference]].
## User provisioning playbook (user-level account)
@@ -565,7 +376,7 @@ downstream —
`category: user` panels from `services.json`.
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
**"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
`Open WebUI` application (`authentik_core_applicationentitlement`,
`Open WebUI` application (`authentik_core_applicationentitlement`
2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
expression were not read; the mapping itself is not visible under that name
@@ -581,7 +392,7 @@ downstream —
artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
and every one carries the `<user>@franksdank.cloud` address the template
prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
*(unverified 2026-10-18)*. Template: `username` +
- (unverified 2026-10-18)*. Template: `username` +
`<user>@franksdank.cloud` + display name + `is_active=True` + the password
set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
`openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
@@ -589,7 +400,7 @@ downstream —
the row carried no MFA device, token, entitlement, or proxy session, and
events survive a user delete because `Event.user` is a denormalized JSON
field, not a FK).
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
Plex, Seerr, Marreta) need no per-user group: any authenticated user
@@ -618,8 +429,8 @@ More field gotchas (each cost a failed exec): `ApplicationEntitlement.applicatio
can be null (guard with `getattr`); `LDAPProvider` has
`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
`organization` — the backing table
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`
`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
(scopes are linked through `ProviderPropertyMapping`, which is why
@@ -673,5 +484,5 @@ podman run --rm --network container:<container> --entrypoint /bin/busybox \
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison
*(the file is gone from the repo and from the host checkout as of
- (the file is gone from the repo and from the host checkout as of
2026-10-18)*
serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md
+5 -5
View File
@@ -47,7 +47,7 @@ torrent.
WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s
The private key is hashed into the VM env file and never enters the repo.
NAT-PMP inbound forwarding is available on Proton (gluetun supports it via
NAT-PMP inbound forwarding is available on Proton (gluetun supports it
`FIREWALL_INPUT_PORTS` + `NATPMP=true`) — deliberately deferred: the e2e
gate passes outbound-only, and swarm peers needing an inbound port is a
seeding-era concern, not a grab-only one.
@@ -86,7 +86,7 @@ are unaffected.
got `Unauthenticated`); the live conf was patched and
`scripts/deploy-remote.sh` fixed to write the `_PBKDF2` key. Gotcha
(live, 2026-09-01): after qB 5.x ran, its conf rewrite left a hash line
4.6.7 rejects (`invalid credentials` for every login, including the arrs);
4.6.7 rejects (`invalid credentials` for every login, including the arrs)
rewriting the same `@ByteArray` line fresh (same password, 100k
iterations) restored login. Any future version flip: regenerate the line.
- `HostHeaderValidation=false` + `ServerDomains=*`: the socat forwarder
@@ -98,7 +98,7 @@ are unaffected.
path (no auto-TMM), so the default itself points at `Finished Torrents`:
every grab lands there regardless of client behavior; human review moves
files onward; nothing writes to the entropy volume.
- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`,
- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`
`tv-sonarr` (all → `Finished Torrents`).
- **Download clients** (gateway `10.88.0.1:8080`, user `admin`): all five test
green against live qB — sonarr (`tv-sonarr`), radarr, lidarr, readarr
@@ -109,12 +109,12 @@ are unaffected.
as "Authentication failure" — that kills runtime auth too, not just API
creates; the fix was pinning qB itself (below), after which readarr tests
200 like the rest.
- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`,
- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`
not `latest`): `latest` resolved to 5.x, whose `204` login breaks
Readarr's frozen client; 4.6.7 answers the legacy `200` + `"Ok."` that
every client accepts. Pinned in
`frankbook-pro/qbittorrent/quadlet/qbittorrent.container`.
- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`,
- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`
gateway ports). Readarr deliberately omitted: prowlarr's indexer-push
test to readarr's `/api/v1/indexer/test` 500s (EOL schema drift).
Legacy `Prowlarr-*` indexer rows carried in the migrated dankserver DBs
serverAppConfigs-docs/Infrastructure/Grafana.md
+4 -4
View File
@@ -46,7 +46,7 @@ The Prometheus datasource is declared in git, not clicked into `grafana.db`:
address Caddy proxies to, never the loopback-published port — and carries
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
alongside it by `dashboards/provider.yml` (provider `fleet`,
alongside it by `dashboards/provider.yml` (provider `fleet`
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
A Loki datasource is still **not** provisioned — the tree contains exactly one
@@ -69,8 +69,8 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
Basic client auth, form-encoded token body, and the fork's userinfo-claims
bug that leaves identity at the `sub` hash only) is in
[[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana:
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
users land as Viewer with `login = <sub hex>`; no email/groups/role claims
reach Grafana until the evaluator bug is fixed — do not write role mappings
@@ -79,7 +79,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
the host without the database or an admin password. *(unverified 2026-10-18)*)
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
for the service, podman then looks for its configuration under `/root`,
for the service, podman then looks for its configuration under `/root`
cannot read it, and the unit dies at start with
`stat /root/.config: permission denied`. The unit reads
`/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md
+3 -3
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Homepage Status Probes
description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
description: "How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions."
tags: [homepage, status, probes, e2e, homepage-status]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -51,7 +51,7 @@ See [[Container Host Reachability]] for the netns rules.
published nothing → structurally unprobeable from the host).
3. Probe from the **correct vantage**: from a bridge-net container the host
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`)
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
they expand **inside** `podman exec bash -c`.
@@ -83,7 +83,7 @@ python3 -m http.server 8899 --bind 127.0.0.1
```
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
(panels render after the async fetch; checking immediately returns 0 panels),
(panels render after the async fetch; checking immediately returns 0 panels)
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
serverAppConfigs-docs/Infrastructure/Homepage.md
+10 -10
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation."
description: "The password-protected service list on Dankclaw: what each card shows and where its data comes from."
tags:
- infrastructure
- homepage
@@ -22,7 +22,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
**Docs page shipped (2026-09-17):** `/docs` serves a static user documentation
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`,
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`
`frank-zealot`) moved from admin-only to the **user** page with live model
descriptions and a `docs` link to `/docs#models`.
@@ -59,7 +59,7 @@ without which the picker showed "No results found" for every family account
"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra
(same era)", frank-zealot "equivalent to last year's frontier level
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic)
pinned in `homepage_config.bats`.
**Protoss rename + Model Information component (2026-10-08):** the model
@@ -91,7 +91,7 @@ same renames, api-id mapping, supervision and compaction notes.
app card now carries hex-clipped art beside its name (`services.json` `art`
field, files under `img/apps/`): 19 brand marks tinted amber from the
Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard)
node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
cards render in the order zealot → dragoon → archon (registry order is the
@@ -143,8 +143,8 @@ per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`)
`warn` on OpenWebUI (searches ride the Seerr→SearXNG path)
`model.compaction` (the amber "set this as compaction model" callout
frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
@@ -175,7 +175,7 @@ titles are chamfered plates with the diamond glyph inside the plate.
Fonts: body is **Share Tech Mono** (legible retro terminal), headings
are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
three pages load the pair from Google Fonts. Four tiles that had
reverted to initials got pictograms: Calibre-Web an open book (orange),
reverted to initials got pictograms: Calibre-Web an open book (orange)
RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
@@ -197,7 +197,7 @@ state — the only animation left is the speaking-voice pulse. The result
is flat navy panels on crisp 1px borders: same Protoss direction, half
the pixels.
**Model ids official (2026-10-08):** the card ids `frank-zealot`,
**Model ids official (2026-10-08):** the card ids `frank-zealot`
`frank-dragoon`, `frank-archon` are now the harness's official model ids
in `models.yml`/`config.yml` (see [[OMP Harness]]); the
systemd services keep the `frank-archon`/`frank-dragoon`/
@@ -205,7 +205,7 @@ systemd services keep the `frank-archon`/`frank-dragoon`/
one name per model.
**Api ids renamed on the wire (2026-10-09):** the promise in
`docs.html` is kept — the served model ids are now `frank-zealot`,
`docs.html` is kept — the served model ids are now `frank-zealot`
`frank-dragoon`, `frank-archon`, so the docs page no longer carries
`api id:` annotations, and the curl/opencode/Pi examples send
`frank-zealot`. The systemd services keep the old names. `docs.html`
@@ -216,7 +216,7 @@ ids, which did not change.
clip-paths replaced with `border-radius` (the clipped corners read as
gaps in the 1px border); section-title diamonds moved from absolute
positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
WIP items grey out (dim type, desaturated art, muted border via
WIP items grey out (dim type, desaturated art, muted border
`:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
(`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
beat; the two SC2 source oggs are trashed); privacy notes became
serverAppConfigs-docs/Infrastructure/LAN-DNS.md
+2 -2
View File
@@ -188,7 +188,7 @@ sudo systemctl restart systemd-resolved && resolvectl flush-caches
```
The `~` (route-only) prefix is load-bearing: it routes only those domains to
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`
`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
@@ -236,7 +236,7 @@ Endpoints (all POST, JSON body):
`.content.records[]` as some docs claim — parse with
`jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`
`type`, `content`, `ttl`, `prio`).
Quirks that bite (verified live 2026-08-31):
serverAppConfigs-docs/Infrastructure/Loki.md
+1 -1
View File
@@ -55,7 +55,7 @@ The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `
The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`
`chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
`find /opt/loki/data -type f` counted **8 files**, all WAL segments,
cluster-seed and delete-request bookkeeping, **no chunk objects**
serverAppConfigs-docs/Infrastructure/Observability Provisioning.md
+9 -9
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Observability Provisioning
description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
tags: [observability, prometheus, grafana, provisioning, deployment]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -21,7 +21,7 @@ committed.
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
Targets and datasources reference **container names** (`grafana:3000`,
Targets and datasources reference **container names** (`grafana:3000`
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
loopback-published ports; `alloy` is **not scraped at all** (no job in the
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
@@ -47,7 +47,7 @@ from `deploy_observability`:
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
`shared/grafana/**` to `grafana` (`:261-264`); changes under
`dankclaw/observability/` select the touched unit and its dependency consumer
@@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint.
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
Current delivered set: **4 alert rules + 2 recording rules** —
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`)
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
@@ -101,7 +101,7 @@ curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.to
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and `up` for
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
listed in [[Prometheus]]. The Grafana query needs the local admin password and
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
@@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect).
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
via `host.containers.internal`) → signal-cli, addressed by number
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
`curl -s http://192.168.0.143:18093/health` → `ok`, and
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
@@ -148,7 +148,7 @@ state lives in Prometheus' process, not on disk — a restart re-stamps
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
`--storage.path=/alertmanager` on a host volume keeps silences and the
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
`/opt/alertmanager/data`.)
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
serverAppConfigs-docs/Infrastructure/Observability.md
+8 -8
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "Observability on Dankclaw"
description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured."
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-09-11
---
@@ -51,13 +51,13 @@ Dankserver runs one member of this family, node-exporter, as its own unit
## Exposure
The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
inside, and several carry lifecycle or query APIs. Browser access goes through
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
`anubis:8923`, and the post-auth handlers route by container name on
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
@@ -70,13 +70,13 @@ and the podman bridge subnet `10.89.0.0/16` before the DROP
## Deployment
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
which installs the unit, copies that service's config
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
`copy_alloy_config`, `copy_grafana_provisioning`, plus
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
`grafana` after `prometheus`, `alloy` after `loki`
(`grafana.container:32-33`, `alloy.container:38-39`). See
serverAppConfigs-docs/Infrastructure/Prometheus.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
type: reference
description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the provisioned Grafana datasource. Down targets today are structural, not outages."
description: "Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural."
tags:
- infrastructure
- prometheus
serverAppConfigs-docs/Infrastructure/authentik-postgres.md
-24
View File
@@ -1,24 +0,0 @@
---
type: reference
description: "The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /"
tags:
- infrastructure
- authentik
- postgres
timestamp: 2026-08-28
---
# authentik-postgres
The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /
OIDC provider. It is the only stateful store in the stack (users, flows,
sessions, events). Redis was removed in Authentik 2025.10; sessions and caching
now live here.
- **Compose service:** `authentik-postgres` (in `dankclaw/compose.yaml`)
- **Image:** `docker.io/library/postgres:16-alpine`
- **Data:** `/home/dank/authentik/postgres` (bind mount; no named volumes)
- **Health:** `pg_isready -d authentik -U authentik`
See [Authentik.md](Authentik.md) for the full architecture, setup, backups
(`pg_dump`), and failure modes.
serverAppConfigs-docs/Infrastructure/authentik-server.md
-24
View File
@@ -1,24 +0,0 @@
---
type: reference
description: "The Authentik server — the core, the admin API, and the embedded Caddy outpost."
tags:
- infrastructure
- authentik
timestamp: 2026-09-16
---
# authentik-server
The Authentik server — the core, the admin API, and the embedded Caddy outpost.
This is the service the Caddy `forward_auth` vhosts proxy to
(`authentik-server:9000`). It also serves the login page at
`https://auth.franksdank.cloud`.
- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` (`authentik-server.service`)
- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (command: `server`)
- **Port:** `9000` (published; the outpost + admin + login all live here)
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker)
- **Health:** `http://localhost:9000/-/health/`
See [Authentik.md](Authentik.md) for the full architecture, the Caddy
forward_auth pattern, first-time setup, and failure modes.
serverAppConfigs-docs/Infrastructure/authentik-worker.md
-23
View File
@@ -1,23 +0,0 @@
---
type: reference
description: "The Authentik worker — background tasks: event processing, provider"
tags:
- infrastructure
- authentik
timestamp: 2026-09-16
---
# authentik-worker
The Authentik worker — background tasks: event processing, provider
synchronization, scheduled flows, and certificate renewal. It runs the same
image as the server; the `command: worker` selects the role.
- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` (`authentik-worker.service`)
- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (command: `worker`)
- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir)
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server)
- **No port** — the worker has no HTTP endpoint.
See [Authentik.md](Authentik.md) for the full architecture, setup, and failure
modes.
serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md
+9 -13
View File
@@ -100,7 +100,7 @@ The DFlash drafter proposes, then ngram-mod runs as an additional implementation
`--spec-type` takes a comma list, and the gate is `ctx_dft` in
`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`
`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
- A chain may hold ONE context-backed drafter plus any number of ngram types.
@@ -201,7 +201,7 @@ Speculative decoding is lossless under rejection sampling: the target verifies e
On the mid-August 2026 Dankclaw shape `draft-dflash` was the benchmarked winner
at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those
three figures are that retired llama.cpp era's property ([[frank-dragoon]] owns
the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see ).
## Draft Models
@@ -257,7 +257,7 @@ DSpark took `ROCm error: an illegal memory access` inside the shared draft
context with two concurrent 16k prompts; DFlash corrupted multi-slot into
premature EOS and was retired first. The timeline, the crash class, the
"concurrency is the trigger, not size" finding and the `PARALLEL=1` workaround
are on [[qwen3.8-spec-matrix]]; the shape they were measured on is
are on ; the shape they were measured on is
[[frank-dragoon]]'s mid-August 2026 era.
## A/B protocol on a live llama.cpp endpoint
@@ -303,7 +303,7 @@ Protocol:
Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
`predicted_per_second`; cumulative `/metrics`
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`
per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
while tokens flow = the proposer is NOT engaged, whatever the flags say. A
zero-draft config is a useful negative control: it proves the other config's
@@ -367,7 +367,7 @@ states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
**Step 4 — acceptance reality checks.** Treat 40–60 % as the honest band for a
matched pair; the arm that anchors the bottom of it on this fleet is the
depth-7 `draft-dflash` row of [[frank-medium-llamacpp-dflash2]]'s spec table
depth-7 `draft-dflash` row of 's spec table
(accepted share, rate and VRAM for every depth are in that table). Cheapest
proxy for `p` at temp 0 when both models are greedy and share a tokenizer:
greedy continuation agreement on identical prompts, two API calls, no weights
@@ -381,10 +381,10 @@ drafting.
**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
engines refuse external drafters by design: halogen-flash-server's
`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
name is a deliberate 400 (a silent downgrade would make comparisons lie);
name is a deliberate 400 (a silent downgrade would make comparisons lie)
there is no device routing in its loopback token protocol, so a drafter can
never sit on another card. Pure llama.cpp DOES route the draft:
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`
`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
One process sees only the devices its backends were built with — CUDA+Vulkan
is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
@@ -411,20 +411,16 @@ measurement that would settle it.
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
- The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by [[RAM CPU Offloading in VLLM]].
-The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by .
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
- Use `rejection_sample_method` default `strict` for lossless output.
## See Also
- [[Quantization by Hardware Architecture]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[deployment]]
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- where the drafter's KV actually lands
- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle
serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+7 -9
View File
@@ -30,8 +30,8 @@ taken in.
- **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm
wins prefill and Vulkan wins decode, and the decode gap widens with context:
the 64k A/B is [[qwen3.8-strix-optimizations]] §4, the 262k A/B is
[[qwen3.8-spec-matrix]] §High-Context Backend Comparison. The "HIP
the 64k A/B is §4, the 262k A/B is
§High-Context Backend Comparison. The "HIP
out-performs Vulkan for decode" claim did not hold on the 8060S.
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
@@ -39,7 +39,7 @@ taken in.
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What
`frank-zealot` runs now is its own Current shape — this row is not it.
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
(ngram-mod) with draft-mtp first — timeline on [[qwen3.8-spec-matrix]]; the
(ngram-mod) with draft-mtp first — timeline on ; the
spec setting each endpoint runs today is on its own page.
- **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded
once on [[Dankclaw System Specs]] (Memory).
@@ -54,8 +54,8 @@ taken in.
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT; the 76.2 GiB row is an
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`,
recorded in §Why Vulkan + GTT; the 76.2 GiB row is an
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`
the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated
carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read
2026-10-18.
@@ -125,9 +125,9 @@ All require root and a reboot for the modprobe change.
## Sources
- [[frank-dragoon]] — measured backend/spec numbers
- [[deployment]] — deploy and update process
- deploy and update process
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`
`common/arg.cpp`, `common/common.h`
- NotebookCheck Ryzen AI MAX+ 395 analysis
@@ -135,5 +135,3 @@ All require root and a reboot for the modprobe change.
- [[frank-dragoon]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
- [[index]]
serverAppConfigs-docs/LLM/Dankclaw System Specs.md
+4 -6
View File
@@ -32,7 +32,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
| Device ID | 0x1586 |
| Architecture | gfx1151 (Strix Halo) |
| GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `mem_info_gtt_total` = 122,402,963,456 B read 2026-10-18; rocm-smi's smaller SVM carve-out is not the budget) |
| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ([[ling-3.0-flash-trial]] §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ( §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
### NPU
@@ -44,9 +44,9 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
| Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) |
| Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) |
| Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] |
| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See |
The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot]], 2026-09-17 → 10-01).
The NPU endpoint is retired (2026-09-11; ). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see ); the Gemma4-12B QAT rung it once pointed at is retired history (, 2026-09-17 → 10-01).
### ROCm
@@ -80,7 +80,7 @@ The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carried the RPC path to Dankserver during the 2026-08-17 pipeline-parallelism trial; no RPC listener runs today (`ss -ltn` shows no :50052, 2026-10-18). Default route via 192.168.0.1. |
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. |
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See section 0. |
| docker0 | bridge | - | 172.17.0.1 | Usually down. |
The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at
@@ -112,5 +112,3 @@ cable between claw and server is required for 10G.
- [[frank-dragoon]]
- [[Shared Memory Architecture Dankclaw]]
- [[frank-redo-2026-08-17]]
- [[FastFlowLM NPU Inference]]
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+10 -11
View File
@@ -1,7 +1,7 @@
---
type: configuration
type: reference
title: Dankserver System Specs
description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium.
description: "Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture."
tags:
- dankserver
- hardware
@@ -30,7 +30,7 @@ read 2026-10-18).
| Property | Value |
|----------|-------|
| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see ) |
| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3` (both bridges: `current_link_width=8`, `current_link_speed=32.0 GT/s`, read 2026-10-18); ~31.5 GB/s per card. Each card sits behind an on-board PCIe switch (`01:00.0`/`02:00.0`, `05:00.0`/`06:00.0` — "Navi 10 XL Upstream/Downstream Port of PCI Express Switch"), so the GPU endpoint itself trains **x16** Gen5 while the CPU-facing leg stays x8 |
| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board (2026-09-20 census; not re-readable today — `lspci -vv` needs root) *(unverified 2026-10-18)* |
| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
@@ -55,7 +55,7 @@ read 2026-10-18).
| Architecture | gfx1201 (RDNA4) |
| VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
| Link | PCIe 5.0, endpoint-trained **x16** each behind the on-board switch; the CPU-facing leg is x8 (`00:01.1` / `00:01.3`, both `current_link_width=8` at 32.0 GT/s, read 2026-10-18). No XGMI. Separate IOMMU groups — `03:00.0` → group **15**, `07:00.0` → group **20** (`/sys/bus/pci/devices/.../iommu_group`, read 2026-10-18) |
| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see [[Dual-R9700-Cooperation]] |
| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see |
| PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2 (rocm-smi GPU[2], 512 MiB). The retired llama.cpp trios hid it with `HIP_VISIBLE_DEVICES=0,1`; the live radiance engine instead enumerates all three AQL devices and ignores it ("3 devices [0:gfx1201 1:gfx1201 2:gfx1036], using 2 of 3, ignoring gfx1036") |
@@ -104,8 +104,7 @@ RCCL-backed rebuild needs no new host packages.
| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
hijack `libhsa-runtime64` were purged on 2026-09-15 — see
[[frank-medium-rocmfpx]].
hijack `libhsa-runtime64` were purged on 2026-09-15 —
### Active Services
@@ -117,12 +116,12 @@ host-network podman container answering `curl 127.0.0.1:7770/health` →
The services this page once called active are gone: `frank-medium.service`
(llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung
ran 2026-09-16 → 09-20 and is documented in [[frank-medium-llamacpp-dflash2]];
ran 2026-09-16 → 09-20 and is documented in
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU
swap. Its archived unit was moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`
and the vllm-radiance replica trio to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`
— neither path exists today (checked 2026-10-18), so the archive location is
unverified. See [[deployment]].
unverified. See .
## Hardware Relevance
@@ -134,7 +133,7 @@ unverified. See [[deployment]].
tokens at 32:33, ~6.3/~5.2 GiB headroom) is history. Either way a batch or
context change needs a re-fit and not a guess.
- 246 GiB system RAM is **not** headroom: the live shape pins a 43.88 GiB host
pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`,
pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`
`scripts/frank-archon.env:184`) plus the 47.68 GiB n-gram/PLE table in RAM,
and `free -g` showed **90 GiB available** with the engine loaded (read
2026-10-18). The frank-medium claim that "nothing in the serving path is
@@ -147,6 +146,6 @@ unverified. See [[deployment]].
## See Also
- [[frank-archon]] — the active serving runbook
- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
- [[deployment]] — the deploy flow
- why the PCIe path matters, and what the swap changed
- the deploy flow
- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
-327
View File
@@ -1,327 +0,0 @@
---
type: research
title: Dual R9700 Cooperation
description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed.
tags:
- frank-medium
- rocm
- rccl
- pcie-atomics
- tensor-parallelism
- dankserver
timestamp: 2026-09-16
---
# Dual R9700 Cooperation
Dankserver's two R9700s run **one tensor-parallel engine across both cards** —
today that is `frank-archon` under radiance 1.0.13, TP=2
([[frank-archon]]). When this page was written the endpoint
was `frank-medium`: first llama.cpp's layer split
([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20
one vLLM-radiance process at `tensor_parallel_size 2`
([[frank-medium-radiance-mxfp4]], 2026-09-20 → 09-26). This page records why
true tensor parallelism was impossible on the previous board (TUF GAMING B650-E
WIFI), and what the 2026-09-20 board swap changed.
## The short answer
**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose
bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden
hostcall-buffer parameter and ROCr refuses to supply it to a device without
atomics, so *no* RCCL collective or P2P operation could be dispatched on that
card — see the [measured matrix](#measured-matrix) and the
[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below.
On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which
bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
recommended, at B850 rather than X870E money. Both cards now hang off CPU root
ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
the exact precondition this page's own test plan named, and it is now
[measured](#measured-verdict-2026-09-20) rather than assumed.
What that bought:
- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted
and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp
`--split-mode tensor` booted and served against an RCCL-backed build — so a
collective HAS been dispatched on this host since the swap.
- **TP=2 is the production shape.** It wins on every axis measured, by 2.1x on
decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day
of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is
retired 2026-09-26, and what runs today is the iq4r publisher under radiance
1.0.13 ([[frank-archon]]).
- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
could not carry the DFlash2 drafter, which is the honest caveat on the
comparison below.
- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
and M.2_2 is free.
## The gate, in source
```cpp
void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp:181
std::vector<amd::Device::LinkAttrType> link_attrs;
link_attrs.push_back(std::make_pair(LinkAttribute::kLinkAtomicSupport, 0));
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) {
if (link_attrs[0].second == 1) {
info_.pcie_atomics_ = true; // no else branch
}
}
}
```
The flag is a **queried hardware property**, consumed at
`rocvirtual.cpp:3954` (the hostcall/AQL dispatch path), in `rocblit.cpp` (SDMA
copy paths) and exported as `hipDeviceProp.hostNativeAtomicSupported`. There is
no environment variable, module parameter, or sysfs toggle around it — verified
by grepping ROCclr for `getenv`/`HSA_*`/`ROCR_*`/`AMD_*` in the gate's
neighborhood: none. Writing the endpoint's Device Control 2 bit 6 by hand
(`setpci -s 0b:00.0 CAP_EXP+28.w=0440`) does nothing because the value is
consumed at driver probe time.
What has to be true upstream of it: every PCIe bridge on the GPU's path to the
CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path
includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability
bit**, so no software can enable it.
### The same gate, as this board reports it (2026-09-16)
`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` →
`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 |
| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 |
| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` |
| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) |
| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 |
Three things fall out of that table, none of them in the source alone:
- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the
chipset, not the platform, which is why a slot on CPU lanes is the fix.
- **The endpoint's own bit differs between the cards, and that is the kernel's
doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses
to set Requester Enable when any bridge cannot route atomics. ROCclr's
`pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it
makes `lspci -vv -s <bdf> | grep AtomicOpsCtl` a one-command diagnosis: on
this host card 0 reads `ReqEn+` and card 1 `ReqEn−`.
- **Card 1's effective link is x1**, not the x16 its own capability block
advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a
bandwidth fact rather than an atomics one, and it only matters for
collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
x4. The layer split's boundary copy does not care.
## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO)
`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` →
`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only
LAN/SATA/WiFi:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 |
| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 |
| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 |
| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 |
| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 |
| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the
kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no
`Routing-` bridge. Same one-command diagnosis, opposite answer.
- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
- `rocm-smi --showtopo` reported `PCIE` between the cards (Navi 48 has no XGMI),
they were in separate IOMMU groups (15 and 20 — re-read 2026-10-18: still 15
and 20), and both root ports had ACS `ReqRedir+`, so this page inferred P2P
would hairpin at the root complex. **That inference is superseded:** the live
engine reports the two cards reaching each other's memory directly, which
[[frank-archon]] and [[Dankserver System Specs]] record. Plain P2P traffic
measured 45.27 GB/s bidirectional on the *old* board — DMA was never the gate,
atomics were.
- Kernel 7.0.0-31-generic with `iommu=pt` then; this host runs 7.0.0-38-generic
with `iommu=pt` today (`/proc/cmdline`, read 2026-10-18). Both were ruled out
as causes ([above](#ruled-out-empirically-do-not-re-derive)) and this host's
IOMMU posture is unchanged.
**Still unproven (as of 2026-09-20).** The gate that matters is ROCclr's
`checkAtomicSupport()` reading the `kLinkAtomicSupport` link attribute; the ReqEn
bit is the same cause seen from the kernel side, but no RCCL kernel had been
launched since the swap. Until the 2-rank probe runs — or a `--split-mode row`
boot that does *not* log `internal AllReduce init failed ... falling back to
meta-backend butterfly` — "collectives work here" was a strong expectation, not a
measurement. **RCCL never got that measurement**; what settled the operational
question instead is production traffic — radiance's TP=2 wire carries cross-rank
all-reduces over the direct P2P path and prints its own precision, and the
measured `wht6` banner with its exact-step bound is on [[frank-archon]].
## Measured matrix
Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
model. Rank 0 = card #1 (CPU lanes), rank 1 = card #2 (chipset):
| Operation | rank 0 (card #1) | rank 1 (card #2) |
|---|---|---|
| `broadcast` | OK | **FAIL** `DistBackendError` |
| `all_reduce` | OK | **FAIL** |
| `barrier` | OK | **FAIL** |
| `send` / `recv` (P2P) | blocked, then timeout | **FAIL** |
| `all_gather` (after failure) | group unusable | — |
The `AMD_LOG_LEVEL=3` trace names the cause: `Pcie atomics not enabled, hostcall
not supported` → `AQL dispatch failed!` → `hipLaunchKernel` returns
`hipErrorIllegalState` for `ncclDevKernel_Generic_4(...)`. Nothing in vLLM or
RCCL mentions atomics; the runtime log is the only place it appears.
**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism
as an option, because vLLM's PP stage handoff is exactly that primitive.
## Software escapes: closed, and the last one is now moot
Every row below was written under the old board. The last row — the only door
that was ever open — stopped mattering on 2026-09-20, because the hardware gate
it existed to route around no longer applies to this host.
| Idea | Finding |
|---|---|
| vLLM's IPC custom allreduce | **Gated off by a table keyed to CUDA sm versions** — `{9.0, 10.0, 10.3, 10.7}`; ROCm reports capability `12.0` for gfx1201, so the gate is `False`. `custom_ar` itself is compiled into the image. |
| ...and if that table were patched | Still insufficient: `broadcast`, `barrier`, and the TP logits `all_gather` have no non-RCCL path. |
| The radiance fork's own allreduce | Already present and *on by default* (`RADIANCE_USE_R4D_AR`, size gate `RADIANCE_AR_MAX_KB`): a one-shot P2P push allreduce wrapping `CudaCommunicator.all_reduce`, exchanging IPC handles over the **gloo** group. With the gate raised it covers prefill and decode, and `ncclDevKernel` disappears from the profile. But it does not intercept `PyNcclCommunicator.__init__`'s **warmup allreduce**, nor `pynccl_comm.all_gather` — both direct RCCL calls. |
| NCCL symmetric memory | Unavailable: the shipped RCCL advertises NCCL 2.30.4 but exposes no `ncclDevComm*` API, so torch's symmetric-memory path cannot run. |
| Prefill/decode disaggregation | Needs a KV-transfer connector plus coordination collectives; both out. |
| gloo for GPU work | CPU/TCP collectives — orders of magnitude too slow for per-layer TP. |
| **Rebuild RCCL so its kernels do not declare a hostcall buffer** | **OPEN — unverified.** The claim is that the hostcall requirement is triggered by device-side `printf`/`assert`, and that `-mprintf-kind=buffered` avoids it. Counter-evidence: RCCL's device source at `src/device/` contains **no live** `printf`, `assert`, or `__assertfail` (checked), so what makes its kernels declare `HiddenHostcallBuffer` is unexplained. The claim traces to search-result summaries, not a primary source. Cheap to settle once a host is available: recompile librccl with `-mprintf-kind=buffered`, dump a kernel's declared arguments (`roc-obj-extract` + `llvm-readelf --notes`) and look for `hidden_hostcall_buffer`; if it is gone, a 2-rank probe should dispatch. **Stop and mask the service first** — see the note below. |
## Bandwidth: why x4 is enough
A 27B TP=2 allreduce at batch 4 moves roughly
$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$
per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**,
or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8
~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound.
That is why the cheap option (the M.2 riser) and the expensive one (the board
swap) were both considered adequate on bandwidth alone — and why the board that
landed, at x8/x8, clears the arithmetic with room to spare.
## Ruled out empirically (do not re-derive)
- Kernel versions 7.0.0-28 / -30 / -31 — byte-identical failure.
- `iommu=pt`, and `amd_iommu=off` is unrelated (that was a Dankclaw prefill knob).
- P2P hardware: 45.274 GB/s bidirectional between the cards works fine — plain
memory traffic is not the problem; *atomic* routing is.
- A from-source librccl rebuild: identical failure at the identical call path.
- Every NCCL transport/algorithm knob.
- A one-card test proves nothing: NCCL short-circuits at `world_size=1`, which
is exactly why the two replicas work at all.
- Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 —
different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure.
## Measured verdict (2026-09-20)
Both cooperation modes were run on this host on 2026-09-20 — the same salted
10-prompt battery (`scripts/bench-conversation.py`), the service stopped and
masked between arms. The arm table is [[frank-medium-radiance-mxfp4]]'s bench of
record (arms R = radiance TP=2 + DFlash2 as deployed, I = the llama.cpp layer
split that preceded it, L = llama.cpp `--split-mode tensor` against RCCL); this
page keeps only the verdict. **TP=2 won on every axis — roughly 2.1× decode and
3.6× at 4-way concurrency over the layer split — and replaced the layer split the
day of measurement.**
Arm L carries a caveat no flag can remove: llama.cpp's tensor mode could not load
the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight
pre-allocated on the non-primary device has no backend once `n_copies == 1`), so
it decoded **without** speculation while the other arms decoded with it; and part
of R's margin over I is the 4-bit MXFP4 checkpoint, not the parallelism.
Two numbers elsewhere on this page are therefore not interchangeable:
- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce,
with RCCL off, on the old board. It was never a TP decode number, and its
paired decode figure is in "The third path" below.
- The butterfly tensor arm and the RCCL tensor arm a day later are the same shape
family on different collective paths; the measured pair lives with
[[frank-medium-radiance-mxfp4]]'s arms.
RCCL needs PCIe atomics on *every* participating GPU — the gate the old board
failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is
why TP=2 runs today rather than merely "being unblocked": the image ships TheRock
RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so
radiance could not have used RCCL on this host even with atomics routed. Ubuntu's
RCCL 2.27 does carry gfx1201 kernels, and that is what arm L linked against.
**Rule for any future multi-GPU collective experiment: stop and mask the
production service first**, per the benchmarking policy, or run it on a host that
is not serving users. The production service is `frank-archon`, a **root** unit,
so on Dankserver that is `sudo systemctl stop frank-archon && sudo systemctl
mask frank-archon` (`scripts/frank-archon.service`; never `disable` — that
deletes the unit file, and `mask` is what the bench policy prescribes). The
2026-09-20 wording below — `systemctl --user stop frank-medium && systemctl
--user mask frank-medium`, frank-medium being a rootless quadlet — is retired
with that service, which today reports `not-found` (read 2026-10-18).
This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live
service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a
power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the
GPU driver is a plausible cause and this host has precedent for going
unreachable until a power cycle.
## The third path (production in the layer-split era): llama.cpp's layer split
Everything above is about *collectives*, and vLLM cannot proceed without them
in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage
handoff is a p2p `send`/`recv` of the hidden state, the primitive this host
also fails. **llama.cpp's `--split-mode layer` needs no collective at all**:
layers are distributed across the cards and the boundary activation is a
plain per-backend copy, once per forward pass. No RCCL kernel is ever launched
on the second card, so the atomics gate never comes up.
That is what `frank-medium` ran from 2026-09-16 to 09-20
([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards,
262144 context. Its measured layer-split rates are that page's; this host's
tensor arm under the butterfly fallback measured 272 t/s prefill / 26.3 tok/s
decode. **That tensor number is not a prediction for an RCCL-backed build** — it
is what tensor split costs when the allreduce degrades to a meta-backend
butterfly, and it is the number the board-swap path exists to retire.
The ranking therefore changed twice. On 2026-09-16: software already gets both
cards onto one model, so the hardware fix stopped being a capability question
and became a performance one — *tensor* parallelism, ~40 % decode, plus one
pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
hardware half of that question was answered by the board swap; the llama.cpp
measurement half never ran, and `GGML_HIP_RCCL` still defaults `OFF` in
`scripts/build-llama-rdna4.sh:41` (re-read 2026-10-18) for exactly the reason
this page documents — moot while no llama.cpp shape is deployed on this host.
## What would change the answer
- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot
since 2026-09-20.** That door only mattered while the hardware gate could not
be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis
and its counter-evidence stay in the escape table for the record.
- The symmetric-kernel path inside RCCL, if a release ever ships the device API
(`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock
nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does
not.
- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic
routing~~ — arrived as new silicon instead.
- A GPU-side workaround: none found in ROCclr, and the gate is per-device,
queried once at probe.
## See Also
- [[Pipeline Parallelism/index|Pipeline Parallelism]] — two-machine research (the PP variant of this problem)
- [[Dankserver System Specs]] — hardware inventory
- [[Observability]] — the scrape/dashboard/alert stack around these endpoints
serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
-448
View File
@@ -1,448 +0,0 @@
---
type: research
title: FastFlowLM NPU Inference
description: FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.
tags:
- fastflowlm
- npu
- xdna2
- dankclaw
- research
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
>
> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today
> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625
> (`ss -ltn`, read 2026-10-18). The runtime is still installed
> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`,
> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is
> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]]
> for the retirement record. Everything below is the 2026-08 research record.
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.
## Verdict (2026-08-16)
- The NPU is a complement, not a replacement. It fits dense models up to about 9B.
- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`.
## What FastFlowLM Is
FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.
Key properties, per the project README and docs:
- Runs fully on the NPU. No GPU or CPU load during inference.
- CLI mode (`flm run <tag>`) and server mode (`flm serve <tag>`).
- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
- Context lengths up to 256k tokens on listed models.
- Vision, audio, embedding, and MoE model support.
- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`.
## NPU Hardware on Dankclaw
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU
with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory
via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at
4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]].
Live verification on Dankclaw, 2026-08-16:
```
$ flm validate
[FLM] Fetching models from: /opt/fastflowlm/share/flm/model_list.json
[Linux] Kernel: 7.0.0-29-generic
[Linux] NPU: /dev/accel/accel0 with 8 columns
[Linux] NPU FW Version: 1.1.2.65
[Linux] amdxdna version: 0.7
[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity.
```
- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18).
- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`).
- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".
## Memory Model
The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.
Budget math for this fleet:
| Machine | RAM | NPU budget (approx. half) |
|---------|-----|---------------------------|
| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB |
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]]
and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate`
(it owns the current number).
Weight sizes (approximate, Q4-class):
| Model | Quant | Weights | Fits ~15 GiB budget? |
|-------|-------|---------|----------------------|
| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes |
| Qwen3-8B | Q4_1 | ~5 GB | yes |
| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge |
| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB |
| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists |
Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.
## Supported Models (Qwen family, plus gpt-oss)
All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.
| Tag | Base model | Context max | Notes |
|-----|-----------|-------------|-------|
| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle |
| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle |
| `qwen3:4b` | Qwen3-4B | 32k | tool calling |
| `qwen3:8b` | Qwen3-8B | 32k | tool calling |
| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking |
| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling |
| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision |
| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | |
| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision |
| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle |
| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling |
| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling |
| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling |
| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling |
| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM |
Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.
## Published Benchmarks (FLM, vendor numbers)
All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.
Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:
| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 |
| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 |
| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 |
| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 |
Prefill t/s for the same run:
| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 |
| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 |
| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 |
| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 |
Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**:
| Metric | 1k | 2k | 4k | 8k | 16k | 32k |
|--------|-----:|-----:|-----:|-----:|-----:|-----:|
| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 |
| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 |
- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current
PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at
14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade
the worker's model.
The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.
## Fit to the Frank Program
| Our model | NPU verdict |
|-----------|-------------|
| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. |
| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. |
| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. |
| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. |
Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.
## Install State on Dankclaw (verified 2026-08-16)
- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.)
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.
| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % |
|-----|-----------:|----------:|-------:|------------:|-----------:|---------:|
| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 |
| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 |
| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a |
Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.
Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.
## Model Bakeoff (measured 2026-08-16, 4 models)
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 |
| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 |
| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** |
| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 |
Notes:
- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.
## Promotion (2026-08-16)
`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused
twice after this window - see the banner - and the NPU unit today is `frank-npu`,
masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`),
wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`.
`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap
`FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-zealot
```
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept.
## Parallel Streams (measured 2026-08-16)
The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
Concurrent-request test, gpt-oss-20b, 128-token cap per stream:
| Streams | Wall s | Aggregate t/s | Per-stream wall s |
|--------:|-------:|--------------:|-------------------|
| 1 | 10.6 | 12.1 | 10.6 |
| 2 | 21.1 | 12.0 | 10.8 / 21.1 |
| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 |
The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.
Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.
## Test Runbook
This runbook targets a retired endpoint; keep it as the procedure if the NPU is
ever revived.
**Host RAM availability.** The gate is half of system RAM by default - a relic of the
install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's
box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM
first, or stop
the service the way the bench policy requires (`sudo systemctl stop
frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
otherwise restart it). The NPU shares the unified pool with everything else on
the box.
```mermaid
graph TD
A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
B --> C{NPU visible to XRT?}
C -- no --> D[Fix driver or firmware path]
C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
E --> F{Generates tokens?}
F -- no --> D
F -- yes --> G[4. flm serve on a scratch port, never 7770]
G --> H[5. Bench decode + prefill t/s]
H --> I{Speed acceptable?}
I -- no --> J[Record results, stop]
I -- yes --> K[6. Attempt gpt-oss:20b]
K --> L{Loads in ~15 GiB?}
L -- no --> J
L -- yes --> M[Bench, record]
M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
N --> O[Record, decision]
```
1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead.
2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device.
3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit.
4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
## Registering packs absent from the catalog (NPU2)
Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
to the FLM registry. Registry facts: user registry
`/home/dank/.config/flm/model_list.json`, shape
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
file_url, size, flm_min_version, files[], vlm, default_context_length,
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
env var set lists custom tags (`⏬` = not downloaded).
1. Back up the registry JSON first (`cp -a`).
2. Insert the entry via a python json edit + atomic `os.replace`. Set
`files` to the FULL HF file list — **including `vision_weight.q4nx` even
for text-only packs**: the qwen3.5-family loader opens it unconditionally
and a missing file makes the first request HANG silently (no error log).
3. **Do not use `flm pull <tag>` for custom entries** — it throws
`json.exception out_of_range key '<tag>:<size>' not found` while building
the download list. Place files manually into
`/home/dank/.config/flm/models/<PackName>/` with
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
— plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
weights; verify byte counts against the HF tree API before trusting a
download.
4. Kernel packs: copy from the same-shape family pack (shape-match
`config.json` on hidden_size/num_hidden_layers/head_dim first) into
`/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
(root-owned dir).
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
the new process loads with all files present.
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
`/v1/models` alone proves registration, not loadability.
Hard constraints: only ONE `flm serve` process can hold the NPU hardware
context — a second concurrent serve process fails every load with
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
0x2000003); never run scratch/verify instances while production holds the
NPU. Tag switching happens INSIDE one serve process; the first request
after a model switch pays ~6–11 s cold-load cost (visible in
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
pattern present in your own command line — use bracket-guarded patterns
(`[q]wen3.5:4b`) or exact PIDs.
## Tag load-failure diagnosis (200 with an error body)
`POST /v1/chat/completions` answers **HTTP 200** with a flat
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
200. A status-code-only client reads this as a successful empty completion —
**always check for the `error` key**. Two distinct causes, distinguished via
`journalctl -u <flm-service> --full`:
1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
Correct behavior, not a bug.
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
downloads missing weight files on the fly (first request hangs ~90–110 s),
then fails `Failed to load model: No such file
'/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
attempts fail fast (~2 s). Root cause: the serve script exports
`FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
kernel lookup resolves under the user dir, which only contains symlinks
someone linked. **A tag listed in `/v1/models` is NOT proof it is
loadable on this host.** Fix: one symlink per missing pack
(`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
/home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
symlink the first request loaded and answered in 7.4 s wall.
Switch-cost behavior: the `model` field routes per-request (no restart
needed); a request whose model differs from the resident one triggers a full
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
alternating tags pays the swap each time. Single-lane NPU: concurrent
requests queue FIFO behind the current lock holder (`NPU Locked!` →
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
the journal for `Failed to load model` (the quoted xclbin path names the
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
## Incident: qwen3.6-moe unreachable (2026-08-16)
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
### Unknown-tag fallback (FLM behavior)
When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`:
- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
- FLM then loads the 1B model in place of the requested model.
- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.
**Correction (live-probed 2026-08-30):** the fallback no longer fires on the
current serve path — an unknown tag now answers HTTP 200 with a flat
`{"error":"Failed to load <tag> model!"}` body and nothing is loaded or served;
`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200.
Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model.
### Request compatibility: the `developer` role (found 2026-08-16)
pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection:
- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template.
- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`.
- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug.
- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`.
- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error.
## Risks and Open Questions
- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot
of Dankclaw killed it. The fleet law today is the opposite: every deployed
container/service ships a boot-persistent, self-healing unit, and the retired
NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it
cannot come back at boot.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then
served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the
`amd_iommu=off` decision that day and the unit is masked today. The remaining
NPU content (benchmarks, runbook steps) is research on a device that is again
usable since 2026-10-18 but serves nothing.
## Citations
1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`.
9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection.
serverAppConfigs-docs/LLM/FreeToken Builds.md
-97
View File
@@ -1,97 +0,0 @@
---
type: reference
title: FreeToken Builds
description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
tags: [freetoken, build, dankclaw, vllm, mtp]
timestamp: 2026-10-05T00:00:00-07:00
---
# FreeToken Builds
FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
different dependency traps. Both were verified end-to-end 2026-09-11/12.
## Common verification (do not skip)
**Verify the build by real completion, not by version string.** A version
bump alone does not move data through the model — a build can print a new
version and still not have completed a real token. Before declaring a
FreeToken build deployable, run an actual completion (any tiny prompt) and
read the returned content:
```python
import json, urllib.request
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req, timeout=600)))
```
A 200 with a real `choices[0].message.content` is the only build green.
## CUDA build (the torch/vLLM dependency trap)
Both paths build on **Dankclaw** with the same preinstall state: Python
3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
download.pytorch.org/whl/nightly/cu130, installed with
`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
--index-url https://download.pytorch.org/whl/nightly/cu130` — use
`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
version). Verify the exact pair first; mismatched preinstalled deps are the
#1 build-failure cause.
Then install the **matching vllm dev wheel**:
```bash
uv pip install --index-strategy unsafe-best-match \
--index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
--extra-index-url https://wheels.vllm.ai/nightly/cu130
```
The wheel URL pattern is fixed:
`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
— read it back from vLLM's CI-run page (`run details →
build_and_install_package` job, "install the vllm wheel" step). **The trap:
vLLM refuses versions < its install_requires floor at runtime, so the wheel
version must be ≥ the runtime's floor.** A wheel built at a lower base
(0.5.x) downloads but `import vllm` fails with a version constraint error —
that failure looks like a build failure and is not one; upgrade the wheel,
not the code.
If `uv pip install -e . --no-deps` refuses (some repos pin the
`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
after `python -m venv --copies .venv && source .venv/bin/activate` and
`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
## ROCm build (gfx1151 via TheRock nightly)
Same venv discipline, ROCm wheels from TheRock's nightly index:
```bash
uv pip install --index-strategy unsafe-best-match \
--index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
"torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
```
Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
prebuilt extension wheels into the tree (the repo's `setup.py` skips
compiling them when found). The ROCm build also needs
`install-fastrak-gfx1151.sh` from the repo's scripts and
`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
`franksdank/freetoken-ops-gfx1151` — fetch it once and point
`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
the extension API changed.
`--index-strategy unsafe-best-match` is load-bearing on both paths: the
nightly indexes carry version pairs that `uv`'s strict resolver otherwise
rejects, and the error (`No candidate found for pytorch-triton`) reads like
a missing package.
## Related
- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
serverAppConfigs-docs/LLM/GGUF Compatibility.md
+8 -8
View File
@@ -1,7 +1,7 @@
---
type: reference
title: GGUF Compatibility and Runtime Lock-in
description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
description: "Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can."
tags: [gguf, llama.cpp, forks, quantization, speculative]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -14,7 +14,7 @@ Studio refuse custom types at load (`unknown tensor type`); a fork-private type
means exactly one engine can serve the file. Measured on this fleet repeatedly:
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
tables live in and
(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
## 1. Read the header, do not guess
@@ -69,14 +69,14 @@ filenames lie about quantization and about MTP/vision heads.
```
Validate before trusting: the parse is right iff the tensor-info block ends
somewhere sane and names look real (`blk.0.attn_q.weight`,
somewhere sane and names look real (`blk.0.attn_q.weight`
`token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
offset — do not work around it.
3. **Diff what engines actually bind on**: `general.architecture`; tensor
count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`
`.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
(`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
tool usually comes out byte-identical in every one of these — that is the
@@ -117,8 +117,8 @@ re-download.
## Related
- [[Llama Cpp Runtimes]] — which fork ships which custom types
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and
- which fork ships which custom types
- the enum tables and code-level differences
* the era that served the ftype-115 file, and
[[frank-dragoon]] for what runs there now
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md
+8 -8
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Heterogeneous Split Triage
description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
description: "Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys."
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -93,7 +93,7 @@ of bytes), so it is tempting. Before proposing it:
(compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
must be a head **trained for that target** (EAGLE/DFlash/MTP are
conditioned on the target's hidden states and are not portable between
models);
models)
- a plain drafter costs **K sequential forwards per round**, so it must be
10–50× faster than the target. Break-even with `r = t_draft/t_target`:
`rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
@@ -107,10 +107,10 @@ of bytes), so it is tempting. Before proposing it:
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device`
(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268),
(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268)
`--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v`
(`-ctkd`/`-ctvd`, 4111/4124) all exist in
`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp),
`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)
read 2026-10-18).
## 6. Expressing a split (llama.cpp, the usual vehicle)
@@ -122,8 +122,8 @@ read 2026-10-18).
- buffer types match by name at load (`ggml_backend_buft_name`), so device
names work as `-ot` values;
- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`,
`common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`;
- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`
`common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`
- several backends in one binary is fine (CUDA + Vulkan is the practical
pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs
do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the
@@ -155,7 +155,7 @@ Rank and report; prefer the boring answer when it wins:
1. **Two independent endpoints** (zero shared traffic) — often the real
answer to "what's a useful combination", especially over a narrow link.
2. **One process, weights-resident, KV co-located with attention on the
fast pool** — promising and quantifiable (+% decode at long context);
fast pool** — promising and quantifiable (+% decode at long context)
needs a dual-backend build and a baseline measurement.
3. **Hot-expert cache / per-token weight streaming** — affordable only if
the miss set per token is small (a few experts; check
@@ -169,5 +169,5 @@ Rank and report; prefer the boring answer when it wins:
## Related
- [[PCIe Link and Atomics]] — the link and collective walls underneath
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
- the deployed split shapes
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md
-225
View File
@@ -1,225 +0,0 @@
---
type: reference
description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing."
tags:
- llm
timestamp: 2026-08-28
---
# Hybrid CPU GPU Processing Dankserver
## Overview
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon]]).
## Architecture
### Model Weights
In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2).
### KV Cache
KV cache moves between GPU and CPU:
1. Active KV cache pages stay in GPU VRAM
2. Inactive pages swap to CPU RAM
3. Swapped pages write to disk swap space if RAM is full
4. Pages reload to GPU when needed
### Compute Distribution
- GPU handles all matrix multiplication and attention computation
- CPU handles tokenization, detokenization, and input processing
- CPU handles KV cache swap operations
- GPU worker processes run on CPU cores via multiprocessing
## Optimal Settings for Dankserver
### GPU Memory Utilization
Set `--gpu-memory-utilization` based on your GPU:
| GPU | Recommended Value |
|-----|-------------------|
| 24 GB (RTX 3090/4090) | 0.7-0.8 |
| 40 GB (A100) | 0.8-0.9 |
| 80 GB (A100 80GB) | 0.85-0.95 |
| 80 GB (H100) | 0.9-0.95 |
Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s.
The live engine does not use this knob at all — it claims an explicit per-card
pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`,
`scripts/frank-archon.env:230-231`).
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
### Swap Space / KV Offload
Current vLLM has no `--swap-space` engine argument, and the CPU KV-offload knob pair
is owned by [[RAM CPU Offloading in VLLM]] (that page carries the upstream
`CacheConfig` verification). In Podman, mount the host cache read-write for KV offload
writes. The table below shows historical `--swap-space` guidance as a sizing reference.
| System RAM | Recommended Swap |
|------------|------------------|
| 32 GB | 8-16 GB |
| 64 GB | 16-32 GB |
| 128 GB | 32-64 GB |
| 256 GB | 64-128 GB |
More swap space allows longer contexts and more concurrent requests.
### Tensor Parallelism
Use tensor parallelism for large models:
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.85
```
### KV Cache Data Type
Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache).
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9
```
FP8 KV cache reduces memory usage by 50% compared to BF16.
## NUMA Optimization
Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies:
```
vllm serve model \
--tensor-parallel-size 4 \
--numa-bind
```
This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.
For explicit NUMA binding:
```
vllm serve model \
--tensor-parallel-size 4 \
--numa-bind \
--numa-bind-nodes 0 0 1 1
```
## CPU Resource Requirements
VLLM needs CPU cores for:
- 1 API server process
- 1 engine core process
- N GPU worker processes (one per GPU)
Minimum physical cores = 2 + N (where N is GPU count).
For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read
2026-10-18):
- Minimum 4 physical CPU cores by the 2 + N rule
- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off)
## Performance Tuning
### Chunked Prefill
Chunked prefill is enabled by default. Tune with:
```
vllm serve model \
--max-num-batched-tokens 8192
```
Higher values improve throughput. Lower values improve latency.
### CUDA Graphs
CUDA graphs improve throughput. Disable for faster startup:
```
vllm serve model \
--enforce-eager
```
### Compilation Cache
Reuse torch.compile artifacts:
```
export VLLM_CACHE_ROOT=~/.cache/vllm
vllm serve model
```
Copy cache between boots for faster startup.
### Skip Memory Profiling
Skip memory profiling on repeated boots:
```
vllm serve model \
--kv-cache-memory <value_from_first_boot_log>
```
## Monitoring
Check GPU memory:
```
rocm-smi
```
Check for preemption warnings in VLLM logs.
Check CPU utilization:
```
htop
```
## Example Commands
### Single GPU, Medium Model
```
vllm serve Qwen/Qwen3.6-35B-A3B \
--gpu-memory-utilization 0.85 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
### Single GPU, FP8 Model
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--max-num-batched-tokens 8192
```
### Multi-GPU, Large Model
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--max-model-len 8192
```
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md
-271
View File
@@ -1,271 +0,0 @@
---
type: reference
title: LLM Bench Windows and Measurement
description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
timestamp: 2026-10-05T00:00:00-07:00
---
# LLM Bench Windows and Measurement
Fleet law first: **STOP the production service before any GPU experiment**
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
GPU and corrupts results; an unisolated GPU experiment has precedent for
wedging a host until power cycle. Reference harnesses:
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
`scripts/autoresearch/frank-archon-arms.sh`.
## 1. The window structure (one config per arm, a driver owns the window)
1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
EXITED containers and can never clear — `rm -f` the exited one first) AND
port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
`ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
then polls `/health` gets **200 from the still-running PRODUCTION engine**
— the arm "passes" with production's boot log, then `podman exec` dies
with `container state improper`.
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
LLM hosts — one card, one server). The serve script takes its config via
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
pointing at a staged copy; the checkout is never mutated.
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
the loop — a refused boot dies in one poll instead of 20 min. Capture the
serve script's FULL stdout on failure (a `tail -8` loses engine-init root
causes; raise happens above the tail).
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
SpecDecoding lines) — they survive measure crashes.
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
`podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
before the next boot; restore production from the EXIT trap and verify the
**conjunction oracle**: health 200 + `is-active` + `is-enabled` +
mask-symlink clean. A bare `systemctl start` is not a restore oracle.
(`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
enough: systemd never auto-restarts an explicitly stopped unit.)
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
mixed speeds.
**Destructive tails (2026-10-01):** a scratch script whose tail does
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
probe loop survived the driver kill — it can destroy a just-deployed
production engine (signature: deploy exits green, then seconds-to-minutes
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
and kill strays by explicit PID; killing the dev-side job does NOT kill the
host-side `deploy-remote.sh` (two of them once managed the same unit).
Two failure shapes inside the window that read as "the bench died":
- **Stale reads from a stopping container.** `podman stop` answers while the
engine inside is still flushing; a `podman logs` taken in that window
returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
`podman wait`) before reading logs, or you will write "no results" into
the table for a run that has them.
- **A D-state worker wedges the stop.** A bench child that wedged in the
kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
then the container is `Stopping` forever. Kill by container first
(`podman kill <ctr>` → SIGKILL path), verify the process is gone from
`podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
window script proceed to a redeploy with a zombie bench worker holding
VRAM.
## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
**What llama-bench cannot express — decide BEFORE designing:** no
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
activation buffers fit beside the DEPLOYED KV — only the real unit answers
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
(`tg` numbers are the drafterless baseline, useful as a control only), and a
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
~zero depth and meaningless for long conversations.
**The separator trap:** `/` separates devices WITHIN one configuration, `,`
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
**Harness shape:** one invocation per cell (an infeasible cell calls
`exit(1)` and kills every later cell of a cross-product invocation — and
infeasibility is usually the question); TSV row per cell as it completes +
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
intended looks fast); resume by skipping non-empty cell outputs; stop the
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
**Run under a transient unit, not a shell:**
```bash
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
--property=LimitMEMLOCK=infinity --property=User=dank \
/home/dank/serverAppConfigs/scripts/<harness>.sh
```
Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
8 MiB and fails the load for an unrelated reason. This also survives ssh
disconnects. **Boot rungs under the production environment, not a bench
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
(systemd `User=` + memlock change what the pinned-expert path holds):
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
Restart=no env VARFILE=… /path/serve.sh`.
**Verify before burning hours:** dry-run against a STUB binary with
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
including one forced-`exit 3` cell — prove the TSV row, failure capture,
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
result); single-sequence decode is usually flat across batch/ubatch (prefill
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
context).
## 3. HTTP benches of server-side parameters
`llama-bench` benches the bench binary, not the running server — server-side
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
HTTP:
- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
`prompt_tokens / wall_time`.
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
spec acceptance all stabilize over thousands of tokens); measure
`predicted_per_second` from `timings` or `completion_tokens / wall_time`;
temperature 0.
Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
reasoning parser streams zero deltas — use `/v1/completions`.
## 4. The measurement law (under speculative decoding this is absolute)
**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
said 23–26 — a 3x false regression that nearly abandoned the change that
worked. Always:
```python
decode_tok_s = usage.completion_tokens / (wall - ttft)
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
```
vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
nearly shipped the wrong setting.
## 5. Honest methodology
- **Match the corpus to the workload.** Token soup under-reports speculative
decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
settings; acceptance, not step time, moves throughput. Build the corpus
from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
`--custom-output-len`.
- **Salt every run** with a unique per-arm token in the first bytes — it is
the prefix-cache key (and a proxy routing key). Without it arm 2 measures
arm 1's warm cache.
- **Two workload shapes need two clients:** distinct prompts (batch
behaviour, prefill parallelism) and sequential turns with growing history
(TTFT of the first streamed chunk per turn — the only measurement that
exposes prefix-cache behaviour). These pull in opposite directions: a cold
batch spreads prefill across replicas, a conversation's next turn only pays
if it lands where its prefix is cached.
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
`GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
is a LIFETIME average across every prompt the replica ever saw — send the
same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
→ 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
three points before treating a knob as a lever.
- **Soak before believing:** 15–30 min of concurrent families, count failures
AND latency; report hit/miss split and medians with min/max — the median of
a bimodal distribution describes neither mode.
- **Verification that is not the benchmark:** run the client the fleet
actually uses; and a deployed unit is not proven until an unattended reboot
brings it back.
- **Record the "no" answers** with numbers and mechanism — placement balance,
concurrency admission and chunk-size tweaks all measured flat or worse
here; the wall was arithmetic (cached prefixes + live sequences share one
pool).
## 6. Sweep hygiene (the arms that produced wrong verdicts)
- **Prefetch uniformity:** every arm in a sweep carries the same
expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
verdict.
- **Knob-landed preflight:** for every `-e` knob, confirm
`podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
produces a "parity" arm that is really the default (v1 of the gather sweep
lost all 3 arms this way; serve-script hooks are additive and can drop
vars).
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
compare against default-off, not "incumbent + one flag".
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
autofit (remainder) so pool numbers are reproducible; compute need =
tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
states — measure, don't assume 2x).
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
`mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
## 7. Bash laws that ate windows (each cost an arm)
1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
`SPEC=3: command not found`, nothing runs. Pass literal assignments:
`SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
an if/else it aborts ONLY that compound statement — the script CONTINUES,
every arm tears down clean with zero measurements. Detection law: `ls`
the per-arm `.measure.txt` files before trusting any summary; arm echoes
are not evidence.
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
it aborts AFTER the harness stopped production. Two statements.
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
mid-flight patches miss arms launched in the preceding minutes. Patch
early or run a make-up window.
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
(`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
fails silently. Same trap in `git commit -m`: backticks become command
substitution.
6. No comments INSIDE a backslash-continued command (the continuation joins
the comment); verify multi-line command construction with `bash -n` AND a
render test.
## 8. Results location and recovery (frank-archon arms)
| copy | path |
|---|---|
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
constants 99.9/88.8 identical across arms by construction; **identical
numbers across arms == stub or bug, never a measurement.** After compaction,
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
window.
## Related
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
- [[LLM Performance Diagnosis]] — profiling and attribution methods
- [[Remote Job Runner]] — detached execution of long sweeps
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md
+9 -9
View File
@@ -1,7 +1,7 @@
---
type: reference
title: LLM Performance Diagnosis
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution."
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -21,7 +21,7 @@ the cost; the problem is per-token expert movement and KV/attention tuning is
ruled out entirely. A genuinely context-bound server shows a clear downward
slope. Record cold vs warm separately — a small gap means the cache is
working. The canonical flat run (fp8 expert-offload shape, a few hundred to
sixteen thousand tokens of context) is in [[Tcclaviger Fork Ops]]'s profile.
sixteen thousand tokens of context) is in 's profile.
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
@@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
No amount of link arithmetic would have produced those two conclusions: the
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It
predicted a negative result in advance (spec decode cannot help a shape whose
per-token cost scales with the token count) which matched an unexplained flat
MTP measurement.
@@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing.
- One `max_tokens=200` request is production traffic and fine; anything that
pins the card is a bench → stop-and-mask first. Check WHICH instance you
are reading (`systemctl status | head -12`; task ids jump across
restarts). The engine's own ledger (`KV self size`, `n_slots`,
restarts). The engine's own ledger (`KV self size`, `n_slots`
`n_ctx_slot`) outranks your model of it.
## 5. The fork's data path (before blaming the link)
@@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as
draft depth grows. Check the threshold's premise: a default tuned at
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required)
check for the skip line before concluding autotuning ran. Gather-geometry
knobs read at import time → each arm is a full reload.
@@ -189,7 +189,7 @@ analyze; `--analyze-only`).
Measure with a **sequential-turn client**, not only a batch benchmark — a
batch of distinct prompts parallelises prefill across replicas and hides the
problem. The two failure modes: round-robin loses every other cache hit
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned)
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
pinned). **The rule that gets both: route by request shape** — a request
carrying more than one message is a continuation (agent loop re-sending its
@@ -209,13 +209,13 @@ is not a per-request verdict (probe directly: same prompt twice to an idle
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
against the live endpoint before dashboards. Test shape: drive the routing
function directly with synthetic bodies (stdlib-only policy module);
function directly with synthetic bodies (stdlib-only policy module)
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
hashes differently per turn and pinning silently never happens.
## Related
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
- the window discipline and usage-counter law
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
- the compile fix for the pointwise-glue bound
serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md
-287
View File
@@ -1,287 +0,0 @@
---
type: reference
title: Llama Cpp Multi-Device Splits
description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
timestamp: 2026-10-05T00:00:00-07:00
---
# Llama Cpp Multi-Device Splits
## 1. Expressibility law — classify the arrangement before running anything
llama.cpp expresses exactly two things across devices, and they are not
interchangeable:
| arrangement | how | expressible? |
|---|---|---|
| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
Upstream anchors, re-read 2026-10-18 against `master`:
`src/llama-context.cpp:517-526` computes `pipeline_parallel` with
`!model.has_tensor_overrides()` as a required term;
`ggml/src/ggml-backend.cpp:1089` carries the verbatim abort
`pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)`
(the `:941` anchor above is where it sat in the `37b53fd` build this was
measured on); and `common/arg.cpp` defines `-sm/--split-mode` (2821),
`-fit/--fit` (2882), `--spec-type` (4295), `--spec-draft-device` (4259),
`--spec-draft-n-max` (4161), `-fa/--flash-attn` (1758) — every flag in §4 is
real upstream, not fork-only
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)).
Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
cross-backend pair → model load fails with only `failed to load model`;
`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
a card can be the trunk OR the offload host, never the complement of an
AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
override, not the driver mix.
If the plan needs tensors on a specific device: stop, the answer is no for
any vendor mix. Expressible alternatives, in order: (1) two independent
endpoints, one model per device; (2) single-vendor layer split; (3) device +
CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
Layer-split arithmetic to check before promising: a whole layer carries its
experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
non-repeating tensors (PLE tables) land on the main device and cannot move —
they cap how small the primary can get.
## 2. No collectives? The atomics wall and the layer-split answer
Decide with one line before designing:
```bash
sudo lspci -vv -s <card2> | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever
lspci -tv # a Routing- bridge is a hardware wall
```
`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
device-backend `send`/`recv` of the hidden state
(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
collective: layers distribute in order, only the boundary activation crosses
PCIe once per forward pass. Full probe/repair detail:
[[PCIe Link and Atomics]].
## 3. Build: pin by feature, bundle the runtime at the exact depth
- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
`ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
(`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
mode degrades to the butterfly fallback; layer split unaffected).
- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
(`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
the engine loads every GiB, answers `/health`, and dies on the FIRST request
with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
`patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
`glslc` + libvulkan headers. **Device names are not what you expect:**
`--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
`Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
llama.cpp print the valid set and exit (self-diagnosing).
- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
CMakeCache.
## 4. Serve flags that matter on a split
```bash
HIP_VISIBLE_DEVICES=0,1 llama-server \
--device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
--gpu-layers 999 --fit off --flash-attn on \
--ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
--jinja --metrics --cache-prompt \
--spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
--spec-draft-device ROCm1 --spec-draft-n-max 7
```
- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
is the concurrency policy, concurrent agent loops queue.
- Pin the drafter to the card holding the target's LAST layers
(`--spec-draft-device`); under tensor mode an unpinned draft joins the
meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
low/medium/high/xhigh and map harness levels before declaring ready.
## 5. Split balancing: measure, never interpolate
**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
single step; the best integer split can sit 0.2 GiB from balance while its
neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
a gate; treat a NEGATIVE fitted constant as a unit bug).
Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
buffers are **~2x heavier on the output/drafter card** — 512→1536 added
~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
only the distribution moves; use that to sanity-check a suspicious row.
**Procedure (one split per load):**
```bash
WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
export FRANK_MEDIUM_ENV_FILE=$WORK
for ts in 39,26 38,27 37,28 36,29 35,30; do
sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
# poll /health (or process exit), then:
rocm-smi --showmeminfo vram | grep 'VRAM Total Used' # sample ~6 s after health
pkill -x llama-server; sleep 6
done
```
Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
and poll from a separate call (a load is 30–90 s). Pick the smallest
inter-card gap that clears the per-card floor with room to spare; one `-ts`
step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
not to solve. Prefer balance: the busy card's headroom bounds the context
(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
not prevent a boundary-copy fault.
## 6. Drafter KV: sized from the TARGET, bounded by its window policy
`common/speculative.cpp` builds the draft context with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — `common/speculative.cpp:2663` on
`master`, re-read 2026-10-18 — **no flag caps it.** So:
```
draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
```
…unless the drafter declares a sliding window. Measured at 262,144 target
ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
balance point, and swapping them invalidated an already-balanced split.
**Failure signature:** the engine loads the whole model, answers `/health`,
and dies on the FIRST draft decode with `ROCm error: out of memory` /
`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
a sizing failure, not a broken drafter; under a self-healing unit it presents
as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
`key_length` from the drafter's GGUF header first. A compact reader (the
three format traps: no KV alignment — v3 alignment applies only to the tensor
section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
is the v2/v3 file order, NOT gguf.h's 2/3):
```python
import struct, sys
def rd(path):
f = open(path, 'rb'); assert f.read(4) == b'GGUF'
struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
nk = struct.unpack('<Q', f.read(8))[0]
def s():
n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
keys = {}
for _ in range(nk):
k = s(); t = struct.unpack('<I', f.read(4))[0]
if t == 8: keys[k] = s()
elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
else:
et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
if et == 8: keys[k] = [s() for _ in range(n)]
elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
else: raise Exception('unsupported arr elem type %d' % et)
return keys
```
## 7. Heterogeneous placement triage (go/no-go with arithmetic)
- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
almost never the constraint; **hop count is** — every boundary is a host
sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
`graph splits` from the loader log.
- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
context` (attention reads the whole cache every token). Highest-leverage
placement: **KV on the fastest-memory device together with its attention
compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
VRAM`.
- **What placement never buys:** prefill is compute-bound and expert weights
are read once per batch — moving them buys nothing and adds a serial stage.
Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
`1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
points (assumes sequential stages — true at batch 1).
- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
window means the "GPU memory" IS the same LPDDR: adding a stage there adds
a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
but needs the exact tokenizer for a plain drafter or a head trained for
that target; a plain drafter costs K sequential forwards (break-even needs
p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
already committing ~1.7 tok/round on its own MTP head is a high bar); and
check the runtime accepts external drafters at all (closed engines 400
unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
capture can abort on unusual graph shapes; make the FIRST measurement a
single-device baseline of the runtime on the new device — a split's
arithmetic is worthless if the baseline loses to the incumbent; a split
consumes the fast device entirely — check what it was serving.
**Verdict ranking:** (1) two independent endpoints — often the real answer;
(2) one process, weights-resident, KV co-located with attention on the fast
pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
overlaps — no mainstream runtime implements it; (4) name what does NOT help:
TP across unequal devices, KV on the far device, loading over the link,
prefill-motivated placements.
## 8. Harness syntax and verification
`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
any placement-dependent number needs an independent memory check
(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
Verification battery: `/health`; both cards' memory ≈ half the weights each
(busier card also carries the drafter; compare `model_bytes/2 + drafter +
ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
N·bytes/2 — neither card's total may equal the model file (that arithmetic
rules out replication); `spec_decode_num_drafts_total` > 0 and a real
completion; a deep prompt (140k) proves a raised window is REAL, not
declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
(default journal output truncates these lines).
Ops traps: a server started over ssh outlives stopping the runner (keeps the
port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
treating a grep count as a live fault.
## Related
- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history
serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md
+8 -9
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Llama Cpp Serving Sizing
description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
description: "Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache."
tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -19,7 +19,7 @@ guesswork.
`n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
`kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
state buffer — slot count is a memory term, not free.
2. **Two boots on the real card** (production stopped, per the bench law),
2. **Two boots on the real card** (production stopped, per the bench law)
each near the target; record `nvidia-smi` used/total plus the boot log's
`n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
`non-KV = used − (tokens × KiB/token)`; the two rows separate
@@ -31,7 +31,7 @@ guesswork.
slot gets an equal integer window. Boot and measure; if outside the band,
walk one step — never re-derive from a formula.
4. **Propagate the per-slot number — it is the client contract:**
`models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
`models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window)
or long conversations die with `400 exceed_context_size_error` instead of
compacting; compaction/summary budgets sized to the old window get
re-checked; endpoint docs and the rollback table move too.
@@ -66,13 +66,13 @@ co-tenants.
- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
contiguous blocks — the honest fragmentation signal: a block census that drops
across a restart means the model did not come back into the same hole; the
before/after census this was learned on is that era's, on [[frank-dragoon]]),
before/after census this was learned on is that era's, on [[frank-dragoon]])
device budget, KV pool/working split, "left for everything
else". llama.cpp prints them at `llama-model.cpp:1827` (per-backend model buffer)
and `llama-context.cpp:692` (compute buffer) on the era's build; on today's
`master` the same `LLAMA_LOG_INFO` lines sit at `llama-model.cpp:2009`
("model buffer size =") and `llama-context.cpp:842` ("compute buffer size =")
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp),
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp)
re-read 2026-10-18) — but **`llama-bench`
suppresses loader INFO and a systemd unit did not put them in the journal**
(verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
@@ -92,7 +92,6 @@ co-tenants.
projector, never weights alone. Do
the capacity subtraction before designing, on the card number the owner page
quotes, not the advertised one.
- **Context lengths: derive, never remember.** `llama-bench` computes
`n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
@@ -116,7 +115,7 @@ of the whole sequence + constant recurrent state + the draft context's KV:
`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
draft_per_token`. The worked qwen35-hybrid instance of that formula (full layer
count, f16 KV, MTP draft) and the slot-size table it prices are on
[[frank-medium-llamacpp-dflash2]]; cross-check any such arithmetic against the
; cross-check any such arithmetic against the
journal's `prompt state size` lines — the two must agree.
**The journal is the instrument:** `exceeds cache size limit … skipping` =
@@ -171,6 +170,6 @@ part of that era's record on [[frank-dragoon]].
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-zealot]] — the endpoint whose Eras rows are the two-boot priced shapes this method produced
- [[frank-medium-llamacpp-dflash2]] — the state-law worked example behind section 3
- the state-law worked example behind section 3
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
- / — the multi-device side
serverAppConfigs-docs/LLM/Model Swap and Rollback.md
+10 -10
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Model Swap and Rollback
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
description: "The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert."
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -51,9 +51,9 @@ express; per-tensor cross-device placement stays refused under
## 2. Ground truth first
- Read the HF model card (runtime pin, hard constraints, recommended args)
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`
`scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
- Check the host before downloading anything: `ls /home/dank/models/`,
- Check the host before downloading anything: `ls /home/dank/models/`
`df -h /home`, `du -sh` — bench-era or prior-deployment files often already
exist.
@@ -129,7 +129,7 @@ arms possible — keep it intact.
`images` is silently ignored; probe with ≥256 px).
- vLLM specifics: first request after boot can take ~18 s (deferred Triton
loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
the contract ([[vLLM Boot Forensics and Config Interrogation]]).
the contract ().
## 7. systemd handoff — THE TRAP
@@ -205,7 +205,7 @@ Pure measurement sessions that changed no state need only pin 1.
ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
MMLU-Pro vs Q8_0).
- GGUF parsing gotcha: expert tensors are
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`
a `len(parts) == 3` filter matches zero layers with no parse error. Filter
on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
@@ -214,7 +214,7 @@ Pure measurement sessions that changed no state need only pin 1.
When a host must serve two models at once (an iGPU runtime plus a second
card), or one endpoint is moving and a new one takes the freed hardware.
**0. Decide the shape before touching hardware.** One provider per
**0. Decide the shape before touching hardware.** One provider
HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
— a second endpoint is a new provider block, not a second model in the
existing one. Read the budget as an inequality: for unified memory the
@@ -235,7 +235,7 @@ suffix — and check what pointed at them: bench harnesses, a chat template
only the old runtime read, and `scripts/test-changed.sh` mappings are the
usual victims (a mapping to a retired test file fails the pre-commit
wrapper silently for weeks). One bats file per service asserting
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation)
not constants; a negative grep must strip comments first
(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
the very thing you removed.
@@ -258,12 +258,12 @@ target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire
contract.
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
<large> --service <small>` (order matters); per endpoint: `/health`,
<large> --service <small>` (order matters); per endpoint: `/health`
`/v1/models` reports the alias, a chat completion returns content AND
`reasoning_content`, a `tools` request returns `tool_calls`, an image
request is answered (or refused with a named reason), the runtime's own
speculation counters move, and the unit is `enabled` and `active`. All three LLM
endpoints are **root** units today (`scripts/frank-archon.service`,
endpoints are **root** units today (`scripts/frank-archon.service`
`scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is
not what gates them; linger matters for rootless quadlets. Read the engine's boot
ledger for a silent clamp. Finally spend one
@@ -282,6 +282,6 @@ predecessor "stopped" (the port race above).
- [[GGUF Compatibility]] — which runtime can load a payload at all
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
- measuring the new deployment
* [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era
(2026-09-29 → 10-05) is a row in its Eras table
serverAppConfigs-docs/LLM/PCIe Link and Atomics.md
+22 -22
View File
@@ -1,7 +1,7 @@
---
type: reference
title: PCIe Link and Atomics
description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
description: "Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives."
tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -61,13 +61,13 @@ H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
LPDDR5X on the Strix Halo — 128 GB LPDDR5x-8000 on a 256-bit bus,
[amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)).
**If the link is >10x slower than the memory behind it (x4 is 31x here),
**If the link is >10x slower than the memory behind it (x4 is 31x here)
streaming loses by construction and no tuning rescues it; doubling the link
only doubles a losing number.**
Worked example: the Dankclaw Gen4 x4 run of 2026-09-16 — measured H2D/D2H
rates, the 3 MiB slab cost, and the streaming-vs-compute comparison it decided
— is recorded on [[x4-pair-experiment]] §The link, measured. Read it once and
— is recorded on §The link, measured. Read it once and
apply the rule above: on that link the streaming design lost by 3–5x, and the
x8 ceiling would only have halved the loss.
@@ -88,7 +88,7 @@ time, so encode them:
+ `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
segfaulting process loses block-buffered stdout and hides the crash point.
- Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
--offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
--offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`
never `| tail -N` (truncation hid a missing `#include <cstring>`).
- `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
- Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
@@ -109,28 +109,28 @@ static double one(size_t bytes, bool pinned, int iters) {
if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
else { host = malloc(bytes); }
memset(host, 0x5a, bytes);
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
hipDeviceSynchronize();
hipEventRecord(a, 0);
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
hipEventRecord(b, 0); hipEventSynchronize(b);
float ms = 0; hipEventElapsedTime(&ms, a, b);
memset(host, 0x5a, bytes)
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b)
for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice)
hipDeviceSynchronize()
hipEventRecord(a, 0)
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice)
hipEventRecord(b, 0); hipEventSynchronize(b)
float ms = 0; hipEventElapsedTime(&ms, a, b)
double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
if (!pinned) free(host); else hipHostFree(host);
hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
if (!pinned) free(host); else hipHostFree(host)
hipFree(dev); hipEventDestroy(a); hipEventDestroy(b)
return gb;
}
int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IONBF, 0);
setvbuf(stdout, NULL, _IONBF, 0)
if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
for (int i = 0; i < 6; i++) {
size_t b = sz[i];
int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
printf(" %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
b/1048576.0, one(b,true,iters), one(b,false,iters));
b/1048576.0, one(b,true,iters), one(b,false,iters))
}
return 0;
}
@@ -185,7 +185,7 @@ sudo lspci -vv -s <bridge> | grep AtomicOpsCap # want Routing+
slot on CPU root ports (atomics fine), the rest behind the chipset (usually
not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
batch 4 is a millisecond-scale figure at x4 — compute-bound, not
interconnect-bound (full derivation: [[Dual-R9700-Cooperation]] §Bandwidth).
interconnect-bound (full derivation: §Bandwidth).
**Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
timeout) — one probe answers several design questions:**
@@ -216,11 +216,11 @@ reporting an outage (`gitea` vs `gitea-server`).
complex; x4 suffices for TP.
2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — the
per-device chain census, including both endpoints reading `ReqEn+`, is
[[Dual-R9700-Cooperation]]'s). Re-checked 2026-10-18 without root: both root
's). Re-checked 2026-10-18 without root: both root
ports report `current_link_width=8` at `32.0 GT/s`
(`/sys/bus/pci/devices/0000:00:01.{1,3}/current_link_width`), and the live
engine states direct P2P between the cards.
3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
3. llama.cpp `--split-mode layer` — no collectives at all (); costs the GGUF conversion plus paged attention /
continuous batching / spec decode.
4. Independent TP=1 replicas + routing proxy — cards never share one model.
@@ -230,7 +230,7 @@ is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
`CUSTOM_ALL_REDUCE_MAX_SIZES` in
`vllm/distributed/device_communicators/all_reduce_utils.py`, whose keys on
today's `main` are exactly `"9.0"`, `"10.0"`, `"10.3"`, `"10.7"`
([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py),
([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py)
read 2026-10-18); nothing keys on a ROCm-reported `12.0`. — and patching it is
insufficient because TP init's `broadcast`/`barrier`/logits `all_gather` have
no non-RCCL path. A
@@ -249,6 +249,6 @@ kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
## Related
- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
- the layer-split path around the wall
- / — measured history
- [[frank-archon]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md
-530
View File
@@ -1,530 +0,0 @@
---
type: reference
title: Ethernet Bandwidth Math
description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-17
---
# Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17)
Everything in this section was measured on the live deployment
([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on
both ends. The theoretical tables below use H=4096/L=32; the real model
is larger (see constants), which raises the per-token bytes but not the
conclusions.
### 0.1 Verified constants (from the GGUF + env, not guessed)
| Constant | Value | Source |
| -------- | ----- | ------ |
| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) |
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured |
| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured |
| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured |
The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more
jitter (mdev 0.334 vs 0.211 ms).
### 0.2 Measured wire bytes (the number that matters)
Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**.
The *measured* bytes per position per direction are ~4x that:
| Direction | Measured | Raw hidden |
| --------- | -------- | ---------- |
| head -> worker | 39.8 KB/position | 10.0 KB |
| worker -> head | 38.4 KB/position | 10.0 KB |
The extra ~30 KB/position is the serialized RPC graph (per-position
nodes for all 45 worker blocks: attention, KV append, SSM ops)
plus protocol framing. It scales with position count just like the
payload, so it does not shrink away.
Per ubatch (256 positions) the measured crossing is **20.5 MB**
(10.4 MB head->worker + 10.1 MB worker->head).
### 0.3 Measured throughput and the validated model
Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE
path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition:
| Term | 2.5 GbE (measured anchor) |
| ---- | ------------------------- |
| wire per ubatch (20.5 MB) | 70 ms |
| compute + overhead per ubatch (residual) | 489 ms |
| ubatch total | 559 ms = 2.18 ms/token |
Using that residual and only swapping the wire time per link speed:
| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty |
| ---- | ----------- | ----------------- | ------------ | -------------------- |
| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% |
| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% |
| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% |
Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**.
### 0.4 The KV snapshot: where the link really saturates
On task release, `llama-server` saves the conversation to the prompt
cache. The KV for the 11 FA layers living on the worker GPU must be
copied **worker -> head** for that snapshot, and the slot is BLOCKED
while it runs (`PARALLEL=1` = one user queues behind it).
Measured for a 98,100-token conversation: 3,454 MB copied (model check:
98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg,
308 MB/s peak).
| Link | snapshot duration (slot blocked) |
| ---- | -------------------------------- |
| 1 GbE | ~31 s |
| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") |
| 10 GbE | ~5 s |
This is the only phase that runs the link at ~100% utilization for a
long time. Prefill runs at 12-26% of line rate in bursts (per ubatch)
and ~6-16% on average; decode runs at ~2% or less.
### 0.5 Decode (MTP verify step, 5 positions)
| Link | net time per step | share of 164 ms step | decode t/s |
| ---- | ----------------- | -------------------- | ---------- |
| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 |
| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) |
| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 |
Decode is latency-touched but bandwidth-trivial. Average wire rate
~19 Mbit/s in each direction.
### 0.6 Verdict for the 10 GbE USB-C adapter question
- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for
free: the claw's native 2.5 GbE `eno1` now carries the RPC path
(the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the
path, the RPC TCP re-attached on the new interface).
- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking.
It requires the L2 device at 192.168.0.1 to be 10G-capable, or a
direct cable between the two machines (dedicated link, second IP
pair, or a lower-metric static route).
- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable.
## 1. Bandwidth Math: Bytes per Token per Layer Boundary
### 1.1 Pipeline Parallelism (PP)
During autoregressive generation, each transformer layer produces a hidden state of shape `(batch_size, hidden_size)`. At a pipeline stage boundary, this tensor must be communicated between nodes.
**Forward pass only (inference):**
```
bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes]
```
**Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length.
**For a 27B dense model (e.g., Qwen3.6-27B):**
- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures).
Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120
(GGUF-verified); see section 0, which supersedes these theoretical
numbers with measurements.
- Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok`
**For a 36B MoE model (e.g., Qwen3.6-35B-A3B):**
- Hidden size: H = 6144 (common for this scale)
- Layers: L = 48 (typical)
- Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok`
**Full example — 27B model, 2 PP stages (1 boundary), batch=1:**
```
1 boundary × 4096 × 2 bytes = 8,192 bytes/tok
```
**Example — same model, batch=64:**
```
1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok
```
**Example — 36B model, 2 PP stages, batch=1:**
```
1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok
```
### 1.2 Tensor Parallelism (TP) — for comparison
In tensor parallelism, each MLP and attention layer performs an all-reduce on the hidden state:
```
bytes_per_layer = batch_size × hidden_size × 2 [one all-reduce per layer]
total_bytes_per_tok = L × batch_size × hidden_size × 2 [L all-reduces total]
```
**For 27B, H=4096, L=32, TP=2, batch=1:**
```
32 layers × 4096 × 1 × 2 bytes = 262,144 bytes = 256 KB/tok
(16 KB per layer all-reduce × 32 layers)
```
**For 36B, H=6144, L=48, TP=2, batch=1:**
```
48 layers × 6144 × 1 × 2 bytes = 589,824 bytes = 576 KB/tok
(12 KB per layer all-reduce × 48 layers)
```
### 1.3 Scaling with batch size
| Model | H | L | tok/s | Batch | PP boundary bytes/tok | TP total bytes/tok |
|-------|-----|-----|-------|-------|----------------------|--------------------|
| 27B | 4096 | 32 | 10 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 50 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 100 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 10 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 50 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 100 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 10 | 64 | 512 KB | 16,384 KB |
| 27B | 4096 | 32 | 50 | 64 | 512 KB | 16,384 KB |
| 27B | 4096 | 32 | 100 | 64 | 512 KB | 16,384 KB |
| 36B | 6144 | 48 | 10 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 50 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 100 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 10 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 50 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 100 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 10 | 64 | 768 KB | 36,864 KB |
| 36B | 6144 | 48 | 50 | 64 | 768 KB | 36,864 KB |
| 36B | 6144 | 48 | 100 | 64 | 768 KB | 36,864 KB |
---
## 2. Required Bandwidth vs. Ethernet Capabilities
### 2.1 Ethernet link capacities
| Link | Raw Rate | Usable TCP (approx.) | MB/s usable |
|------|----------|---------------------|-------------|
| 1 GbE | 1,000 Mbit/s | ~940 Mbit/s | ~118 MB/s |
| 2.5 GbE | 2,500 Mbit/s | ~2,350 Mbit/s | ~294 MB/s |
| 10 GbE | 10,000 Mbit/s | ~9,400 Mbit/s | ~1,175 MB/s |
### 2.2 Required bandwidth (Mbps) by model, batch, and throughput
**27B model (H=4096), 1 PP boundary:**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 0.13 Mbit/s | 2.05 Mbit/s | 8.19 Mbit/s |
| 50 | 0.65 Mbit/s | 10.24 Mbit/s | 40.96 Mbit/s |
| 100 | 1.31 Mbit/s | 20.48 Mbit/s | 81.92 Mbit/s |
**36B model (H=6144), 1 PP boundary:**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 0.20 Mbit/s | 3.07 Mbit/s | 12.29 Mbit/s |
| 50 | 0.98 Mbit/s | 15.36 Mbit/s | 61.44 Mbit/s |
| 100 | 1.97 Mbit/s | 30.72 Mbit/s | 122.88 Mbit/s |
**27B model, TP (L=32 layers all-reduce):**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 2.05 Mbit/s | 32.77 Mbit/s | 131.07 Mbit/s |
| 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s |
| 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s |
### 2.3 Feasibility matrix
Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).
**27B model — PP (1 boundary), batch=1:**
| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) |
|---|---|---|---|
| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s |
| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s |
| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s |
**27B model — PP (1 boundary), batch=16:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
**27B model — PP (1 boundary), batch=64:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s |
| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s |
| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s |
**27B model — TP (32 layers), batch=1:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
**27B model — TP (32 layers), batch=16:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s |
| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s |
| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s |
**27B model — TP (32 layers), batch=64:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s |
| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s |
| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s |
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE**
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s ≈ 2.5 GbE usable (2500 Mbit/s) — ~52% saturated**
**Key finding:** PP is feasible across all ethernet classes for all batch sizes up to at least 100 tok/s. TP becomes bottlenecked on 1 GbE at batch=64 and 100 tok/s.
---
## 3. Why PP is the Only Realistic Option over Ethernet
### 3.1 Communication cost comparison
| Scheme | Communication per tok per rank | 27B, H=4096, batch=1 | 36B, H=6144, batch=1 |
|--------|------|---|---|
| PP (Nₚₚ boundaries) | Nₚₚ × H × 2 | Nₚₚ × 8 KB | Nₚₚ × 12 KB |
| TP (L layers all-reduce) | L × H × 2 | 32 × 8 KB = 256 KB | 48 × 12 KB = 576 KB |
PP communication per token is **L/Nₚₚ times smaller** than TP communication per token (factor of ~16× for 2 PP stages).
### 3.2 Latency-bound nature of autoregressive generation
During autoregressive generation, each layer computes one token at a time. In TP, every layer requires an all-reduce (2× ring latency or 1× tree latency per layer). For L=32 layers, that's 32 synchronization points per token.
In PP, activations cross the stage boundary only once per layer (or L/Nₚₚ times). The critical path is:
```
Stage 0 computes → send hidden state across boundary → Stage 1 computes → ...
```
This is a single bottleneck per token, with latency proportional to network RTT × number of stages, not L.
### 3.3 Bandwidth utilization
With PP, each boundary crossing transfers `H × 2` bytes. For 8 KB at 1 GbE, the transfer completes in ~68 µs. This is negligible compared to compute time (milliseconds per layer on a GPU). The link is underutilized.
With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 KB total at 1 GbE, that's ~2 ms of pure transfer time — potentially comparable to compute time on a fast GPU.
### 3.4 Real-world evidence
- Petals uses PP exclusively over the public internet and achieves usable inference at 1–2 tok/s even with 100 Mbit/s links [1].
- llama.cpp RPC splits layers across nodes in pipeline order, not tensor sharding [2].
- exo uses TP but achieves speedup only with RDMA over Thunderbolt (~50 Gbit/s) [3].
- Petals paper Table 3: inference performance does **not depend much on bandwidth** (1 Gbit/s vs 100 Mbit/s gives nearly identical results) but **degrades significantly with higher latency** [1].
**Conclusion:** For gigabit-class ethernet, PP is the only practical multi-node parallelism strategy for inference. TP can work on 10 GbE for small batches but saturates at 1 GbE with moderate batch sizes.
---
## 4. Real-World Multi-Node Inference Benchmarks
### 4.1 Summary table
| System | Model | Hardware | # Nodes | Network | tok/s (batch=1) | Notes |
|--------|-------|----------|---------|---------|-----------------|-------|
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 1 Gbit/s, <5 ms RTT | 1.71 | PP, 22 stages, quantized [1] |
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, <5 ms RTT | 1.66 | ~3% slowdown vs 1 Gbit/s [1] |
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, 100 ms RTT | 1.23 | Latency matters more than BW [1] |
| Petals (wan) | BLOOM-176B | 14 consumer GPUs | 14 | Real internet, 100–1000 Mbit/s | 0.83 | Europe+N. America [1] |
| Petals | Llama 2 70B | consumer GPUs | various | Internet | up to 6 | Interactive chat [4] |
| Petals | Falcon 180B | consumer GPUs | various | Internet | up to 4 | Interactive chat [4] |
| llama.cpp RPC | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | Ethernet (2.5 GbE) | ~20 | Scales down with more nodes [3] |
| llama.cpp RPC | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~25 | Scales down with more nodes [3] |
| llama.cpp RPC | Kimi K2 (1T) | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~30 | A32B sparse, 32B active [3] |
| Exo (RDMA) | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | 32 | TP, scales UP with more nodes [3] |
| Exo (RDMA) | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | ~30 | 671B dense TP [3] |
| vLLM PP | Llama-3-70B | 2× A100 80GB | 2 | NVLink + Ethernet | ~15 | PP across nodes, TP within [5] |
### 4.2 Detailed notes
**Petals [1]:**
- Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
- For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
- Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)
**llama.cpp RPC [2]:**
- Model weights and KV cache distributed across nodes proportionally to memory
- Default: pipeline-style distribution of layers (first layers on node A, last on node B)
- Each node runs `ggml-rpc-server`, main host uses `--rpc` flag
- RDMA transport supported (RoCEv2), negotiated automatically with libibverbs
- Over ethernet, performance decreases with more nodes (each extra node adds a hop)
- Over RDMA/TB5, performance scales up with more nodes (less latency per hop)
**Exo [3]:**
- Uses tensor parallelism with MLX distributed backend
- With RDMA over Thunderbolt 5: up to 1.8× speedup on 2 devices, 3.2× on 4 devices
- Topology-aware auto-parallelism considers latency/bandwidth between links
- Thunderbolt 5 provides ~50–60 GbE real-world throughput (RDMA)
**Key insight from all systems:** Ethernet (1 Gbit/s) is sufficient for PP inference of 20–40B models because per-token activation traffic is small (8–12 KB/boundary). The bottleneck is **latency** (RTT between nodes), not bandwidth.
---
## 5. Overheads
### 5.1 TCP/IP overhead
| Overhead | Amount |
|----------|--------|
| Ethernet header + trailer | 18 bytes (14 + 4 FCS) |
| IPv4 header (no options) | 20 bytes |
| TCP header (no options) | 20 bytes |
| Total per packet | ~58 bytes overhead |
| MTU (jumbo frames 9000) | 9000 bytes payload → 0.6% overhead |
| MTU (1500 standard) | 1500 bytes payload → 3.9% overhead |
**Practical impact:** For 8 KB transfers (27B PP boundary), use jumbo frames where possible to minimize TCP overhead. For small transfers (< 1.5 KB), a single TCP packet is used and the overhead is 58/8192 = 0.7% per transfer — negligible.
### 5.2 RPC/serialization overhead
- llama.cpp RPC: Protobuf serialization adds ~10–20% overhead for small tensors
- Petals: PyTorch tensor serialization over hivemind/gRPC, adds serialization time
- vLLM PP: gRPC with protobuf, similar overhead
- Serialization scales with tensor size: negligible for 8 KB+ tensors
### 5.3 Pipeline bubble overhead
With Nₚₚ pipeline stages and microbatching, the pipeline bubble fraction is:
```
bubble_fraction = (Nₚₚ - 1) / (Nₚₚ + microbatches - 1)
```
For 2 stages, 1 microbatch (single tok): bubble = 1/2 = 50% idle time.
For 2 stages, 8 microbatches: bubble = 1/9 = 11%.
In autoregressive generation with batch=1 and no microbatching, each stage processes one token sequentially:
- Stage 0 finishes, sends hidden state, waits for Stage 1
- Stage 1 processes, returns result, Stage 0 processes next token
- Effective utilization ≈ 1/Nₚₚ if no pipelining overlap
**Mitigation:** Process multiple tokens before waiting. Send next token to Stage 0 while Stage 1 is computing — pipelining reduces bubble.
### 5.4 Per-hop LAN latency
| Network | Typical RTT | One-way latency |
|---------|------------|-----------------|
| 1 GbE (Cat6) | 30–50 µs | 15–25 µs |
| 2.5 GbE (Cat6) | 30–50 µs | 15–25 µs |
| 10 GbE (Cat6a) | 20–30 µs | 10–15 µs |
| RoCEv2 (IB/25GbE) | 1–5 µs | <5 µs |
| RDMA (Thunderbolt 5) | 10–30 µs round-trip end-to-end | — |
**Impact on PP inference:** With Nₚₚ = 2 stages and batch=1, one token takes:
```
t_compute + t_roundtrip + t_compute
```
At 30 µs RTT and ~2 ms layer compute time per GPU: the network adds ~30 µs, which is 1.5% of total. Negligible at LAN speeds.
With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute time.
### 5.5 Quantization overhead
- Petals uses dynamic blockwise 8-bit quantization: halves hidden state bandwidth from 16-bit to 8-bit [1]
- For 27B: 8 KB → 4 KB per boundary with 8-bit quantization
- Accuracy impact: <1% on standard benchmarks [1]
- Quantization/dequantization adds ~1–5 ms per stage on modern GPUs (negligible vs. compute)
---
## 6. Conclusions and Recommendations
### For the 27B / 36B models on gigabit-class ethernet:
1. **1 GbE is sufficient for PP inference** up to at least 100 tok/s at batch=64 (81.92 Mbit/s required, well under 1000 Mbit/s capacity). Even with TCP/IP and RPC overheads, margin is >90%.
2. **Pipeline parallelism is the correct choice** over ethernet — PP sends 8–12 KB per boundary vs. 256–576 KB for TP, a 32× advantage for 2 PP stages.
3. **Latency matters more than bandwidth** — the Petals paper demonstrates that dropping from 1 Gbit/s to 100 Mbit/s only reduces throughput by ~3%, while increasing RTT from 5 ms to 100 ms reduces it by ~28%.
4. **With 8-bit quantization**, bandwidth requirements halve, providing additional safety margin.
5. **Jumbo frames recommended** to minimize TCP overhead on larger transfers.
6. **For batch sizes ≥ 16**, 1 GbE remains viable for PP but consider 2.5 GbE for headroom and faster model loading.
---
## References
[1] Borzunov et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL 2023. arXiv:2209.01188. Available at: https://arxiv.org/pdf/2209.01188 — Primary benchmark source: Table 3 shows inference throughput at various bandwidths and latencies for BLOOM-176B. Key finding: inference insensitive to bandwidth, sensitive to latency. Recommended 25 Mbit/s bidirectional bandwidth.
[2] ggml-org/llama.cpp RPC documentation. Available at: https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md — Architecture for distributed LLM inference via RPC. Supports RDMA transport (RoCEv2), automatic negotiation, local tensor caching.
[3] exo-explore/exo README. Available at: https://github.com/exo-explore/exo — Multi-node LLM inference on Apple Silicon with TP and RDMA over Thunderbolt 5. Benchmarks with Qwen3-235B, DeepSeek v3.1, Kimi K2 on 4× M3 Ultra Mac Studio.
[4] Petals chatbot demo and health monitor. Available at: https://chat.petals.dev, https://health.petals.dev — Real-world performance of Llama 2 70B (up to 6 tok/s) and Falcon 180B (up to 4 tok/s) on public swarm.
[5] vLLM pipeline parallel documentation. Available at: https://docs.vllm.ai/en/stable/dev/engine/llm_engine.html — vLLM supports PP across nodes using gRPC with CUDA-graph support for low-latency stage transitions.
---
## Appendix: Full bandwidth formulae
### Pipeline Parallelism (generation, batch=1)
```
bytes_per_boundary_per_tok = hidden_size × 2 [fp16]
total_bytes_per_tok = num_boundaries × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
```
### Pipeline Parallelism (generation, batch=N)
```
bytes_per_boundary_per_tok = hidden_size × N × 2
total_bytes_per_tok = num_boundaries × hidden_size × N × 2
```
### Tensor Parallelism (generation, batch=1)
```
bytes_per_layer_allreduce = hidden_size × 2
total_bytes_per_tok = num_layers × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
```
### Tensor Parallelism (generation, batch=N)
```
total_bytes_per_tok = num_layers × hidden_size × N × 2
```
### With 8-bit quantization (like Petals)
```
# Simply divide all fp16 byte counts by 2
bytes_per_boundary_per_tok = hidden_size × 1 [int8]
```
serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md
-97
View File
@@ -1,97 +0,0 @@
---
type: reference
title: Expected Performance
description: Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors.
tags:
- pipeline-parallelism
- performance
- estimate
timestamp: 2026-08-14
---
# Expected Performance
This page estimates the speed of a 25-40B model split across two machines
over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
and the measured numbers in [[Mixed-Vendor Experiments]].
## The Bandwidth Ceiling Is Not the Limit
A pipeline boundary moves only the hidden state. For a 27B model (hidden
5120), that is 10240 bytes per token per boundary in fp16. Even at 64
tokens per second with batch 64, the traffic is about 262 Mbit/s. 1 GbE can
carry it. The real limits are latency, RPC overhead, and the slowest node.
The Petals paper measured this directly. Dropping from 1 Gbit/s to
100 Mbit/s changed throughput by 3 percent. Raising round-trip latency from
5 ms to 100 ms dropped throughput by 28 percent. Latency matters more than
bandwidth for pipeline inference.
## Real Benchmark Anchors
| Experiment | Model | Network | Decode (t/s) |
| ---------- | ----- | ------- | ------------ |
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-72B Q4 | 10 GbE | 6 |
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-7B Q4 | 10 GbE | 53 |
| RTX 4070 + MI50 + 3050 (CUDA+ROCm) | Qwen3-VL-30B Q5 | 1 GbE | 51.5 clean, 17.8 at 32k ctx |
| Two Strix Halos (ROCm RPC) | MiniMax-M2.5-139B Q8 | USB4 | 15.35 |
| AMD four-node Strix Halo (5 GbE) | Kimi K2.5 (1T) | 5 GbE | 8.30 |
| Tesla V100 + R9700x2 (Vulkan, one box) | Qwen3.6-35B-A3B Q8 | PCIe | 62 |
Pattern: RPC decode runs about 2-5 times slower than the best single node.
Prefill benefits from more compute across nodes. The 30B-class model over
1 GbE with warm cache reached 51.5 t/s on a short context and 17.8 t/s at
32k context.
## Estimate for Qwen3.6-27B Q8, Two Nodes
Scenario: RTX 3090 (24 GB VRAM, NVIDIA desktop) plus Strix Halo NUC
(Radeon 8060S, 30 GB physical RAM). The model is the dense 27B at Q8
(29-32 GB). Each node holds part of the weights and KV cache.
| Link | Decode (t/s) | Prefill (t/s) | Notes |
| ---- | ------------ | ------------- | ----- |
| 1 GbE | 8-15 | 200-500 | Latency-bound; per-token RPC round trip |
| 2.5 GbE | 10-18 | 250-600 | Some headroom; latency still dominant |
| 10 GbE | 12-20 | 300-800 | Closest to the published 10 GbE runs |
The dense 27B decodes slower per token than the MoE 30B A3B class. Expect
the lower half of the ranges at full context.
### With Speculative Decoding
Speculative decoding (MTP, DFlash, or ngram-mod) raises effective decode.
Tightwad measured 1.86x on a slow RPC pool. Estimate the effective rate at
1.4-1.8x the raw RPC decode:
| Link | Raw decode (t/s) | With spec (t/s) |
| ---- | ---------------- | --------------- |
| 1 GbE | 8-15 | 12-24 |
| 2.5 GbE | 10-18 | 15-28 |
| 10 GbE | 12-20 | 18-32 |
## Context That Fits
The two real machines give about 51 GiB of usable memory combined
(24 GiB VRAM plus about 27 GiB on the Strix Halo, which has 30 GiB physical
RAM; the 96 GiB driver ceiling is not physical). A 29-32 GB model leaves
about 20-22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85-100
KiB per token with q8_0 K and f16 V (measured on the large service at
602100 tokens = about 50 GB). That budget gives a total context pool of
roughly 200k-260k tokens across both nodes.
## Caveats
- Every node must run the exact same llama.cpp build. Version skew hangs
the handshake.
- The slowest node sets the pace for every token.
- Model load crosses the network once per start. Expect minutes, not
seconds, for the split load.
- These are estimates. The exact combination of an RTX 3090 desktop and a
Strix Halo NUC has no published benchmark yet.
## See Also
- [[Ethernet Bandwidth Math]]
- [[Mixed-Vendor Experiments]]
- [[Sample Script Set]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "Mixed-Vendor Experiments"
description: "Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "Qwen3.6 27B Spec Methods"
description: "Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md
-313
View File
@@ -1,313 +0,0 @@
---
type: reference
description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX"
tags:
- llm
- pipeline
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
> [deployment.md](../deployment.md) for the active configuration.
>
> **2026-10-18: nothing pipeline-parallel runs on the fleet.** No RPC listener
> exists on either host (`ss -ltn` shows no :50052), every artifact named below
> is gone (§Files), and the worker hardware left the box: the RTX 3090 was
> removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
> runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
> inside one process ([[Dual-R9700-Cooperation]],
> [[frank-archon]]).
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
3090 (worker) using llama.cpp RPC. This page is the design record and the
morning-of runbook. Research background:
[Qwen3.8 Strix optimizations](../qwen3.8-strix-optimizations.md).
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
This split is the production `frank-dragoon` service. Current
state, verified against the live journal and output smoke tests
(2026-08-17, load 02:06):
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
(3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
zero worker impact — measured, not assumed).
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
layer weight. Maximum for this config; see "Split limits" below.
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
* MTP: standalone draft head, acceptance 0.83 then 0.75, mean len 4.0-4.3,
per-position ~0.86/0.71/0.71/0.71 — stronger at every position than the old bundled
head, which measured 0.739/0.554/0.427/0.340.
- Spec chain as deployed: `--spec-draft-n-max 4` on the standalone draft, with
the `--spec-default` ngram-mod fallback at n_match 24 / n_min 48 / n_max 64
(the preset that survived the DSpark/DFlash retirements —
[[qwen3.8-spec-matrix]]).
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, `--gpu-layers all`. The
pi `models.json` declared 262144 (`contextWindow = maxTokens`, the house
convention of that era).
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
Vulkan).
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
stable; all allocations are upfront).
- Unit `frank-dragoon.service`, launched through
`serve-pp-head.sh` + `frank-dragoon.env`.
## Service enablement (2026-08-17)
The PP pair is the only inference service enabled for boot. Both units
(`frank-dragoon` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-zealot` on Dankclaw) and all retired
units are disabled. Restart order: worker first, then head. The Dankserver
helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-zealot` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. The "deploy.sh dankserver is broken until retargeted" note is long fixed:
`./scripts/deploy.sh dankserver --service frank-archon` is the documented
invocation today (`scripts/deploy.sh:38`, dispatch case
`dankserver:frank-archon` at `scripts/deploy.sh:149`). See
[[deployment]] for the full boot-policy table.
## FA-all-quants rebuild (2026-08-17)
Flash Attention with mismatched K/V cache types requires the
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
worker (CUDA sm_86) compile the same sources, so the same CMake flag
covers both:
cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j --target llama-server # dankclaw
cmake --build build -j --target ggml-rpc-server # dankserver
Both hosts at commit 9b05354ec (b10433). The flag was already cached
ON in the worker build (Aug 16 objects carry it); the head required a
full ggml-hip rebuild. Keep both binaries on the same commit — version
skew hangs the RPC handshake.
## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
(f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
9408 -> OOM by ~0.85 GiB.
The KV-per-device count = round(f x 66) layers, independent of the
weight boundary (same rule as the Q8_K_XL config). FA blocks at
3,7,...,63.
## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
With the pool shrunk to 262144, the KV per full-attention layer drops to
exactly 1024 MiB, freeing headroom. Measurements:
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
free. Healthy (2026-08-17 00:29 load).
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
cudaMalloc OOM by ~740 MiB.
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
The KV-per-device count rounds the split fraction over 66 layers
(round(f x 66)), independent of the weight boundary; the 9-FA flip
sits at f ~= 0.538, just below the weight threshold for block 35, so
the 35-block config is unreachable.
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
includes the MTP head for weights and token_embd for KV — the two
boundaries diverge, which is why 0.54 split them apart.
## Cache incident (2026-08-17): garbage output from a full worker disk
Symptom: after a split change, the service loaded "healthy" but every
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
content, degenerate logits). Root cause chain:
1. The worker RPC tensor cache keys per session. Each model load writes
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
4434 files / 96 GB.
2. The Dankserver root disk filled to 100%. Cache writes started
failing, leaving 0-byte entries. A subsequent reload hit corrupt
cache entries and the worker served garbage weights for the
newly-needed layers without validation.
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
worker, free the disk, reload the head (full re-stream ~2.5 min).
A clean cache and a healthy disk produced correct output immediately
(acceptance 0.89 on the first smoke test).
Guard was `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap). That
script is gone from the repo and from the host, yet the crontab line still
runs every 10 minutes against it
(`*/10 * * * * /home/dank/vLLM/scripts/prune-rpc-cache.sh`, `crontab -l`, read
2026-10-18 — the target does not exist, so it fails silently). Never let the
cache directory grow unchecked; a full disk silently corrupts it. Signature to
recognize: healthy load + degenerate repeated-token output.
## Topology
```
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
llama-server b10433 ggml-rpc-server b10433
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
model file (31.5 GB) -d CUDA0 -c (tensor cache)
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0-44 (45 of 65 blocks)
Strix: layers 45-64 + lm_head
hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
measured wire bytes ~40 KB/position/direction (serialized graph
dominates); 20.5 MB per ubatch both directions. Details:
[[Ethernet Bandwidth Math]] section 0.
```
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
the worker, the rest on the head, in order. One hidden-state handoff per
token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5%
of step time). PREFILL and the prompt-cache KV snapshot ARE
bandwidth-bound: the link runs at line rate for the wire burst of every
ubatch and ~100% during the snapshot copy. Measured numbers:
[[Ethernet Bandwidth Math]] section 0.
- **Why the head is on Dankclaw:** the model file is already there; the
worker streams its tensor share at load (no 31.5 GB copy to the server).
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
worker build lived in a worktree `/home/dank/llama-cpp-rpc` on Dankserver so the
production b10121 build stayed untouched (that path no longer exists, read
2026-10-18). A version skew hangs the RPC handshake with no error.
## Files
| File | Machine | Purpose |
|------|---------|---------|
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
| `scripts/frank-dragoon.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
| `scripts/frank-dragoon.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
**Every PP file in that table except `scripts/frank-dragoon.env` no longer
exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
`serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
`pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
`scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
head env ([[frank-dragoon]]).
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
1. **No `--device` on the head.** Setting `params.devices` makes
`llama_prepare_model_devices` use exactly that list; the RPC device is
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
puts RPC first.
2. **The iGPU side is ROCm by default.** Backends register in fixed order
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off; removed — the
path is gone on 2026-10-18) — set `HEAD_BIN` in the env. Trade: ROCm side =
faster prefill, ~3% slower decode (measured on this box).
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
3090's 24 GB, so the default lands ~20/80 on the slow device. 1,1 = 50/50 =
balanced per-layer latency. The 2026-08-15 4B smoke test that proved it
(production untouched): at a 0.2/0.8 split only 4 of 36 layers reached the
3090, and at a real split worker VRAM rose ~819 MiB with 95 t/s prefill /
58.8 t/s decode on the tiny model; link RTT ~0.3–0.5 ms.
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
I/O.
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
the head before load (warm-cache loads crash fast; cold loads survive).
## Runbook (morning of 2026-08-16)
Run from any box with ssh access to both (repo path on claw:
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM` — both still exist as
stale checkouts; the deployment checkout today is `/home/dank/serverAppConfigs`
on both hosts).
```
# 0. one-time: pull the kit on both boxes
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
# 1. checks (read-only)
./scripts/pp-test.sh preflight
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
# or step by step:
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
./scripts/pp-test.sh verify # layer split + worker VRAM
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
./scripts/pp-test.sh bench 16 256 1
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
```
Config variants: edit `scripts/frank-dragoon.env` on the claw (git commit +
pull if the bench runs from the repo), then re-run `start` + `bench`:
- No spec baseline: `SPEC_NONE=1`
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
Verification checklist after each `start`:
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
1,1 split.
3. `nvidia-smi` on the server showed ~15–16 GB used by the worker
(production small service + worker share). No NVIDIA card is in that host
today — the equivalent check is `rocm-smi --showmeminfo vram`.
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
## Expected numbers
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
## Rollback
`pp-test.sh restore` starts both production services. If a PP test wedges a
box: `systemctl restart <unit>` after killing the stray processes
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
with a pattern matching your own ssh command line).
## Open questions for the test day
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
in the head log (`draft acceptance = X`).
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
serialized per compute — expect the 2-slot speedup to be weaker than
single-node. Measure, do not assume.
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
the ubatch crossing the RPC boundary is the suspect.
serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md
-261
View File
@@ -1,261 +0,0 @@
---
type: reference
title: Sample Script Set
description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.
tags:
- pipeline-parallelism
- sample
- llamacpp
- educational
timestamp: 2026-08-14
---
# Sample Script Set — Two-Node Qwen3.6-27B Q8 Split
> WARNING. This page is for educational reading only. Do not run these
> scripts. Do not deploy them. They are a template, not a tested product.
> The RPC backend is a proof of concept that the upstream README calls
> "fragile and insecure". Running it on a production network is unsafe.
## Goal
Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two
machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA
desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395,
Radeon 8060S). Use the largest context that fits on both nodes together.
## How the Split Works
```mermaid
graph LR
subgraph NodeA[NVIDIA Desktop — HEAD]
A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
end
subgraph NodeB[Strix Halo NUC — WORKER]
B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
end
A1 <-->|hidden states over ethernet| B1
```
Node A runs the head `llama-server`. Node B runs `ggml-rpc-server`, which
exposes its GPU as a remote device. The head splits weights and KV cache
across its own GPU and the remote device in proportion to free memory.
Each token generation passes the hidden state once across the boundary.
Key facts that shape the scripts:
- Both nodes must run the exact same llama.cpp build tag. Version skew
hangs the handshake.
- The head build needs its backend plus RPC. The worker build needs its
backend plus RPC. CUDA and ROCm do not mix inside one binary; each node
uses its own backend.
- Speculative decoding runs on the head. The draft model loads on the head.
- The model file must be present on every node with the same bytes.
## File Layout
```
pipeline-pp/
README.md
build.sh # build llama.cpp with RPC on each node
workers/worker-strix.sh # run ggml-rpc-server on the Strix Halo NUC
head/head-nvidia.sh # run llama-server on the NVIDIA desktop
head/frank-27b-pp.env # tunables for the head
head/verify.sh # health and split check
```
## Build Script
File: `build.sh`
```bash
#!/usr/bin/env bash
# Build llama.cpp with RPC support. Run ONE build per node, with that
# node's own backend enabled. Educational only. Do not run in production.
set -euo pipefail
LLAMA_DIR="$1" # path to a llama.cpp checkout
NODE="$2" # "nvidia" or "strix"
cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_RPC=ON \
$( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
|| echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
-DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
cmake --build "$LLAMA_DIR/build-pp" --config Release -j
# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli
```
Notes: the HEAD does not need a GPU if you offload everything to RPC
workers. Keep the head on the NVIDIA desktop so it can also use its local
3090 for part of the split.
## Worker Script (Node B, Strix Halo NUC)
File: `workers/worker-strix.sh`
```bash
#!/usr/bin/env bash
# Expose the Strix Halo GPU as an RPC device for a remote head.
# Educational only. Do not run on an unprotected network.
set -euo pipefail
BIN=/home/dank/llama.cpp/build-pp/bin
# Bind to the LAN address. The default binds 127.0.0.1 only.
HOST=192.168.0.143 # the Strix Halo NUC address
PORT=50052
# -c enables a local model-file cache so the split load runs once, not per boot.
exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
# Use --device to name the backend device explicitly if auto-detect fails:
# ggml-rpc-server --host $HOST --port $PORT --device gfx1151
```
Security note: `ggml-rpc-server` exposes all memory to any client that can
reach the port. Keep it on a trusted LAN. Do not bind it to the public
internet.
## Head Tunables (Node A, NVIDIA Desktop)
File: `head/frank-27b-pp.env`
```env
# Educational tunables for the two-node split. Do not deploy.
MODEL_DIR=/home/dank/models
MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf # ~32 GB
# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
MODEL="$MODEL_DIR/$MODEL_FILE"
# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
RPC_SERVERS=192.168.0.143:50052
# Context. 262144 is the native max for this model. Drop to 200000 if
# memory pressure is too high. This is the single-slot ceiling.
CTX_SIZE=262144
PARALLEL=1
# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16
# Generation batching. Decode is latency-bound over RPC; keep batch modest.
BATCH_SIZE=1024
UBATCH_SIZE=512
# Compute. Match the physical core count on the head box.
THREADS=7
# Sampling (HauhauCS thinking-general defaults).
TEMPERATURE=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0
```
## Head Script (Node A, NVIDIA Desktop)
File: `head/head-nvidia.sh`
```bash
#!/usr/bin/env bash
# Run the split model head on the NVIDIA desktop.
# Educational only. Do not run in production.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-27b-pp.env"
BIN=/home/dank/llama.cpp/build-pp/bin
PORT=7770
RPC_ARGS=()
for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done
# Pick ONE spec variant by uncommenting the matching pair below.
# --- Variant 1: MTP (recommended by research) ---------------------------
# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"
# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"
# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"
# --- Variant 4: no speculation --------------------------------------------
SPEC="--spec-type none"
exec "$BIN/llama-server" \
--model "$MODEL" \
"${RPC_ARGS[@]}" \
${SPEC} \
--ctx-size "$CTX_SIZE" \
--parallel "$PARALLEL" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--threads "$THREADS" \
--host 0.0.0.0 --port "$PORT" \
--alias frank-27b-pp
```
### Spec Variant Summary
| Variant | draft model file | --spec-type | Notes |
| ------- | ---------------- | ----------- | ----- |
| MTP | `Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B) | `draft-mtp` | Parallel-safe, best in benchmarks |
| DFlash | `Qwen3.6-27B-DFlash-Q8_0.gguf` | `draft-dflash` | Known premature-EOS bug |
| ngram-mod | none (built-in) | `ngram-mod` | Zero VRAM cost |
| DSpark | vLLM only | (not in llama.cpp) | Use vLLM PP instead; see below |
DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If
you want DSpark, use vLLM pipeline parallelism, which has its own limits
(no spec decode with PP as of 0.15.x). The llama.cpp path here covers
MTP, DFlash, and ngram-mod.
## Verify (Node A)
File: `head/verify.sh`
```bash
#!/usr/bin/env bash
# Check the split model is up and that both nodes are in the split.
set -euo pipefail
PORT=7770
# Health must return "ok" after model load (minutes, not seconds).
curl -sf "http://127.0.0.1:$PORT/health"
echo
# Show the device split from the server log.
curl -sf "http://127.0.0.1:$PORT/props" | head -40
# Expected: the model and KV cache distributed across CUDA0 (node A)
# and the remote RPC device (node B).
```
## Context Sizing Math
The model is about 29 to 32 GB. The two machines give about 51 GiB of
usable memory. That leaves about 20 to 22 GiB for KV cache. The
Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K /
f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens.
A single 262144-token slot is the practical ceiling for this split.
To run more context, shrink the model quantization. Q6_K (about 24 GB)
frees several GB for KV. Do not try to run the 96 GiB driver figure on the
Strix Halo as real memory. It is an overcommit ceiling; the physical RAM
is 30 GiB.
## What This Does Not Cover
- The remote draft-model-over-RPC path is not confirmed. The draft stays
on the head.
- `--tensor-split` is available to force a manual layer split. The sample
uses the automatic proportional split.
- RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few
percent over TCP.
## See Also
- [[llama.cpp RPC]]
- [[Expected Performance]]
- [[Ethernet Bandwidth Math]]
- [[Qwen3.6 27B Spec Methods]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md
-70
View File
@@ -1,70 +0,0 @@
# Pipeline Parallelism
This section covers running one large model across two machines. It focuses
on pipeline parallelism (layer split) over ethernet. The research answers
what speeds are possible and which tools work. The sample scripts show a
two-node Qwen3.6-27B Q8 split for reading only.
## Terminology
Pipeline parallelism (PP) splits model layers across machines. Each machine
computes its layers and passes hidden states to the next machine. Tensor
parallelism (TP) splits each layer across machines and requires an
all-reduce per layer. PP is the only realistic option over ethernet.
Speculative decoding is a separate feature. It uses a small draft model (or
n-grams) to propose tokens that the main model verifies in one pass. It can
run together with PP in llama.cpp.
## The Two Runtimes
```mermaid
graph LR
subgraph llamaRPC[llama.cpp RPC]
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
end
subgraph vllmPP[vLLM PP]
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
end
```
- [[llama.cpp RPC]] — the working, production-capable path for two
heterogeneous nodes over ethernet.
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
multiprocessing and does not compose with speculative decoding yet.
## What the Research Found
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
tokens per second on paper; the real limit is latency and RPC overhead.
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
published numbers yet.
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
## Deployment
- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
`--rpc`, default split is proportional to reported free memory (pin 1,1),
the iGPU side defaults to the ROCm backend.
## Sample Scripts
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
the largest context that fits. For educational reading only. Do not run
it in production.
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[frank-zealot]]
- [[Dankserver System Specs]]
- [[Dankclaw System Specs]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "llama.cpp RPC"
description: "llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "vLLM Pipeline Parallelism"
description: "vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Quantization Production.md
+22 -12
View File
@@ -1,3 +1,13 @@
---
type: "howto"
title: "Quantization Production"
description: "Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file."
tags:
- llm
- quantization
timestamp: 2026-10-19
---
# Quantization Production — making our own quants, per card and runtime
**Research page (2026-10-08).** Question: how do WE produce quantizations
@@ -73,18 +83,18 @@ Laws that steer everything here:
### Ablation / abliteration (directional ablation)
- Arditi et al. 2024, arXiv 2406.11717: refusal is mediated by a single
direction; orthogonalizing weights against it removes refusals with a
rank-1 edit. Open implementations: `p-e-w/heretic` (automated, above),
rank-1 edit. Open implementations: `p-e-w/heretic` (automated, above)
Artio/abliteration-toolkits, `EleutherAI/repeng`-style rep-overlap
variants **[verified (tool existence: heretic; rest survey)]**.
- Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm
weights, so it composes with any expert quantization.
- Fleet status: frank-dragoon runs an abliterated base **by Frank's choice**
(AGENTS: censored-base refusals were the accepted trade of the 27B era);
(AGENTS: censored-base refusals were the accepted trade of the 27B era)
frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.
### HauhauCS QX_K_P "Perfect" quants
- Our frank-zealot publisher. Method, from the publisher's own cards
**[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced"),
**[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced")
(2) quantize with **imatrix calibration on the ABLITERATED weights**,
(3) K_P = a **per-model, per-tensor mixed-precision profile** —
"model-specific analysis to selectively preserve quality where it
@@ -194,8 +204,8 @@ u2 draft — all recipe-expressible (see the w4dn8 runbook).
#### Recipe portability: the w4dn8 shape onto other MoEs (2026-10-08 audit)
The pin 1.0.13 container ships SIX arch plugins
(`/opt/radiance/share/radiance/architectures/`): `qwen4exp_fp8`,
**`qwen35moe_fp8`**, `qwen35_fp8`, `qwen35_bf16`, `llama_fp8`,
(`/opt/radiance/share/radiance/architectures/`): `qwen4exp_fp8`
**`qwen35moe_fp8`**, `qwen35_fp8`, `qwen35_bf16`, `llama_fp8`
`llama_dense_fp8` (census via `podman run --rm --entrypoint sh <image> -c`
— the image has no `ls`/`strings`; grep the `.so` files directly, and
the image lives in ROOTLESS storage as `docker.io/stilldeadcode/radiance`).
@@ -204,7 +214,7 @@ the image lives in ROOTLESS storage as `docker.io/stilldeadcode/radiance`).
calibration→GPTQ path exists for the qwen3.5-MoE family; GPTQ itself
lives in `rad-convert`, not the plugin (0 hits in both `.so`).
Worked example — `Qwen/Qwen3.6-35B-A3B` (config: `qwen3_5_moe_text`,
Worked example — `Qwen/Qwen3.6-35B-A3B` (config: `qwen3_5_moe_text`
40 layers, `full_attention_interval=4` DeltaNet hybrid, **256 experts
top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
248320, multimodal wrapper with vision tower):
@@ -248,13 +258,13 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA; Ampere
decode incl. 3090 documented in v1.0 performance write-ups).
- Ampere capability edges: vLLM FP8 docs ("only Hopper and Ada …
W8A8; Ampere supported for W8A16 weight-only utilizing Marlin"),
vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6),
W8A8; Ampere supported for W8A16 weight-only utilizing Marlin")
vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6)
TensorRT-LLM Support Matrix (FP8 and INT4-AWQ-with-FP8-acts need
SM≥89; NVFP4 Blackwell), SGLang issue #12887 (Ampere FP8-via-Marlin).
- Repo: `scripts/convert-swift15-w4dn8.sh`,
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
- Repo: `scripts/convert-swift15-w4dn8.sh`
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`
`scripts/build-kairic-edge-gfx1151.sh`; endpoints [[frank-archon]],
[[frank-dragoon]], [[frank-zealot]]; method [[qwen3.8-strix-optimizations]]
, ; method
and [[GGUF Compatibility]].
serverAppConfigs-docs/LLM/Quantization by Hardware Architecture.md
+1 -4
View File
@@ -132,15 +132,12 @@ quantizer exists for it ([[Quantization Production]]).
|---------|--------|-----|
| 1 | `.hgn` (deployed) | the halogen engine's own format; production door is closed |
| 2 | GGUF K/I quants, K_P blends | the only self-produced option on this card ([[Quantization Production]]) |
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: [[ling-3.0-flash-trial]] |
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: |
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[Shared Memory Architecture Dankclaw]]
serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md
-206
View File
@@ -1,206 +0,0 @@
---
type: reference
description: "vLLM uses CPU RAM and swap space to extend GPU memory capacity."
tags:
- llm
- vllm
timestamp: 2026-08-28
---
# RAM CPU Offloading in vLLM
## Overview
Stock vLLM offloads **KV pages**, not weights: the scheduler moves KV cache between GPU
and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the
tcclaviger/R4D fork ran a real **weight** offload (`--enable-expert-offload`, tiered
expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a
246 GiB box: [[RDNA4 MoE Expert Offload VRAM Budget]] and the tcclaviger era row of
[[frank-archon]]. First-hand in-repo evidence outranks a generic upstream claim about our
own hosts ([[Model Swap and Rollback]] §10).
> **Verified 2026-10-19 against `vllm/config/cache.py` on `main`:** there is no
> `swap_space` field — `--swap-space` is gone from current engine args. CPU KV offload is
> `--kv-offloading-size`, and it activates **only** together with `--kv-offloading-backend`
> (default `native`); both fields live in that one file. This page is the fleet's owner of
> that knob contract — other pages link here instead of restating it.
## How CPU Offloading Works
VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.
The KV cache swap mechanism works as follows:
1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space
## Key Parameters
### gpu_memory_utilization
This parameter controls the fraction of GPU memory used for KV cache.
| Value | Description |
|-------|-------------|
| 0.7 | Uses 70% of GPU memory for KV cache |
| 0.8 | Uses 80% of GPU memory for KV cache |
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
| 0.95 | Uses 95% of GPU memory for KV cache |
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
### swap_space (removed) / kv-offloading-size + kv-offloading-backend (current)
`--swap-space` no longer exists in current engine args (verified above); the live pair is
`--kv-offloading-size` with `--kv-offloading-backend`, and offload stays off until the size
is set. The table below is the historical `--swap-space` sizing guidance, kept as a sizing
reference only.
| Value | Description |
|-------|-------------|
| 0 | No CPU swap space |
| 4 | 4 GiB swap space (historical default) |
| 8 | 8 GiB swap space |
| 16 | 16 GiB swap space |
| 32 | 32 GiB swap space |
| 64 | 64 GiB swap space |
The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.
### max_model_len
This parameter sets the maximum context length.
| Value | Description |
|-------|-------------|
| auto | VLLM calculates optimal length |
| 2048 | 2048 tokens |
| 4096 | 4096 tokens |
| 8192 | 8192 tokens |
| 32768 | 32768 tokens |
Smaller values reduce memory usage but limit context length.
### max_num_batched_tokens
This parameter sets the maximum number of tokens per batch.
| Value | Description |
|-------|-------------|
| 2048 | Default for online serving |
| 4096 | Moderate batch size |
| 8192 | Large batch size |
| 16384 | Very large batch size |
### max_num_seqs
This parameter sets the maximum number of sequences per batch.
| Value | Description |
|-------|-------------|
| 128 | Default for online serving |
| 256 | Large batch |
### kv_cache_dtype
This parameter sets the KV cache data type.
| Value | Description |
|-------|-------------|
| auto | Uses model default dtype |
| fp8 | Uses FP8 for KV cache (reduces memory by 50%) |
| fp8_e5m2 | Uses FP8 E5M2 format for KV cache |
## Memory Configuration Examples
All three examples carry the retired `--swap-space` syntax and are kept as sizing
references; the live knob pair is named above.
### Small Model on Single GPU
```
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
--gpu-memory-utilization 0.9 \
--swap-space 4 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
### Large Model with CPU Offload
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.8 \
--swap-space 16 \
--max-model-len 8192
```
### Memory-Constrained GPU
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.7 \
--swap-space 32 \
--max-model-len 4096 \
--max-num-batched-tokens 2048
```
## Preemption Modes
VLLM uses two preemption modes when GPU memory is insufficient:
### Recompute Mode (Default in V1)
VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.
### Swap Mode
VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.
## Monitoring
Watch for preemption warnings:
```
WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.
```
If you see frequent preemptions:
1. Increase `gpu_memory_utilization`
2. Raise the KV offload budget (`--kv-offloading-size`, with `--kv-offloading-backend`)
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage
## Memory Conservation Tips
- Use quantization (FP8, INT8) to reduce model size
- Reduce `max_model_len` to match actual needs
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Set `--kv-cache-memory-bytes` to pin the KV budget instead of letting the profiler guess
(the flag our own `scripts/patches/gpu_worker-kv-cap.patch` reads)
- Limit multi-modal inputs with `limit_mm_per_prompt`
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| vLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| vLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| vLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| vLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
serverAppConfigs-docs/LLM/RDNA4 Container Serving.md
-258
View File
@@ -1,258 +0,0 @@
---
type: reference
title: RDNA4 Container Serving
description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.
tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
timestamp: 2026-10-05T00:00:00-07:00
---
# RDNA4 Container Serving
Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201 — card count,
per-card VRAM and host RAM are [[Dankserver System Specs]]'s) with
`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
with native MoE quant kernels and a first-class expert offload planner). The
tuning history and current deployed shape live in
[[frank-archon]]; this page is the bring-up law.
## 0. Know the box — and its phantom third GPU
`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
the box as 186.7 GiB. The third device is the 7700X's integrated Vega
(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
ships no code for that arch. **Never sum device `total_memory`; never count
the iGPU as headroom or a workload.**
## 1. Preflight: is the quant class EXECUTABLE, not merely present?
Expert offload is mandatory when weights exceed VRAM, and it requires the
checkpoint's experts class to have BOTH a compiled kernel AND an
`offload_recipe` ("a pinned expert has no host copy, so there is no
read-through fallback" — startup aborts otherwise). Check BEFORE downloading
anything:
```bash
podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
```
| quant | kernel in `r4d.so` | experts class | verdict |
|---|---|---|---|
| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
| RFA | none | no class at all | out |
Also confirm the architecture routes natively with NO overlay
(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
apply vendor overlays** against a fast-moving pre-release image — the breakage
misleads (a missing symbol in the overlay's OWN copy of a module masks the
real cause, then moves to another symbol each patch;
`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
`transformers_utils/config.py`). The image ships quantizers for
`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.
## 2. Fetch with verification, always
Gated repos: token never in argv (`pgrep -a curl` prints command lines).
Build a mode-600 curl config once:
```bash
{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
> ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
```
`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
the error body as a "successful artifact" (a gated repo once yielded three
153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
resumes torn shards; HTTP/2 cancels leave silently truncated files, so
re-verify EVERY shard's byte count against the Hub manifest (and the LFS
sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
law; [[frank-medium-model-swap]] has the same detail for GGUFs and
`.partNofM` fragments.
## 3. Serve — the known-good invocation and its traps
```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
--cap-add SYS_PTRACE \
--device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-v /home/dank/models/<model>:/models:ro \
$IMAGE /models --served-model-name <name> --port 7770 \
--tensor-parallel-size 2 --max-model-len 65536 \
--kv-cache-memory-bytes 2147483648 \
--max-num-seqs 4 --gpu-memory-utilization 0.95 \
--enable-prefix-caching --trust-remote-code
```
- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
first argument and do NOT pass `serve` (podman then tries to pull `serve`
as an image name). `-e VAR=1` must come BEFORE the image name.
- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
- `podman run -d` without `-a` sends container stdout nowhere — read
`podman logs <ctr>`; and vLLM writes `Application startup complete` to
**stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
fires.
- Start at 65536, not the native ceiling; prove the plan, then scale.
- Load takes ~5 min with `/health` refusing the whole time — do not kill the
process (deploy.sh waits 20 min).
## 4. Container GPU-init triage (ordered)
1. **Enumerate-vs-context trap (most common):** individual
`renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
nodes — it enumerates agents (`device_count()` returns N) and then refuses
every context: `RuntimeError: No CUDA GPUs are available` on the first
real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
/dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
never with device nodes.
2. **`device_count()` is not evidence.** Prove usability with a real op
(`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
`mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
cross-check `rocm-smi --showmeminfo vram`.
3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
larger than the limit fails as
`torch.AcceleratorError: CUDA error: out of memory` raised from
`pin_memory()` — it names the accelerator but the fault is host pinned
memory. It cannot be raised from inside a container or even from root's
own shell (`Operation not permitted`); only PAM applies it at login, so
fixes only affect NEW sessions:
`/etc/security/limits.d/99-memlock.conf` with
`* - memlock unlimited` + `root - memlock unlimited`; the container needs
`--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
the container reports `unlimited` AND small pins succeed; if pins still
fail, memlock was never the cause.
4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
`grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
host with hundreds of GiB free, which invites aborting something healthy.
Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
Use `scripts/dankstatus.sh` instead of hand-rolling.
5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
(grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
`| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
`rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
yields 0.00; filter to `^card[0-9]+$`.
6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
and confirm the drop was actually in effect for the run before treating a
failure as a capacity wall — a stale "it doesn't fit" from a contaminated
run is expensive to unwind.
## 5. Memory budgeting: the levers are not where they look
Every figure in this section is a tcclaviger-era boot on the port that is
[[frank-archon]]'s today; the laws they teach are general. The planner builds
`budget = total × util`, subtracts a KV reserve, and gives
the REMAINDER to the expert arena. Consequences, all measured:
- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
context shrinks the reserve, handing the space to the arena — measured 65536
→ 0.64 GiB KV, 32768 → 0.34 GiB KV.
- **Hybrid-attention undercount:** the planner's per-token KV figure for
Qwen4Exp is an undercount — the printed figure and the true one are both on
[[frank-archon]] — because it misses the large fixed per-sequence Mamba state
(~2x), so the reserve is always short and the arena silently covers it.
Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (era reading: a
2 GiB pin = 113,595 tokens ≈ 3.47x concurrency at 32k — that pair is this
page's record of the tcclaviger boots, not [[frank-archon]]'s). For long
context,
`--kv-offloading-size` (GiB) spills KV to host.
- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
(`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
util (see [[frank-archon]] for the
util=1.0 unbootability law).
**Read the plan — it is the real acceptance test:**
```
expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
budget (total x util) 30.27 GiB
non-expert weights 6.28 GiB
KV reserve 0.36 GiB <- sanity-check against the model's own per-token figure
CUDA graphs 0.66 GiB
runtime overhead 7.60 GiB
expert cache space 15.36 GiB 6712 slots of 2.344 MiB
host-backed experts/rank 52.74 GiB 23040 experts
host: 105.48 GiB across 2 ranks of 160.89 GiB usable
```
Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
SHARED across TP ranks.
**Loading mechanism** (for reasoning about peak RAM): weights never materialise
in VRAM — expert Parameters start as CPU staging tensors; each layer's
`weight_loader` counts shard events and, on layer completion,
`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
and compacts the rest into exact-size pinned host tensors behind UVA. Peak
host RAM = `host-backed set + layers still loading`, not the whole model. At
runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
rest straight over PCIe via UVA views — no staging copy in the critical path.
**Emulation check:** a dequantize-to-compute path doubles real host RAM — the
measured pair (offload request vs host footprint) is on [[frank-archon]] — and
the doubling is invisible in the weight count. Read the log for `Using
'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".
## 6. Performance profile and honest measurement
Decode on this pair is **PCIe-gather-bound, context-independent**: the rate is
flat from a few hundred to sixteen thousand context tokens, so KV attention is
not the cost (the flat figure and its MTP/gather deltas are
[[Tcclaviger Fork Ops]]'s profile of the same shape). With ~29 % of experts
arena-resident, under a gigabyte crosses PCIe per token at the engine's
effective read rate; cold-vs-warm differing ~10 % confirms the LRU works;
growth is linear in hit rate, and only speculative decode cuts sequential STEPS
rather than making each cheaper.
**Benchmark traps that produced fake numbers here:**
- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
length). Correct TTFT: stream and time the FIRST content-bearing chunk.
- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
a large vocabulary in shuffled order and print `cached_tokens`.
- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
prefill at 30k+ only.
- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
pollutes the metric); guessing words-per-token silently overran the window
(HTTP 400).
- Always run a canary: plant a unique token at the END of a long prompt and
ask the model to quote it — proves prompts reach the model at all.
- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
concurrent streams scaled to 47 aggregate.
**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
see [[frank-archon]] for why it was rejected);
`--expert-precise-memory` swaps the fitted overhead figure in.
## Related
- [[frank-archon]] — deployed shape, plan law,
measured campaigns
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
- [[frank-dragoon]] and [[frank-zealot]] — window discipline on live hosts

Some files were not shown because too many files have changed in this diff Show More